topic Re: Optimization/Speed up help needed on the script in Operating System - HP-UX

Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 14:53:45 GMT

Hi All,

I am attaching the script that takes more CPU usage.

Could you pls anybody help here to optimize the script in the way to reduce the burden on the CPU.

Thanks,
Gobinathan.S

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 14:57:58 GMT

It's a small script only but contains so many if then loops.

Anybody pls help me here, Points would be granted and appreciate on your valuble answer.

Re: Optimization/Speed up help needed on the script

James R. Ferguson — Thu, 18 Feb 2010 15:10:06 GMT

Hi:

Your script isn't doing a great deal of work expect for whatever 'opcmon' does.

For cases like:

# CMD_OUTPUT=`cat $SEARCH_NAME | grep -i "$4" | wc -l`

...you can eliminate a couple of processes by doing:

# CMD_OUTPUT=$(grep -c -i "$4" ${SEARCH_NAME})

Notice that 'grep' can count using the '-c' option and that 'grep' can read a file directly. Thus, both the 'cat' (to do I/O) and the 'wc' to count can be eliminated.

In optimizing, look for extra processes in pipelines that can be eliminated.

Regards!

...JRF...

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 15:33:21 GMT

Thanks, James!!

Do you mean that I need to replace the line with your provided line?

And, Please let me know what else can be modifed in order to reduce the usage of CPU, what do you mean by pipeline here? - Through some light directly on to it to make me understand very clear.

Actually speaking, I am new to scripting and this is an existing script in my environemnt.

Thanks,
Gobinathan.S

Re: Optimization/Speed up help needed on the script

Patrick Wallek — Thu, 18 Feb 2010 15:43:00 GMT

You may want to be careful when modifying this script.

All the calls to the "opcmon" program leads me to believe that this script could be part of the Openview Operations product.

Trying to do too much with this script **could** cause problems with Openview Operations and its ability to monitor your system.

What is the actual script name?

Re: Optimization/Speed up help needed on the script

James R. Ferguson — Thu, 18 Feb 2010 15:43:22 GMT

Hi (again):

> It's a small script only but contains so many if then loops

Based on how many files you match, yes. The 'if' evaluations aren't going to be very costly though.

Another side note is that you call the system for the date and time twice. This isn't a good idea since you can end up with data for different minutes, hours or even days.

It would be better to do something like:

# DATE=$(date +%H%M)
# value=$(echo ${DATE}|cut -c1-2)
# extension=$(echo ${DATE}|cut -c3-4)

Notice again, that instead of using the archaic backticks for command evaluation, I used the POSIX '$(...)' form.

Regards!

...JRF...

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 15:44:09 GMT

Also, Is it possible to use the statement as below or suggest me the way to go in a good way

Instead of using like:

case $4 in "Too many open files")
STATUS=0
if [ $value -eq 11 ] || [ $value -eq 15 ] then
if [ $CMD_OUTPUT -gt 500 ]
then
if [ $extension -lt 12 ]
then
STATUS=1
fi
fi
fi

Can I use:

if (( ( $value -eq 11 || $value -eq 15 ) && ( $CMD_OUTPUT -gt 500 ) && ( $extension -lt 12 ) )) then;
STATUS=1

Is this a good way of doing it, will it optimize my requirement.

Re: Optimization/Speed up help needed on the script

James R. Ferguson — Thu, 18 Feb 2010 15:50:40 GMT

Hi (again):

> Do you mean that I need to replace the line
with your provided line?

Yes. The line in question when replaced, eliminates a 'cat' and a 'wc' process.

> And, Please let me know what else can be modifed in order to reduce the usage of CPU, what do you mean by pipeline here?

Again, using:

# CMD_OUTPUT=`cat $SEARCH_NAME | grep -i "$4" | wc -l`

...versus:

# CMD_OUTPUT=$(grep -c -i "$4" ${SEARCH_NAME})

In the first case, a 'cat' process runs to read a file. That process then writes what it reads to a new process --- the 'grep'. This reads what appears in the pipe and writes whatever matches to yet another process --- 'wc' to count the matches. The 'wc' has to read the pipe's output too.

In the second case, I let the 'grep' read the input file; match and count what you want; and output (write) one result.

A pipeline is fundamental to UNIX processing, but every command in the pipe has to be spawned (created as a new process). Process creation is expensive. In the examples above, too, you repeat I/O reads and writes needlessly. This probably slows down the whole execution more than just creating the processes to do it.

Regards!

...JRF...

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 15:59:16 GMT

Could you pls answer my previous question?

and, pls give me clarity on pipeline fundamental and explain the lines and what i can modify it?

That would be better from next time i can consider while writing the script.

Re: Optimization/Speed up help needed on the script

James R. Ferguson — Thu, 18 Feb 2010 16:03:12 GMT

Hi (again):

> Can I use:

> if (( ( $value -eq 11 || $value -eq 15 ) && ( $CMD_OUTPUT -gt 500 ) && ( $extension -lt 12 ) )) then;

I would do:

# if [ $ $value -eq 11 -o $value -eq 15 $ -a $ $CMD_OUTPUT -gt 500 -a $extension -lt 12 $ ]; then

...see the manpages for 'test'.

It is good defensive shell coding to enclose your variables in double quotes when testing them. That way, empty (unassigned) variables don't lead to syntax errors.

It is also good practice to do variable evaulation with curly braces. In all:

# if [ $ "${value}" -eq 11 -o "${value}" -eq 15 $ -a $ "${CMD_OUTPUT}" -gt 500 -a "${extension}" -lt 12 $ ]; then

Regards!

...JRF...

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 16:11:45 GMT

Thanks and proud to interacting with Scripting master!!

"A pipeline is fundamental to UNIX processing, but every command in the pipe has to be spawned (created as a new process). Process creation is expensive. In the examples above, too, you repeat I/O reads and writes needlessly. This probably slows down the whole execution more than just creating the processes to do it."

Is it possible for you to give me more clarity on this and give me a solution how to modify this script on this pipeline issue? So that I can take it forward from my next script....

Thanks,
Gobinathan.S

Re: Optimization/Speed up help needed on the script

James R. Ferguson — Thu, 18 Feb 2010 16:22:05 GMT

Hi:

> Is it possible for you to give me more clarity on this and give me a solution how to modify this script on this pipeline issue?

I did. I have, and I don't know how else to explain it. Compare the lines of code that I showed you that set the value of 'CMD_OUTPUT'

Yours has a pipeline involving a 'cat', a 'grep' and a 'wc'. I removed the pipe altogether in this case. Pipes are very important and eliminate the use of temporary files that would otherwise be written and read. My point was to let a process do as much work as it is capable without piping its output to another process when possible. In the case of 'grep' it can read input; match; and output the matches or the count of the matches without the need to invoke another process.

Your scripting skills will grow with practice and by studying other people's code. A very good site containing lots of scripts and techniques is:

http://www.shelldorado.com/

Enjoy the journey!

Regards!

...JRF...

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 16:27:56 GMT

Thanks for all your prompt reply!!

You can expect many more questions in future!! :-)

Also, I'll start learn from my side!!

One last question, As we discussed by changing the if then statement as,

# if [ $ "${value}" -eq 11 -o "${value}" -eq 15 $ -a $ "${CMD_OUTPUT}" -gt 500 -a "${extension}" -lt 12 $ ]; then

Will it improve performance of Script? (Instead of using the old one which I'm having as below)

if [ $value -eq 11 ] || [ $value -eq 15 ] then
if [ $CMD_OUTPUT -gt 500 ]
then
if [ $extension -lt 12 ]
then
STATUS=1
fi
fi
fi

Thanks,
Gobinathan.S

Re: Optimization/Speed up help needed on the script

James R. Ferguson — Thu, 18 Feb 2010 16:35:48 GMT

Hi:

> One last question, As we discussed by changing the if then statement...Will it improve performance of Script? (Instead of using the old one which I'm having as below)

No, it really won't change the performance you see, but it makes the logic of your script a bit easier to understand.

Good code creates a balance between readability/maintainability and tricks to be fast. Remember, its the poor human that has to maintain the code, and that human being may not be the original writer!

Regards!

...JRF...

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 16:49:22 GMT

Hi Patrick,

I did not answer for your question, Sorry for that.

What is the actual script name?

testpattern.sh

Yes, I'm using this for Openview monitoring template only. If any suggestions, Pls add to it.

Re: Optimization/Speed up help needed on the script

Dennis Handly — Thu, 18 Feb 2010 17:12:37 GMT

>let me know what else can be modified in order to reduce the usage of CPU, what do you mean by pipeline here?

What OS version and architecture are you using? Do you have glance?
What does top(1) show as taking time? If not your shell, you probably can't do much.

>if (( ( $value -eq 11 || $value -eq 15 ) && ( $CMD_OUTPUT -gt 500 ) && ( $extension -lt 12 ) )) then

You should toss the goofy shell operators and go with C:
if (( (value == 11 || value == 15) && CMD_OUTPUT > 500 && extension < 12 )); then

This won't help much on performance.

Re: Optimization/Speed up help needed on the script

Gobinathan — Thu, 18 Feb 2010 17:38:53 GMT

OS/HW: HP-UX B.11.31 U ia64
Also, some cases we need to run on RHEL 4.0

Do you mean to use C program?

We are limited to Shell and Perl as the product that supports only these scripts.

Is it possible to use C operators as you mentioned in Shell?

Thanks,
Gobinathan.S

Re: Optimization/Speed up help needed on the script

Dennis Handly — Thu, 18 Feb 2010 17:42:17 GMT

>Do you mean to use C program?
>Is it possible to use C operators as you mentioned in Shell?

No, just use C operators inside shell (( )) arithmetic expressions. Exactly as I had.

Re: Optimization/Speed up help needed on the script

Jean-Philippe Henry — Sun, 21 Feb 2010 02:50:45 GMT

Hello Gobinathan,

My advice are the same as given above and additionally the following :

- suppress variable assignations as much as possible (although it compromises readability);

- reduce calls to echo or print (intermediate variables might be displayed in one shot);

- reduce execution of external commands (date might be executed only once and cut not used);

- use grep -F which should be, in theory, faster than grep (but this is probably not true anymore)

This is the result :

#!/bin/sh

typeset STATUS=0

integer value extension
date +'%H %M' | read value extension

integer CMD_OUTPUT=$( grep -Fci $4 $2/$3 )

print "$value\n$extension\n$CMD_OUTPUT"

case $4 in

"Too many open files")
(( ( value == 11 || value == 15 ) && CMD_OUTPUT > 500 && extension < 12 )) \
&& STATUS=1
;;

"ERR_REASON:255" |\
"ERR_REASON:20")
(( CMD_OUTPUT > 500 )) \
&& STATUS=1
;;

"ERR_REASON:8#")
(( CMD_OUTPUT > 10 )) \
&& STATUS=1
;;

"ERR_REASON:88")
(( CMD_OUTPUT > 100 )) \
&& STATUS=1
;;

"ERR_REASON:10" |\
"ERR_REASON:11" |\
"ERR_REASON:2" |\
"ERR_REASON:3")
(( ( value == 23 && extension < 55 ) && CMD_OUTPUT > 1 )) \
&& STATUS=1
;;

esac

print "/opt/OV/bin/OpC/opcmon $1=$STATUS -option PROCNAME=$CMD_OUTPUT"
/opt/OV/bin/OpC/opcmon $1=$STATUS -option PROCNAME=$CMD_OUTPUT

I copied out the actions that were done in every case conditions after esac.

What might improve the speed of your script is also to order the case conditions so that the most likely to be found comes first and the least comes last.

Cheers,

Jean-Philippe

Re: Optimization/Speed up help needed on the script

Jean-Philippe Henry — Sun, 21 Feb 2010 22:20:07 GMT

Hello again,

Sorry, I forgot to surround $4 with double quotes in the grep command line.

Please read:

integer CMD_OUTPUT=$( grep -Fci "$4" $2/$3 )

instead of:

integer CMD_OUTPUT=$( grep -Fci $4 $2/$3 )

Cheers,

Jean-Philippe