Optimization/Speed up help needed on the script

[Gobinathan]

Hi All,

I am attaching the script that takes more CPU usage.

Could you pls anybody help here to optimize the script in the way to reduce the burden on the CPU.

Thanks,
Gobinathan.S

[Gobinathan]

It's a small script only but contains so many if then loops.

Anybody pls help me here, Points would be granted and appreciate on your valuble answer.

[James R. Ferguson]

Hi:

Your script isn't doing a great deal of work expect for whatever 'opcmon' does.

For cases like:

# CMD_OUTPUT=`cat $SEARCH_NAME | grep -i "$4" | wc -l`

...you can eliminate a couple of processes by doing:

# CMD_OUTPUT=$(grep -c -i "$4" ${SEARCH_NAME})

Notice that 'grep' can count using the '-c' option and that 'grep' can read a file directly. Thus, both the 'cat' (to do I/O) and the 'wc' to count can be eliminated.

In optimizing, look for extra processes in pipelines that can be eliminated.

Regards!

...JRF...

[Gobinathan]

Thanks, James!!

Do you mean that I need to replace the line with your provided line?

And, Please let me know what else can be modifed in order to reduce the usage of CPU, what do you mean by pipeline here? - Through some light directly on to it to make me understand very clear.

Actually speaking, I am new to scripting and this is an existing script in my environemnt.

Thanks,
Gobinathan.S

[Patrick Wallek]

You may want to be careful when modifying this script.

All the calls to the "opcmon" program leads me to believe that this script could be part of the Openview Operations product.

Trying to do too much with this script **could** cause problems with Openview Operations and its ability to monitor your system.

What is the actual script name?

[James R. Ferguson]

Hi (again):

> It's a small script only but contains so many if then loops

Based on how many files you match, yes. The 'if' evaluations aren't going to be very costly though.

Another side note is that you call the system for the date and time twice. This isn't a good idea since you can end up with data for different minutes, hours or even days.

It would be better to do something like:

# DATE=$(date +%H%M)
# value=$(echo ${DATE}|cut -c1-2)
# extension=$(echo ${DATE}|cut -c3-4)

Notice again, that instead of using the archaic backticks for command evaluation, I used the POSIX '$(...)' form.

Regards!

...JRF...

[Gobinathan]

Also, Is it possible to use the statement as below or suggest me the way to go in a good way

Instead of using like:

case $4 in "Too many open files")
STATUS=0
if [ $value -eq 11 ] || [ $value -eq 15 ] then
if [ $CMD_OUTPUT -gt 500 ]
then
if [ $extension -lt 12 ]
then
STATUS=1
fi
fi
fi

Can I use:

if (( ( $value -eq 11 || $value -eq 15 ) && ( $CMD_OUTPUT -gt 500 ) && ( $extension -lt 12 ) )) then;
STATUS=1

Is this a good way of doing it, will it optimize my requirement.

[James R. Ferguson]

Hi (again):

> Do you mean that I need to replace the line
with your provided line?

Yes. The line in question when replaced, eliminates a 'cat' and a 'wc' process.

> And, Please let me know what else can be modifed in order to reduce the usage of CPU, what do you mean by pipeline here?

Again, using:

# CMD_OUTPUT=`cat $SEARCH_NAME | grep -i "$4" | wc -l`

...versus:

# CMD_OUTPUT=$(grep -c -i "$4" ${SEARCH_NAME})

In the first case, a 'cat' process runs to read a file. That process then writes what it reads to a new process --- the 'grep'. This reads what appears in the pipe and writes whatever matches to yet another process --- 'wc' to count the matches. The 'wc' has to read the pipe's output too.

In the second case, I let the 'grep' read the input file; match and count what you want; and output (write) one result.

A pipeline is fundamental to UNIX processing, but every command in the pipe has to be spawned (created as a new process). Process creation is expensive. In the examples above, too, you repeat I/O reads and writes needlessly. This probably slows down the whole execution more than just creating the processes to do it.

Regards!

...JRF...

[Gobinathan]

Could you pls answer my previous question?

and, pls give me clarity on pipeline fundamental and explain the lines and what i can modify it?

That would be better from next time i can consider while writing the script.

[James R. Ferguson]

Hi (again):

> Can I use:

> if (( ( $value -eq 11 || $value -eq 15 ) && ( $CMD_OUTPUT -gt 500 ) && ( $extension -lt 12 ) )) then;

I would do:

# if [ \( $value -eq 11 -o $value -eq 15 \) -a \( $CMD_OUTPUT -gt 500 -a $extension -lt 12 \) ]; then

...see the manpages for 'test'.

It is good defensive shell coding to enclose your variables in double quotes when testing them. That way, empty (unassigned) variables don't lead to syntax errors.

It is also good practice to do variable evaulation with curly braces. In all:

# if [ \( "${value}" -eq 11 -o "${value}" -eq 15 \) -a \( "${CMD_OUTPUT}" -gt 500 -a "${extension}" -lt 12 \) ]; then

Regards!

...JRF...

[Gobinathan]

Thanks and proud to interacting with Scripting master!!

"A pipeline is fundamental to UNIX processing, but every command in the pipe has to be spawned (created as a new process). Process creation is expensive. In the examples above, too, you repeat I/O reads and writes needlessly. This probably slows down the whole execution more than just creating the processes to do it."

Is it possible for you to give me more clarity on this and give me a solution how to modify this script on this pipeline issue? So that I can take it forward from my next script....

Thanks,
Gobinathan.S

[James R. Ferguson · Accepted solution]

Hi:

> Is it possible for you to give me more clarity on this and give me a solution how to modify this script on this pipeline issue?

I did. I have, and I don't know how else to explain it. Compare the lines of code that I showed you that set the value of 'CMD_OUTPUT'

Yours has a pipeline involving a 'cat', a 'grep' and a 'wc'. I removed the pipe altogether in this case. Pipes are very important and eliminate the use of temporary files that would otherwise be written and read. My point was to let a process do as much work as it is capable without piping its output to another process when possible. In the case of 'grep' it can read input; match; and output the matches or the count of the matches without the need to invoke another process.

Your scripting skills will grow with practice and by studying other people's code. A very good site containing lots of scripts and techniques is:

http://www.shelldorado.com/

Enjoy the journey!

Regards!

...JRF...

[Gobinathan]

Thanks for all your prompt reply!!

You can expect many more questions in future!! :-)

Also, I'll start learn from my side!!

One last question, As we discussed by changing the if then statement as,

# if [ \( "${value}" -eq 11 -o "${value}" -eq 15 \) -a \( "${CMD_OUTPUT}" -gt 500 -a "${extension}" -lt 12 \) ]; then

Will it improve performance of Script? (Instead of using the old one which I'm having as below)

if [ $value -eq 11 ] || [ $value -eq 15 ] then
if [ $CMD_OUTPUT -gt 500 ]
then
if [ $extension -lt 12 ]
then
STATUS=1
fi
fi
fi


Thanks,
Gobinathan.S

[James R. Ferguson]

Hi:

> One last question, As we discussed by changing the if then statement...Will it improve performance of Script? (Instead of using the old one which I'm having as below)

No, it really won't change the performance you see, but it makes the logic of your script a bit easier to understand.

Good code creates a balance between readability/maintainability and tricks to be fast. Remember, its the poor human that has to maintain the code, and that human being may not be the original writer!

Regards!

...JRF...

[Gobinathan]

Hi Patrick,

I did not answer for your question, Sorry for that.

What is the actual script name?

testpattern.sh

Yes, I'm using this for Openview monitoring template only. If any suggestions, Pls add to it.

[Dennis Handly]

>let me know what else can be modified in order to reduce the usage of CPU, what do you mean by pipeline here?

What OS version and architecture are you using? Do you have glance?
What does top(1) show as taking time? If not your shell, you probably can't do much.

>if (( ( $value -eq 11 || $value -eq 15 ) && ( $CMD_OUTPUT -gt 500 ) && ( $extension -lt 12 ) )) then

You should toss the goofy shell operators and go with C:
if (( (value == 11 || value == 15) && CMD_OUTPUT > 500 && extension < 12 )); then

This won't help much on performance.

[Gobinathan]

OS/HW: HP-UX B.11.31 U ia64
Also, some cases we need to run on RHEL 4.0

Do you mean to use C program?

We are limited to Shell and Perl as the product that supports only these scripts.

Is it possible to use C operators as you mentioned in Shell?

Thanks,
Gobinathan.S

[Dennis Handly]

>Do you mean to use C program?
>Is it possible to use C operators as you mentioned in Shell?

No, just use C operators inside shell (( )) arithmetic expressions. Exactly as I had.

[Jean-Philippe Henry]

Hello Gobinathan,

My advice are the same as given above and additionally the following :

- suppress variable assignations as much as possible (although it compromises readability);

- reduce calls to echo or print (intermediate variables might be displayed in one shot);

- reduce execution of external commands (date might be executed only once and cut not used);

- use grep -F which should be, in theory, faster than grep (but this is probably not true anymore)

This is the result :

#!/bin/sh

typeset STATUS=0

integer value extension
date +'%H %M' | read value extension

integer CMD_OUTPUT=$( grep -Fci $4 $2/$3 )

print "$value\n$extension\n$CMD_OUTPUT"

case $4 in

"Too many open files")
(( ( value == 11 || value == 15 ) && CMD_OUTPUT > 500 && extension < 12 )) \
&& STATUS=1
;;

"ERR_REASON:255" |\
"ERR_REASON:20")
(( CMD_OUTPUT > 500 )) \
&& STATUS=1
;;

"ERR_REASON:8#")
(( CMD_OUTPUT > 10 )) \
&& STATUS=1
;;

"ERR_REASON:88")
(( CMD_OUTPUT > 100 )) \
&& STATUS=1
;;

"ERR_REASON:10" |\
"ERR_REASON:11" |\
"ERR_REASON:2" |\
"ERR_REASON:3")
(( ( value == 23 && extension < 55 ) && CMD_OUTPUT > 1 )) \
&& STATUS=1
;;

esac

print "/opt/OV/bin/OpC/opcmon $1=$STATUS -option PROCNAME=$CMD_OUTPUT"
/opt/OV/bin/OpC/opcmon $1=$STATUS -option PROCNAME=$CMD_OUTPUT

I copied out the actions that were done in every case conditions after esac.

What might improve the speed of your script is also to order the case conditions so that the most likely to be found comes first and the least comes last.

Cheers,

Jean-Philippe

[Jean-Philippe Henry]

Hello again,

Sorry, I forgot to surround $4 with double quotes in the grep command line.

Please read:

integer CMD_OUTPUT=$( grep -Fci "$4" $2/$3 )

instead of:

integer CMD_OUTPUT=$( grep -Fci $4 $2/$3 )

Cheers,

Jean-Philippe