perl script taking too much cpu time

[Shivkumar]

we are using a perl script for parsing log files. we noted it is taking 94% cpu time which is too high i guess.

what could be the reason ? how to fix it ?

[Muthukumar_5]

Usually perl will take more CPU time. May be tuning of perl script will help or writing a shell scripting to parse that log file will help.

Post the script if you wish.

--
Muthu

[Arunvijai_4]

Hi Shiv,

Which version of Perl you are using ? Try with
# swlist |grep -i perl.

Also, are you looping infinitely inside the Perl script ?

-Arun

[Arunvijai_4]

Hi Shiv,

This could be helpful,

http://www.ccl4.org/~nick/P/Fast_Enough/

When perl is not quite fast enough

-Arun

[Arunvijai_4]

Hi Shiv,

Try to use the latest Perl version around for HP-UX,

http://h20293.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=PERL

For PA & IPF : 5.8.3 and 5.8.7 are available.

Also, you can download Procura's perl depots,

http://mirrors.develooper.com/hpux/downloads.html

-Arun

[Muthukumar_5]

If you want to make a script for parsing your log files then post here. Shell scripting will not much CPU as like perl.

--
Muthu

[Senthil Prabu.S_1]

Hi shiva,
It is advised to use shell script or java, if u feel Perl is using lot of memory.


Also, post your code, can we can try to fix if any memory leak issues are there.


HTH,
Prabu.S

[H.Merijn Brand (procura · Accepted solution]

Senthil, where did you get that advise from?

Perl will take memory. As much as it needs or as much as it can get, at least for the operations you tell it to do. Memory is also much faster than process switches, something that will be done extremely often when using shell scripting.
Unless you can use unix utilities that do exactly what you need (cut, join, sort, comm, ...) perl is almost always faster and easier in use than a shell script, and in 99.99% of the cases (much) faster than java.
Unless you cannot do without, I'd advice to never use java.

When parsing a large log, compare these two:

while () {
...
}

and

for () {
...
}

different? no? yes? I'll tell you, you'll notice.
The first reads the log line by line. The second reads the complete log at once, and thus taking extremly more memory, which it will not free till after the loop.

But you said you were using a lot of CPU, which can easily be caused by a less than optimally written regex in which backtracking occurs. So if CPU is the bottleneck, try find some complicated regular expressions in your code and post them here.

Enjoy, Have FUN! H.Merijn [ who writes perl instead of shell for everything that would take more than 5 lines ]

[Muthukumar_5]

Procura,

I hope Perl will generally use more cpu than other utilities for code execution. But it is more faster.... than other.

But will the code change in perl will reduce in cpu usage.. May be regular expression is using memory for () operation right.

I feel for cpu usage, we have to move to base level utilities in shell itself.

--
Muthu

[Arunvijai_4]

Hi Shiv,

As far as perl concerned, it is quite faster than other core utils when parsing log files. This is proven and many organizations are moving to perl now a days due to gain performance.

But, when you do a small (basic) operations, shell (unix core utils) are faster.

-Arun

[Steven E. Protter]

Shalom Shiv,

I don't think going to a different version of perl is going to have a significant impact on resource use.

If there is a specific bug causing this issue, going to a current version of perl will help.

The real issue is the code, which should be reviewed. The way you are doing this may be improved. There are good language forums here and I've had perl code review several times with excellent results.

As far as 94% cpu use.

This may not be a problem at all. If a system is idle and you run a script, there is absolutely nothing wrong with that script using 100% of CPU. After all the system isn't doing anything else important anyway.

We have several unix/linux servers here which run at 100% all day and all night. There is nothing wrong with that.

Bill Hassell has recommended looking at other parameters to determine whether or not we have a problem. I agree:

1) Are the users having a response problem?
2) Is the system slow in doing normal administration tasks?

Examples. If the system is running correctly and user response is acceptable, the entire premise of this question may not be valid. If the system is fine, its not broke, so don't try and fix it.

SEP