topic Re: high cpu util in Operating System - HP-UX

high cpu util

trebor_1 — Mon, 15 Feb 2010 03:09:47 GMT

Hi Experts,

Just want to ask your expert advice on this..I have a machine which consistently hits cpu of 100%

CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
avg 8.39 0.2% 26.2% 73.6% 0.0% 0.0% 0.0% 0.0% 0.0%

system value is 73.6% is this normal?

top process is the user tprxxxx w/c has 270 process.

If you need other logs i would gladly provide it.Thanks

Re: high cpu util

Bill Hassell — Mon, 15 Feb 2010 03:52:08 GMT

Sounds like you are getting your money's worth out of the computer. Seriously, a single user tprxxxx with 270 processes running sounds a bit unusual. What are all the processes? Use this command to count the processes:

ps -u tprxxxx -ocomm= | sort | uniq -c | sort -rn

Contact this user and ask what they are doing.

Re: high cpu util

Bill Hassell — Mon, 15 Feb 2010 03:54:06 GMT

Almost forgot -- the complete command line is:

UNIX95= ps -u tprxxxx -o comm= | sort | uniq -c | sort -rn

Re: high cpu util

trebor_1 — Mon, 15 Feb 2010 05:08:32 GMT

Hi Bill,

Thanks for the reply.

here's the output of the command

# UNIX95= ps -u tpr8060 -o comm= | sort | uniq -c | sort -rn
71 sh
47 perl
43 ksh
15 ls
13 ssh
13 grep
13 awk
7 tee
7 ssh-rand-helper
7 sftp
5 sshd:
4 -ksh
2 who
2 rm
2 ftp
1 view
1 sftp-server
1 netstat
1 mv
1 gzip
1 find
1 cp
#
Most of this process is perl scripts and native scipts to transfer file from server to server.

Re: high cpu util

Dennis Handly — Mon, 15 Feb 2010 06:40:55 GMT

>system value is 73.6% is this normal?

It sounds like you are doing lots of I/O or waiting.
Do you have glance installed find the cause?
You probably need some sar(1m) reports.
What does "swapinfo -tam" show?

Re: high cpu util

trebor_1 — Mon, 15 Feb 2010 07:46:11 GMT

Hi Dennis,

output of swapinfo

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16192 0 16192 0% 0 - 1 /dev/vg00/lvol2
dev 10000 118 9882 1% 0 - 0 /dev/vg00/lvswap
dev 51188 117 51067 0% 0 - 0 /dev/vgswap/lvswap
reserve - 2110 -2110
memory 6540 1727 4813 26%
total 83920 4072 79844 5% - 0

Output of sar

# sar 5 5

HP-UX server01 B.11.11 U 9000/800 02/15/10

15:36:11 %usr %sys %wio %idle
15:36:17 83 17 0 0
15:36:22 82 18 0 0
15:36:27 85 15 0 0
15:36:32 87 13 0 0
15:36:37 88 12 0 0

Average 85 15 0 0
#
System value drops from 76% to an average of 17%.I but didnt do anything yet.

Re: high cpu util

Dennis Handly — Mon, 15 Feb 2010 07:59:38 GMT

>memory 6540 1727 4813 26%

It seems you have enough memory now.

>I but didn't do anything yet.

Well, waiting enabled those processes to finally finish. You'll need to monitor them again, when you have that high system load.

Re: high cpu util

trebor_1 — Mon, 15 Feb 2010 08:02:24 GMT

Hi Guys,

Do you have any scripts that can monitor this so i can track them when peak time comes?

Thanks

Re: high cpu util

Johnson Punniyalingam — Mon, 15 Feb 2010 13:16:08 GMT

Check out the following thread...Victor attached a script that monitors resources including CPU. You could probably apply at least a part of it to suit your needs.

http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1035476&admit=109447627+1266239601019+28353475

HTH,

Johnson

Re: high cpu util

Bill Hassell — Mon, 15 Feb 2010 13:49:03 GMT

> here's the output of the command
>
> # UNIX95= ps -u tpr8060 -o comm= | sort | uniq -c | sort -rn
> 71 sh
> 47 perl
> 43 ksh
> 15 ls
> 13 ssh
> 13 grep
> 13 awk

Why do you think there is a problem? You have 13 ssh sessions running and they look quite normal. Of course if you know that there is supposed to be only one session running, then talk to the user tprxxxx and find out why the processes don't look normal.

> Most of this process is perl scripts and native scipts to transfer file from server to server.

What does the author of the scripts say about this condition? Are these scripts started when certain conditions exist? How are these scripts started? By cron? Could it be that when there is a file to transfer and it takes a long time, the script is started again and again? You need to use lock files to prevent this plus a monitor script to report on the overall transfer status.

> System value drops from 76% to an average of 17%.I but didnt do anything yet

Sounds like the scripts are running, then completing their tasks. There's almost nothing you can do until you understand what the script are supposed to do and when they are supposed to run. It is trivial to write a script that uses 100% of your CPU. That does not prevent users from logging in, running vi or even running more scripts. In HP-UX, CPU usage for users is a low priority, that is, a process that uses 100% for a while will share CPU cycles with other users.

On the other hand, a very high sys value (ie, 70% versus 25% user) can indicate a massive I/O condition. In your case, the load average indicates that there are a very large number of processes waiting to run (8.39) but whether this is normal depends on the scripts and processes that are running.

Be sure to look at /var/adm/syslog/syslog.log for possible errors and warnings.