system processes consuming high CPU resource.

[Phani_4]

Hi,

I have HP9000/K380 server with three 240 Mhz CPU.

When i looked at the top o/p the system parameter consuming high cpu...it is varying from 0% to 70%...But no one system processes consuming more than 3% of CPU individually..
"vhand" process consuming 3% of CPU even though there is no much Paging.

Can somebody please guide me how to resolve this issue..

Waiting for your prompt reply.

Thanks & Regards,
Phani Varma P.

[Todd McDaniel_1]

Have you looked at "sar -uM" see what your output is since midnite. it will show you individual cpu util

Also, many times on a small box with few cpu, you will see a "favorite" cpu based on the "shortness" of the path.

Are all or most of these procs on one CPU?

[Phani_4]

Todd,

Thanks for your response..Please find the attached sar o/p on this server.

One month back server was rebooted with HPMC, & found it is becasue of Cache parity error on one of the CPU..But we have not changed it..server is up & running fine..

Phani

[Todd McDaniel_1]

Was this for the time period you were concerned about?

Show the whole day of the occurences you are concerned with.

Are these few hours what you are worried about?

[Todd McDaniel_1]

Also, without knowing how your "normal" day goes and what are your high and low use times it would be hard to diagnose anything.


What you show is not unusual, imho. but it may be for your environment. Just b/c one cpu is at 70% is not a great concern to me.


If you can give a bit of detail on your normal day, it would be great.

[Phani_4]

Hi Todd,

I am waiting for my user's response on the server's performance in 1 Hr. time...becasue they are facing performance issue in the night time..

If you are available for some time, so that i can send you o/p of below commands.

# sar -uM 2 50
# sar -d 2 50

# vmstat 2 50

Thanks,
Phani

[Sundar_7]

Hi Phani,

# UNIX95= ps -ef -o pid,ppid,pcpu,args | sort -nbk 3 | tail -10

Post the output

Sundar

[Phani_4]

Sundar,

Please find the attached doc with o/p which u requested.

-o is not available in hp-ux.

Thanks,
Phani

[Sundar_7]

Hi Phani,

Are you kidding me ? :-) . I copied and pasted the command line from my test HP-UX box.

# UNIX95= ps -ef -o pid,ppid,pcpu,args | sort -nbk 3 | tail -10

Make sure there is a space after UNIX95=

or try this

# export UNIX95=1
# ps -ef -o pid,ppid,pcpu,args | sort -nbk 3 | tail -10
# unset UNIX95

Sundar

[Phani_4]

Hi Sundar,

I am not sure which HP-UX OS Version you are using..i didn't find the -o option in 10.20 & 11.11 as well.

# uname -a
HP-UX suboz039 B.11.11 U 9000/800 818622396 unlimited-user license
#
# ps -ef -o pid,ppid,pcpu,args | sort -nbk 3 | tail -10
ps: illegal option -- o
usage: ps [-edaxflP] [-u ulist] [-g glist] [-p plist] [-t tlist] [-R prmgroup]

Thanx,
Phani

[RAC_1]

UNIX95 variable is used to conform to XPG4.
Just do as follows.

UNIX95= ps -ef -o "all options here"

This will work. Post the output.

Anil

[Phani_4]

Hi Anil,

Thanx for your information..please find the below o/p

suboz026:/ > UNIX95= ps -ef -o "pid,ppid,pcpu,args"| sort -nbk 3 | tail -10
29669 19130 0 sh -c $GEMMS_ADDON/start_prem_scr
29670 29669 0 /PRDRS_cust_appl/add_on/start_prem_scr /PRDRS_cust_appl/add_
29679 29670 0 runform30x -c sc_appl:vt220 /PRDRS_cust_appl/add_on/prem_scr
29687 29679 0 oraclePRDFC (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq))
29800 1 0 /PRDRS_cust_appl/osb/program/killer.sh /PRDRS_cust_appl/osb/
29803 29800 0 sleep 1200
29834 1480 0 telnetd
29835 29834 0 -ksh
29913 1 0 oraclePRDNR (LOCAL=NO)
27488 1 1 oraclePRDNR (LOCAL=NO)

[doug mielke]

I would be a bit concerned if the system % went up suddenly. Don't put too much weight in the devision SAR makes between user and syatem, some conditions seem to show up in catagories that you wouldn't expect.

Since you have sar history on, try comparing a day's worth of sar with some from 2 weeks ago. Disk re- reads, as when a disk is starting to go bad, should show up as w i/o, but the real number to see is the sar -d service time. A network problem could show as nothing more thatn Hhigh system usage. Don't forget to look at name resolution. Changes here can hide themselves pretty well.

[Phani_4]

Hi Doug,

Please find the attached sar o/p on CPU & disk i/o when the end users experienced performance issues.

Thanx,
Phani

[RAC_1]

What is total system process usage and what is users and what is idle?

Did you look at swapinfo -mat, if vhand is in play system is paging out. Do get you get any pageouts in vmstat 5 5?

If not you should not worry about it. The point here is if enough CPU is avilable to users or not?

Anil

[Phani_4]

Hi Anil,

I have just replied to doug's message with all the o/p which you have asked..

Please find the attached o/p in my last message.

Thanx,
Phani

[RAC_1]

I did not fine anything unusual with sar -uM output. In fact when users usage is high, the idle has gone down. Which is obivous.

But for disk c8t3d3, what is the problem? The % wait time, %serve time, %busy -- everything is high. What applications you run on this machine? What is causing performance problem for disk-c8t3d3. What are your buffer settings- dbc_max_pct and dbc_min_pct?
What is swap size and how it is configured.

Anil

[Phani_4]

Hi Anil,

What applications you run on this machine?

Running Oracle Databse on this LUN

What is causing performance problem for disk-c8t3d3. What are your buffer settings- dbc_max_pct and dbc_min_pct?

Disk i/o going high because of the Oracle Databse HOT backup ( exp & compress ).

dbc_max_pct 50 - - -
dbc_min_pct 5 - - -

Physical Memory : 1.2 GB

Swap Defined : 1:5 GB


What is swap size and how it is configured.

The existing swap is Primary Swap ( Device Swap on root disk )..no secondary swap configured.

Thanx,
Phani

[RAC_1]

Disk problem OK. That explains why the disk is so busy.

About your dbc_max_pct and dbc_min_pct. The setting dc_max_pct is too high. In fact it default. That means buffer cache varies from 60mb to 600mb. are you getting it used?

Check sar -b and %rcache. If it is more than 90 % most of the time, you are effectively using it. Otherwise there is no need to set it that high. You can set dbc_max_pct to 10/15 %. This will give more memory to users and less memeory to system. You can it with glance -m

Anil

[Phani_4]

Hi,

The read cache utilzation is 100% & write cache utilization is on an average 98%..

What i observed was the 'sys' CPU utilization is high, when compared to other servers..

How to find out which system process is consuming high CPU ?

Regards,
Phani

[Navin Bhat_2]

You could very well be overloading the capacity of the box. You could reduce your dbc_max_pct and dbc_min_pct to gain some memory. Other wise I think you might just need a faster box. If you think this is something thats happened overnight and no changes were made then analyze the measureware log files.