Glance - 100% CPU Utilisation

[Conal O'Kelly]

Hi
From glance I'm noticing 100% CPU utilisation on one of our servers. Drilling down with the "c" option I can see that Vfaults is accounting for approximately 60% of the CPU's time. I know this is telling me the CPU is spending a large amount of time handling page faults. But how can I determine what is causing this & why is the CPU doing this. Looking at the top processes by CPU usage - there are 4 Oracle processes that account for most of the CPU usage and have accumulated large amounts of CPU time. In addition a process called native_thread has also got a very high cumulative CPU time.

There is no memory pressure on the system & no swapping.
System is an N Class with 4 x 550Mhz CPU's and 6GB of memory. Currently there is about 2.5GB of memory free.

Cheers
Con

[Sanjay_6]

hi Con,

You may want to increase the kernel parameters max_thread_proc and nkthread to resolve the native_thread problem on your system. Here is a link on configurable kernel parameters,

http://docs.hp.com//hpux/onlinedocs/os/11i/kcparams/KCparams.OverviewAll.html

Here is a thread on max_thread_proc parameter,

http://docs.hp.com//hpux/onlinedocs/os/11i/kcparams/KCparam.MaxThreadsPerProcess.html

And here is the thread on nkthread,

http://docs.hp.com//hpux/onlinedocs/os/11i/kcparams/KCparam.NkernelThreadsMax.html

Here is a link on the kernel parameter you should look into for oracle,

http://docs.hp.com//hpux/onlinedocs/os/11i/kcparams/KCparam.ProcessParmsOverview.html

Hope this helps.

Regds

[Conal O'Kelly]

Thanks for the reply. I have looked at the threads you posted but am not sure how this affects vfaults for CPU util. Currently nkthread is set at 14800 ( nproc * 7), max_thread_proc is at 256. Im not sure at this stage if increasing these params will resolve my problem. If these values are too low is this likely to show up in high CPU Util??

Thanks.

Cheers
Con

[Sanjay_6]

Hi Con,

The max_thread_proc and nkthread are to solve your native_thread problem.

Hope this helps.

Regds

[Sanjay_6]

Hi Con,

To look into your vfault problem, looks like you need some LVM/filesystem patches. See if you need any of those patches. Do a custom patch analysis for your box. There is a possibility you may have a patch that has been recalled and it might be causing problem for you. There are some patch related to this on 10.20 but you must be having 11.0/11i.

Hope this helps.

Regds

[Sanjay_6]

Hi Con,

Try this patch if you are using 11.0,

http://us-support.external.hp.com/wpsl/bin/doc.pl/screen=wpslDisplayPatch/sid=35d507d903d1b166d3?PATCH_PATH=/hp-ux_patches/s700_800/11.X/PHKL_22932&HW=s800&OS=11.00

And, these if you are using 11i,

http://us-support.external.hp.com/wpsl/bin/doc.pl/screen=wpslDisplayPatch/sid=7a84fc050e8294e17e?PATCH_PATH=/superseded_patches/hp-ux_patches/s700_800/11.X/PHKL_23425&HW=s800&OS=11.11

http://us-support.external.hp.com/wpsl/bin/doc.pl/screen=wpslDisplayPatch/sid=7a84fc050e8294e17e?PATCH_PATH=/hp-ux_patches/s700_800/11.X/PHKL_23335&HW=s800&OS=11.11

Hope this helps.

Regds

[Conal O'Kelly]

Hi Sanjay
Thanks for all replies - I'll post the points later.
Am I correct to assume this patch is only related to UFS. All our filesystems are vxfs (except /stand).
After examing more glance data I can report the following.
Page faults are extremely high - from glance ("m" option) the cumualtive rate is 161700 page faults per sec ! On all our other servers its approx 100-300 per sec. Also as I mentioned before the CPU is spending approximately 70% of its time handling page faults.
Any suggestions on how to determine which processes are responsible or how this can get so high. Also as mentioned there is no memory pressure on this system.
One suggestion was that it was linked to a fairly low buffer cache (fixed at 200MB) but I can't see how this could be.

Cheers
Con