High Page Fault

[Hakki Aydin Ucar]

and besides , of course top and glance to see which process consumes too much CPU as Venkatesh said before..

[NDurand]

i have checked the system for several hours and it stay stucked at 100%.

The process which are consumming are java process with only one thread and doing a lot of Vfault (found with glance)

[Duncan Edmonstone]

So this java program - who wrote it and what does it do? Has it *always* behaved like this, or just started to recently - who setup the startup parameters for the java program? have they been changed recently?

Perf tuning for java apps is quite a specialist area about which I know very little... you probably need to go back to the devlopers of the application and discuss with them... if you feel up to it, you could do worse than start reading this manual:

http://docs.hp.com/en/5992-4687/index.html

HTH

Duncan

[Dennis Handly]

>The process which are consuming are java process with only one thread and doing a lot of Vfault

Hmm, with that much vfaults, I wonder why no paging? Are your processes short lived as Duncan suggested? Perhaps you are getting too many TLB faults?

You may want to renice your unmutual java process and upon restart, give it less memory.

Can you provide "swapinfo -tam" output.

[NDurand]

here's the swapinfo -tam output

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 6144 0 6144 0% 0 - 1 /dev/vg00/lvol2
reserve - 6144 -6144
memory 32737 23754 8983 73%
total 38881 29898 8983 77% - 0 -

[Dennis Handly]

>here's the swapinfo -tam output
dev 6144 0 6144 0%
reserve - 6144 -6144
memory 32737 23754 8983 73%
total 38881 29898 8983 77%

You have reserved all of your device swap but haven't used it.
You have about 8.9 Gb free. Are these values the same during your "huge number of page faults"?

[NDurand]

Hi,

Yes, these values was mesured during the high page faults.

Rgds

[Don Morris_1]

Well, I had a much longer reply and the forum submittal ate it. Here's the quicker version:

Assuming the vmstat output is also from the high Page Fault period, the low at field and higher system call field along with the original high Faults but low KB Paged In stats makes me suspect that the java process may be doing some sort of page release/invalidation and re-instantiation activity (perhaps as part of Java GC work? And the change may be in the object creation/utilization in the workload?). The data as given certainly indicates that the application is frequently calling down to the kernel and that the pages generated are Demand fill pages, not anything to do with paging in or out.

tusc might give some indication if you sample the application over a period to see what syscalls it is making.

Another alternative would be to pick up Caliper (http://www.hp.com/go/caliper) and point it at the application, being sure to profile the kernel as well as the application to get an idea of what's going on in the system faults. I think this would be more likely to produce results.

[NDurand]

Thanks a lot for your help. I will try to profile the application beginning by the analyze of the garbage collector of the jvm to see if there is lot of GC or not.

Rgds

[Duncan Edmonstone]

Don,

> Well, I had a much longer reply and the forum submittal ate it. Here's the quicker version

Yes that regularly happens to me - many times my beautifully crafted essays are eaten by the forum submission monster and my follow up post is just "please do xyz" without any explantion why!

Still from my point of view, at least your (no doubt stinging) critique of my horribly vague ramblings on the inner workings of VM kernel voodoo will never see the light of day and I can go on pretending I know what I'm talking about!

Duncan