High Page Fault

[NDurand]

Hi,
We actually encounter a problem on rp4460 with hp ux 11.23. The problem is the huge number of page fault wich, i think, is the cause of high cpu usage.

I'm not sure the memory is really under pressure as free mem is around 4GB/32GB.

I've noticed that the problem increases with the number of processes, but i don't know why.

Thanks for your help
MEMORY REPORT Users= 10
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 4988k 1010559029 959273.0 620393.5 1177835.9
Page In 47 1686 9.0 1.0 1291.3
Page Out 0 0 0.0 0.0 0.0
KB Paged In 188kb 6.6mb 36.1 4.1 465.3
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 1 201 0.1 0.1 24.0
VM Writes 0 636 0.0 0.3 10.0

[Venkatesh BL]

Is this causing a problem?

When new processes are run, the pages the process touches, have to be faulted on demand. This would be indicated as Page In.

[NDurand]

Thanks for your reply,

Yes it is a problem as the cpu is always 100% used and 90% by system .

Rgds

[Duncan Edmonstone]

Not necessarily a problem... do you fully understand what page faults are?

It would be interesting to see if you also have high values in the at (address translation) column in a vmstat output.

If I run  a new process for the first time on a system, the text pages (program code) of that process will not be in physcial memory so this will generate a page fault, and then (as the process has not been run before) a page-in from physical disk (i.e. loading the program code into physical memory).

When the process terminates, we don't need to write out for the static program code, so we don't get a page out from that and the pages will stay in physical memory (although not referenced by any processes) unless there is memory pressure.

Now if I run the same process again - the kernel virtual memory subsystem will know that those pages are already in physical memory and instead of having to do another page in, it can just do an address translation (which you can see from vmstat)

The upshot of all this is that you have a lot of processes that run and then terminate very often you can see very high page faults without a high number of page-ins, as these are just address translations.

There may be other reasons too (I'm not sure if memory mapped files play in this space as well), but we'll need a VM wizard such as Don Morris to answer that sort of question (and probably correct a few errors in my explanation above as well!)

HTH

Duncan

[Dennis Handly]

>the huge number of page fault which, I think, is the cause of high CPU usage.

As mentioned by Venkatesh, there are no page outs. Typically lots of paging wouldn't cause high CPU usage but lots of I/O.

[NDurand]

Thank you Denis and Duncan for your clear explanation.



Here's an output of vmstat, as you can see cpu is really occuped:
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
12 2 0 7676486 816524 220 29 3 0 0 0 4 4028 261890 3660 12 74 15
12 2 0 7676486 816447 176 29 0 0 0 0 0 3104 85448 3531 7 93 0
12 2 0 7676486 816510 152 25 0 0 0 0 0 3084 71618 3488 8 92 0
16 2 0 7680128 815360 163 27 0 0 0 0 0 3032 69571 3348 8 92 0
16 2 0 7680128 815592 344 61 0 0 0 0 0 3244 89772 3689 12 88 0

[NDurand]

here an output of sar:
HP-UX su0374 B.11.23 U ia64 09/15/09

12:13:42 %usr %sys %wio %idle
12:13:45 11 89 0 0
12:13:48 7 93 0 0
12:13:51 10 90 0 0
12:13:54 8 92 0 0

Can you confirm high %sys is a normal beahviour ?

Thanks in advance

[Duncan Edmonstone]

Well you certainly don't seem to have a lot og page-ins going on there, but then neither do you appear to have a lot of address translations...

A %sys of 90% is not particularly normal - what application is running on this host? Can you identify waht processes are using all the system CPU time? (from your original output it looks like you have glance)

HTH

Duncan

[Venkatesh BL]

Use top or glance to find the cpu usage of the processes.

[Hakki Aydin Ucar]

most of CPU is in use by system process(es)
it is slightly normal %90 percent CPU ( my customer server sometime > %90 CPU but not all day long!)
I wanted to state, if this situation happens temporarily not prolonged it is not too bad. But if this happens due to bottlenecks OR got stuck processes then need to be solved.

you can check it lsof OR fuser tools..

[Hakki Aydin Ucar]

and besides , of course top and glance to see which process consumes too much CPU as Venkatesh said before..

[NDurand]

i have checked the system for several hours and it stay stucked at 100%.

The process which are consumming are java process with only one thread and doing a lot of Vfault (found with glance)

[Duncan Edmonstone]

So this java program - who wrote it and what does it do? Has it *always* behaved like this, or just started to recently - who setup the startup parameters for the java program? have they been changed recently?

Perf tuning for java apps is quite a specialist area about which I know very little... you probably need to go back to the devlopers of the application and discuss with them... if you feel up to it, you could do worse than start reading this manual:

http://docs.hp.com/en/5992-4687/index.html

HTH

Duncan

[Dennis Handly]

>The process which are consuming are java process with only one thread and doing a lot of Vfault

Hmm, with that much vfaults, I wonder why no paging? Are your processes short lived as Duncan suggested? Perhaps you are getting too many TLB faults?

You may want to renice your unmutual java process and upon restart, give it less memory.

Can you provide "swapinfo -tam" output.

[NDurand]

here's the swapinfo -tam output

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 6144 0 6144 0% 0 - 1 /dev/vg00/lvol2
reserve - 6144 -6144
memory 32737 23754 8983 73%
total 38881 29898 8983 77% - 0 -

[Dennis Handly]

>here's the swapinfo -tam output
dev 6144 0 6144 0%
reserve - 6144 -6144
memory 32737 23754 8983 73%
total 38881 29898 8983 77%

You have reserved all of your device swap but haven't used it.
You have about 8.9 Gb free. Are these values the same during your "huge number of page faults"?

[NDurand]

Hi,

Yes, these values was mesured during the high page faults.

Rgds

[Don Morris_1]

Well, I had a much longer reply and the forum submittal ate it. Here's the quicker version:

Assuming the vmstat output is also from the high Page Fault period, the low at field and higher system call field along with the original high Faults but low KB Paged In stats makes me suspect that the java process may be doing some sort of page release/invalidation and re-instantiation activity (perhaps as part of Java GC work? And the change may be in the object creation/utilization in the workload?). The data as given certainly indicates that the application is frequently calling down to the kernel and that the pages generated are Demand fill pages, not anything to do with paging in or out.

tusc might give some indication if you sample the application over a period to see what syscalls it is making.

Another alternative would be to pick up Caliper (http://www.hp.com/go/caliper) and point it at the application, being sure to profile the kernel as well as the application to get an idea of what's going on in the system faults. I think this would be more likely to produce results.

[NDurand]

Thanks a lot for your help. I will try to profile the application beginning by the analyze of the garbage collector of the jvm to see if there is lot of GC or not.

Rgds

[Duncan Edmonstone]

Don,

> Well, I had a much longer reply and the forum submittal ate it. Here's the quicker version

Yes that regularly happens to me - many times my beautifully crafted essays are eaten by the forum submission monster and my follow up post is just "please do xyz" without any explantion why!

Still from my point of view, at least your (no doubt stinging) critique of my horribly vague ramblings on the inner workings of VM kernel voodoo will never see the light of day and I can go on pretending I know what I'm talking about!

Duncan