very high page out rate on 11.31

[Vandee Chimcherd]

Dear Support

I generate report from zamboni, it show following message
-------------------------------------------
MEMORY
This system has 143 gigabytes of memory installed. Memory utilization averaged 82.9% during the prime shift. Memory pageout rate was 718.6 pages per second. Memory Swapout rate was 0.0. Average memory queue length was 0.2.
CONCLUSION: A pageout rate greater than 3.0 indicates a serious shortage of memory.
-----------------------------------------


I wonder why memory is shortage but mem util is about 82%? (from my understanding, it mem util should be higher) Does anyone know this could be any problem? Should I upgrade memory?

[Venkatesh BL]

What is the value of 'filecache_min' and 'filecache_max' kctune tunables?

What is the nature of your workload?

[Matti_Kurkela]

As it says, 82.9 % is the _average_ memory utilization over the entire prime shift (several hours maybe?).

You might want to look at _peak_ memory utilization, but from the pageout rate I'd guess that the peak memory utilization was a full 100.0% at some point.

If the pageout rate of 718.6 pages per second is likewise an _average_ over the entire prime shift, this machine has a really, really serious shortage of memory. Even if it is just a peak rate, it may be bad enough.

What do the users say? Did the application(s) work OK over the entire time, or was there slowness or crashes?

MK

[Hein van den Heuvel]

Thanks for providing the attachment. The prevents a lot of back-and-forward questions and anwers.

But it raises one major question... why was such a short, odd, sample window chosen?

Does the window 15:00 - 01:00 next day cover the critical processing window for the system in question?
The reporting tool considers (is told) that 08:00 - 17:00 is the prime production window. Well, there is insufficient data available in that window to be able to trust any recommendation.

I think the report is confused, and not to be trusted. For example it writes:
"User mode processing used 25.4% and System mode used 18.8%. The User to System ratio of 1.3:1 is within the guideline that suggests, for best performance, User mode CPU should
be at least three times System mode CPU."
In which universe is 1.3 more than 3.0 ?

I'm with Matti... put aside the report, and do some general observation. Does the phone ring / are pages goign off? Are the end users happy or unhappy? When you look with 'top' or glance', during what you believe to be a busy hour, does it 'look' reasonable? What stands out?

I suspect that the reporting is broken, probably triggered by a bad, and short, reporting period. Try again with a week worth of data, and adjust (if possible!) the rules (hot window) to the reality.

hth,
Hein van den Heuvel
HvdH Performance Consulting.

[Rita C Workman]

Check your system parms, with 143 GB of memory (wow...) you want to be sure how your parms were affected.
On 11.31 IA there are so many new parms and changes to old ones, we are all still adapting to them.

The old dbc_max & dbc_min_% parms that Vanketesh mentioned changed to filecache_min & filecach_max. But it seems the default value of 50 for max followed. They are supposed to be more properly dynamic, but...... why not try and setting the values manually to reduce that 50% on max and see if you don't see an improvement.

Here's another to watch out for...ninode and vx_ninode.
Ninode can get auto-set to a large number depending on the values of other parms. I tend to set this manually to reduce the size of the table that takes up space in cache. It's a roll-over table so the ninodes just keep recycling. 8192 is very generous. You can monitor with sar -v to see where you need to be.
The vx_ninode is another. Set to 0 it allows the system to control this table as well. You may want to leave it there, or you may try playing with it and setting the value manually. Anything from 40,000 to 100,000 is very generous.
==========================================

Now here's another thought...can you tell us..

What are you running on your box - Oracle, SAP?
What about any Application add-on that the system needs? Is it running on the same HPUX box, or is it running on Windows and interfacing back to HPUX?
< We once had a third party software using Oracle, with an Apps server on Windows that hammered the HPUX box with hundreds-thousands of worthless poll calls/second. Spiked the CPU.>
Have you run any packet capturing during peaks to see what's going on? tusc is a good tool to use.

Just some thoughts,
Rita