vmstat : Page allocate failures (including 4K) means memory/swap shortage?

[Roman Guoussev-Donskoi]

I have following statistics from "vmstat -s" output from client's site that is experiencing some problems running their business apps.

Output contains quite a few "page allocation" failures for all page sizes (including 4K) and 300K page outs.

Can I conclude from this result that they have(or have had since the last "vmstat -z") memory (and possibly also swap) shortage on their server?

"vmstat -s" output is attached

Thank you.

[A. Clay Stephenson · Accepted solution]

It appears that this box has run out of virtual memory at least a few times which would mean that you need more memory and/or swapspace. However, adding more memory/swap may only delay the problems if you are dealing with memory leaks -- and I suspect that is the real problem. I would not be a very happy camper if someone sold me (or told me to buy) several tens of thousands of dollars worth of memory and/or disk and the only effect was that instead of crashing every day, I now crash every other day.

You need to do more digging and determine if you have processes growing significantly in size over time.

[Sridhar Bhaskarla]

Hi,

Better tools are 'swapinfo -t' and 'vmstat 2 20' over the time. If the system is running short of memory and is paging out, you should see a non-zero number in KB Used or PCT Used in 'swapinfo -dft' output and the po column in 'vmstat' output is constantly a double digit number.

To find out the leaks, you can run the command

UNIX95= ps -e -o 'vsz pid args' |sort -n

in frequent intervals and note the processes that have vsz (first column) growing constantly.

May be the system's buffer cache is left at default which is 50%. Check with kmtune -l -q dbc_max_pct, You can lower it to get around 400MB. It's supposed to be dynamically changed with the utilization but I found it doing it better only with 11i.

Glance\gpm can be an excellent tool to have a look the resources.

-Sri

[Roman Guoussev-Donskoi]

Thank you for both replies!
For political/administrative reasons before we proceed with further investigation on customer site I had to submit preliminary report based only on very limited info.
Usually I go with tools you both mentioned (like glance advisor, vmstat with interval or pstat interface to check processes memory) to verify what/how much is using memory.
Never really needed "vmstat -s" before.
But this practically is the only output I have on memory usage at this point.
Therefore needed second opinion on this.
And you responses have been most helpful.
Thanks again!

[Bill Hassell]

Page outs are all you need right now. However, vmstat -s can be misleading if you don't know when vmstat -z was last run. These stats (page out) would be terrible if the measurement period was 1 day, but may be normal if the period is one month. Assuming that the measurement period is just a few days, the page out value indicates a severe shortage of RAM. You can keep things plodding along with more swap space but page outs can degrade application performance as much as 50 to 100 times slower. In a modern server, trying to run with small RAM just doesn't make sense. It usually cripples otherwise high-performance programs.

Verify the running page out rate with vmstat or Glance. If the rates are single digits for a 5 second measurement period, you're probably OK. 2 digits is a warning and 3 or more digits indicates severe RAM shortage.