vmstat and memory thresholds

[Mihai Matei]

Hi,
I ran a vmstat while stresing the system, but the values shown don't quite match the ones that 'should be' according to HP-UX documents.
Please see the attached document for vmstat output.
My questions:
1.If deactivations occur only when free memory falls below minfree, why is the so column non-zero, even above that threshold?
2.What's the difference between the 'w' column (runnable or short sleeper, but swapped (deactivated)) and the 'so' column (deactivations). Shouldn't they be the same?

[Chris Vail]

The first entry from vmstat records all entries since you last booted the system. This merely means that, since the last reboot, 5788 processes were swapped out, and back in again. The first line of vmstat is almost useless for this reason. It is the second and subsequent lines of the vmstat output that indicate your system is running out of RAM.
You didn't mention what kind of system this is, how many processors and such, but there's little doubt in my mind that it is out of memory. The PI and PO columns indicate this.
Large amounts of PI (page in) is common even in well-tuned Oracle installations. This merely means that there are a lot of stored procedures that had been paged (actually aged) out to disk because they weren't in use. But you also show large numbers in the PO column--indicating that your processes have run out of memory and are using paging space. This massively slows down the system. The best (really only) way to fix this is to add memory. Faster processors won't hurt, but even they'll be starved for memory.
You also have a lot of processes blocked waiting on I/O. Usually this is because multiple processes are trying to access the same I/O channel for the same chunk of data at the same time. This one is a lot harder to fix. Speeding up I/O is one way to fix this (faster disks, more cache, more I/O channels). But were I you, I wouldn't do anything until the RAM in that computer was doubled or quadrupled.

Chris

[Michael Steele_2]

To the best of my knowledge (* which always clouds up under virtual memory discussions *) 'w' under 'proc' stands for runnable but swapped out processes while <20 seconds indicates an active page in memory 'if' the process associated to it is 'runable'. (* runable and complete in <20 seconds = active page in memory *)

And an active page in memory can not have a sleeping process associated to it.

A swapped out process that changes to a 'sleep' state is worse than a 'runable' process that's been swapped out so I believe 'w' is trying to shed light upon this.

(* Sleeping processes are waiting on a signal from I/O before continuing. *)

'deactivated' means the process has either run to conclusion and deactivated itself, or, the process doesn't have a high enough priority to continue running and the entire process is swapped out. When this occurs its called thrashing or disastrous paging or disastrous swapping. I believe all three are interchangeable.

'minfree' signals when to start deactivating processes.

'so' indicates processes swapped out at 1 sec intervals.

'w' is a report of different process states.

So "...why is the 'so' column non-zero...Shouldn't they be the same?...", the 'so' number is only for the last second.

'minfree' is also a tunable kernal parameter:

http://docs.hp.com/cgi-bin/otsearch/getfile?id=/hpux/onlinedocs/os/11.0/tuningwp.html&searchterms=desfree|lotsfree&queryid=20030414-06091

[Michael Steele_2]

"..while stresing the system,..."

If you have a problem and this report is accurate then you have two choices, reduce the number of processes or buy more memory.

[Mihai Matei]

Chris,
I am aware of the accumulated counters on the first line of vmstat.
I don't doubt that the system has a memory bottleneck, and I guess the 'blocked on IO' column reflects processes waiting for VM.

Michael,
Does that mean that the 'so' column counts the processes that finished normally, in addition to the ones 'swapped out'? If this is the case, that would answer my question about why have deactivations occured, even though free memory is above minfree.

2 new questions:
1. I got the values for lotsfree, desfree and minfree using pstat() functions. Is there any other way to retrieve those values?
2. those parameters are tunable starting in HP-UX 11, although not recommended. How exactly does one change them? (they don't show up under 'Configurable Parameters' in sam)

[Mihai Matei]

please help, see the last 2 questions above.

[Bill Hassell]

Being able to see the memory management values (lotsfree, desfree and minfree) won't help. Your pageout rate is way too high (3 digits) and there is a simple fix: buy a *LOT* more memory. If you are running Oracle, start with 4Gb and work up from there, especially if you run multiple instances, and/or 64bit versions of Oracle. There are a few tunables in the kernel but you'll need a couple of weeks and about 3 dozen reboots to adjust these values, only to find that the net effect will be minimal (HP-UX does a good job of running lots of processes in very small memory).

Your system has run out of gas (memory) and no matter what management says about $$$ for more gas, you can't run very far without some petrol in your system. Unfortunately, a lot of docs about recommended memory for Oracle (and servers in general) are for minimums (system will boot up) but these docs fail to say: "Unacceptable performance degradation". Page out rates and deep disk I/O queues botn indicate an undersized server.

[Mihai Matei]

The system those bottlenecks occured on is not a production machine, it's only a test machine, and I'm interested in the answer to the last 2 questions as a 'general hpux knowledge', not from 'improving the performance' point of view. I should have said this from the beginning...
So, anybody knows about the values of memory thresholds minfree, desfree, lotsfree and how can they be changed?

Thank you.

[Michael Steele_2]

Regarding "...So, anybody knows about the values of memory thresholds minfree, desfree, lotsfree and how can they be changed?..."

http://docs.hp.com/hpux/onlinedocs/os/11.0/tuningwp.html#lotsdes

"...On a system with up to 2GB of memory:

lotsfree no larger that 8192 (32MB)
desfree no larger than 1024 (4MB)
minfree no larger than 256 (1MB)
on a system with 2GB to 8GB of memory:

lotsfree no larger than 16384 (64MB)
desfree no larger than 3072 (12MB)
minfree no larger than 1280 (5MB)
on a system with a whole group of memory:

lotsfree 131072
desfree 32768
minfree 8192..."