99% Swap utilization

[Anoop P_2]

Hi Folks,

I have a system which reports near 100% memory and swap itil by glance:

CPU Util SA | 3% 3% 3%
Disk Util | 0% 0% 0%
Mem Util S SU UB B | 92% 92% 92%
Swap Util U UR R | 99% 99% 99%


swapinfo however says that the whole of device swap is untouched:

Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2097152 0 2097152 0% 0 - 1 /dev/vg00/lvol2
reserve - 2097152 -2097152
memory 3194712 3132628 62084 98%
total 5291864 5229780 62084 99% - 0 -

Could some one tell me why the devide swap is not touched and still the system says that the swap util is near 100%?

Thanks and Merry Christmas in advance!

Anoop

[RAC_1]

when you see a "memory" line in swapinfo output, it means that you have swapmem_on parameter set to 1.

What does this parameter do?? It sets the swap space configured value as follows.

"actual swap space configured + 75 % of RAM"

This is only for kernel calculation purpose. The use of this parameter is that the systems with large RAMs (this varies from an individual to another. I see at 32GB as high RAM), it can maange with low primary swap.
So in you case it depends on how much RAM you have.

If you see double digits pageouts in vmstat (po column) contineously, then system is performing badly. And on your system if memory usage is 100%, you performnace is going to be bad on account of memory bottleneck. glance -m would be very helpful. If system is swapping, what you need is additional RAM and not swap. Adding more swap is not going to help.

[Frank de Vries]

Ver strange indeed,
I wonder if we can trust glance ?

It also pays to check which jobs (if any)
are heavy on the ram so that swap becomes
inevitable. Do you notice any significant
performance degradation ?

You may have looping processes or
some heavy i/o going on.

Look with top which jobs are most hungry.

Cross-check the statst given by glance
with sar -u / sar -q
to get the bigger picture.

[RAC_1]

Also check what is buffer cahe setting. May times default cache settings are used. This results in more memory given system and less to users. check what you use?? dynamic cache (use of dbc_max_pct and dbc_min_pct)or static?? (use of bufpages/nbuf parameters)

[Don Morris_1]

Not strange at all.

What you've got is a workload that's doing huge malloc() / mmap() / whatnot calls and not really using what it asks for all that much. This consumes a lot of virtual address space, but not enough RAM to cause actual paging to occur.

Note that your "reserve" line is exactly equal to your device swap above. And your memory line is almost completely gone. HP-UX requires memory allocations by default (there are exceptions with flags/chatr/etc.) to reserve swap equivalent to the virtual address space consumed. malloc() for 2Mb -- reserve 2Mb of swap. This doesn't meant that the 2Mb has a particular location on disk yet, just that the system knows that 2Mb of swap is spoken for. In your case, the whole system is spoken for, and new allocation requests are going to start seeing ENOMEM failures.

This kind of thing is what maxdsiz{_64bit}, maxssiz{_64bit} are designed for. They're fence lines used to rein in your applications before they consume all the swap (if this is one or two over-consuming apps). If Glance is able to run enough to tell you -- check the Virtual Size of the running processes and see where the big users are... and if it is all in Stack or Private Data. Lowering the corresponding tunable will cause that application to fail the allocation sooner (how the application handles that failure is up to it... but for Stack growth failure, it will terminate).

Another possibility is that no one application is consuming the virtual address space -- you just have so many that they're nibbling you to death. Lowering nproc to restrict the number of running processes would help with that.

Of course, you may want the workload that's running -- and you may be okay with the allocations they're doing. If so -- you're going to have to add more swap or memory. Given that your actual RAM use seems to be sufficient and disks are pretty cheap -- I'd add swap first. Filesystem / secondary disk swap would be easiest. If *that* gets eaten up quickly, you really need to track down what's consuming all the virtual address space and deal with it.

[Steven E. Protter]

Shalom Anoop,

Probably time to buy some memory.

http://www.hpux.ws/system.perf.sh

A good set of monitor scripts.

SEP

[Ted Buis]

You have run out of vitual address space. You need to increase it by either buying more RAM, reducing dynamic buffer cache, creating more physical device swap space, or all of the above. First look at how much dynamic buffer cache you have and see if it can be reduced. Typically more than 1GB of RAM for dynamic buffer cache yields very diminishing returns, but many times I see the default value of 50%. So do a kmtune and look at max_dbc_pct and multiply that percentage by your total RAM to see just how much you have.

[john kingsley]

Check your buffer cache settings (dbc_max_pct/dbc_min_pct). By default, HPUX sets the dbc_max_pct to 30% which is way too high. This means they you may not even be using a third of your memory. In most cases, you can get by with dropping dbc_max_pct to 5-10%.

[Anoop P_2]

Hi.
Thanks to all who responded.
dbc_max_pct was already set as low as 15%.
I found that Oracle was reserving a lot os space, that it actually requires, and hence arranged for this setting to be lowered. Now the system is fine.
Anoop