Memory allocation on HPUX 11.11

[AshishJain_USA]

I have a system with HPUX 11.11, 96 GB RAM .

I am constantly finding that the swapinfo is showing greater than 90% utilisation. The problem may be that the swapinfo shows only 74 GB memory.

#-> swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol2
dev 2048 0 2048 0% 0 - 0 /dev/vg00/lvol9
reserve - 10240 -10240
memory 76288 71570 4718 94%
total 86528 81810 4718 95% - 0 -

Does someone know how to interpret this ?

#-> dmesg|grep Physical
Physical: 100611072 Kbytes, lockable: 77964672 Kbytes, available: 89487288 Kbytes

Can someone also help me in understanding the "lockable" and "available" in the above dmesg output ?

[Steven E. Protter]

Shalom,

You need more information.

vmstat

Will show you if you are paging to swap or not.

What concerns me about the ouptut above is that no swap is even reserved. That tells me your system is not using or needing swap at all.

With 96 GB of RAM, I can see why.

90% memory utilization is meaningless, because the OS simply allocates as much as it can to various pools for use by processes. On normal systems that can be as high as 99%.

I'll leave the dmesg question to someone else.

SEP

[AshishJain_USA]

Steve

The vmstat o/p is here

#-> vmstat 1 10
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 5 0 3881301 6853607 140 48 1 0 0 0 0 7901 46406 3933 8 3 88
1 5 0 3881301 6853549 83 19 0 0 0 0 0 8104 47153 4459 5 2 93
3 2 0 3879875 6853549 66 15 0 0 0 0 0 8205 43697 4474 3 3 94
3 2 0 3879875 6853549 52 12 0 0 0 0 0 8176 46278 4522 6 2 92
3 2 0 3879875 6853549 42 9 0 0 0 0 0 8066 43731 4316 4 2 94
3 2 0 3879875 6853549 66 22 0 0 0 0 0 8132 41956 4398 4 2 94
3 2 0 3879875 6853549 53 17 0 0 0 0 0 8077 44488 4370 6 3 91
3 2 0 3879875 6853549 43 13 0 0 0 0 0 8187 39784 4379 3 3 94
3 2 0 3879875 6853305 152 32 0 0 0 0 0 8220 45219 4433 7 3 90
3 2 0 3879875 6853549 122 25 0 0 0 0 0 8367 64961 4548 3 3 94

[A. Clay Stephenson]

Since your pageouts are zero, you are under no memory presuure. The "memory" line in swapinfo doesn't mean what you think it means. You only see this line if swapmem_on=1 (pseudoswap enabled) as it should be in your case. However, you should really increase your swap space to 24GB (25% of 96GB) so that you can maximize your virtual address space. Pseudoswap counts 75% of your physical memory as swap space for virtual memory availability purposes.

The kernel tunable unlockable_mem defines the minimum amount of memory that is always available for system overhead; increasing unlockable_mem decreases lockable memory.

You should note that lockable memory is available for normal virtual memory allocation except when it is locked.

[Don Morris_1]

Ok... all your physical swap is reserved (8192 + 2048 = 10240, which is your reservation line).

You've also reserved 94% of the memory available for pseudo-swap. [Pseudo-swap is an accounting trick used in the kernel for configurations such as yours where the amount of RAM is larger than the amount of swap. A percentage of the RAM (usually about 75%, hence why this line is 74Gb not 96Gb..) is allowed to be considered as swapping to itself -- and hence counts as swap reservation.

Swap reservation is consumed on HP-UX in general whenever virtual address space is consumed. (Unlike Linux, the default policy in HP-UX is that you must set aside enough space to handle swapping out the memory you're creating up front... this allows the system to always be able to swap out any existing objects, with the failure due to insufficient swap at the point of the object creation (malloc/mmap/etc.). Lazy swap reservation is an option on some HP-UX objects (such as mmap, etc.) -- which waits to reserve the swap until actual physical memory is used to back the object (so you can get a failure in a page fault because of inadequate swap).

So what does all that mean to you? If your system is working properly and is running what you consider your maximum workload -- congratulations, you've scaled your workload to your box well, and you have about 4Gb of virtual address space remaining for additional user space consumption. If you're not at peak load, however -- I'd be a little worried that you'll exhaust that 4Gb (say you start another 256 programs, each of which need 16Mb of virtual address space... you'll exhaust the swap reservation and no new virtual objects can be created... malloc/mmap calls will start failing, as will fork/exec/etc.). If this is the case, you can add additional FS or device swap so you have more reservation space to consume.

On your second question, physical memory (RAM) backing a virtual object is usually fair game for the kernel to push to swap if it needs to. Performance sensitive processes don't want to take the chance of having the cost of a page fault while in timing-critical parts of their code -- so they can ask the OS (if run with sufficient priviledge) to pre-construct the RAM they require to back a given virtual range and not allow that memory to be paged out until they are done with it. This is done with the plock or mlock interfaces -- and unsurprisingly, this is referred to as locked memory (because it is locked in RAM and can not be swapped).
Note that on HP-UX, most (close to all) kernel memory is implicity locked (and accounted for as such) because the kernel is not swappable.

The system can't allow *all* of RAM to be locked in general - because then the system would never be able to recover from memory pressure at all. The unlockable_mem tunable allows some control over how close that margin is, by defining how much RAM needs to be held back as swappable (and thus invalid for locking requests.. note that this is an amount, not a specific range).

Hence the dmesg output, telling you how much total RAM is on the box, how much the system considers valid for locking requests - and how much is left at this point [basically, how much the kernel has already consumed to boot to this point].

Hopefully that rambling screed helps.

[Don Morris_1]

Sheesh, my mind is slow this morning.

I forgot to mention - there are kernel tunables to allow the administrator to set fence posts to prevent ill-behaved applications from consuming too much virtual address space (by large malloc requests that are leaked/never touched, etc.). These are the maxtsiz, maxdsiz, maxssiz (and their corresponding _64bit versions on 64-bit systems... note that due to the way these limits are inherited, a 64-bit tunable value below the 32-bit tunable overrides the 32-bit one..). These get translated as resource limits (man getrlimit, setrlimit) in the processes, so you can also use ulimit to lower the limits further.. the tunables set what is known as the "hard" limits.

If you suspect you have ill-behaved applications, Glance or just judicious use of ps / pstat (what you're looking for is large Virtual Set Size [VSS] usage with very little RSS [physical memory usage]) may give you ideas. A large VSS isn't always a problem - but if you monitor/log it and a process shows a constantly increasing VSS over time it may have a memory leak -- and that can be eating up your swap reservations.

In a more complex fashion, the SysV shmem tunables also play into this (if you can only create N segments of size M bytes you'll have a maximum swap reservation cost of (N * M)), but runaway SysV usage is less common - so I'd only go there if you have to.

I'm sure others here can post helpful hints and tips on using the tools to check on your virtual address consumption if you need more information.