hp-ux swap utilization so high >90%

[joseph123]

hi guys

i have a question regarding swap.
i have read some infomation from book or ITRC...but, i still confuse should i increase swap.
==================
# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20480 0 20480 0% 0 - 1 /dev/vg00/lvol2
dev 10240 0 10240 0% 0 - 1 /dev/vg00/lvol9
reserve - 30720 -30720
memory 40925 35298 5627 86%
total 71645 66018 5627 92% - 0 -
==========================
this server is a SAP system.
i use a tool named sappfpar which provided by sap. the result is "the worst case requirement need 37G". (current swap 30G)
from sap point of view, i should increase swap to my system.

but, from OS point of view, we can see there is no memory page out occuring.

so, i'm confuse now. should i increase the swap and is it possible to identify the root cause why the swap utilization is so high?

[Steven E. Protter · Accepted solution]

Shalom,

When a process starts, it reserves swap. In case it needs to be swapped out.

You are good so long as you are not paging.

vmstat will tell you if you are paging.

If you start paging, increasing swap will let you page and slow you down. If you start paging the solution is either to reduce memory demand or increase memory supply.

SEP

[Patrick Wallek]

>>but, from OS point of view, we can see there is no memory page out occuring.

Correct. Your device swap usage percent is 0. That is very good.

>>should i increase the swap

I don't see any reason to do this from the information provided.

>>is it possible to identify the root cause why the swap utilization is so high

There is no swap utilization at this point, other than for reservation purposes.

[joseph123]

Shalom,
so ~ as long as swap usage percent is 0, even thought swap utilization become 100% , then the system is fine ?

but, why our swaw usage is 0, it never used ? and what does reserve mean?

i saw the total used is reserve used + memory used = 30720 + 35298 = 66018

[Don Morris_1]

A system going to 100% swap used+reserved means that new virtual memory allocations will be denied. In practice, that will mean you'll start getting ENOMEM from calls like malloc(), fork(), etc. I wouldn't encourage you to be in that state for very long or to explore it. (Not being in that state is the reason for tunables like maxdsiz/maxssiz in the first place).

Reserved means just that -- these are swap resources which have been claimed for _possible future use_ by a client. HP-UX (like other Enterprise OS's) requires that a virtual memory object upon creation or grow be given the corresponding swap resources so that if memory pressure does occur -- everything can find a place on the swap areas. So... while you don't know exactly what disk block or FS entry your process has, you know it is guaranteed X amount of swap. Correspondingly, when no more swap is available for reservations -- no new virtual address space can be consumed, and that's why the system must start to fail these requests.

There's a little more detail in `man 1M swapinfo`, but that's the basics.

Memory swap is an accounting trick that allows the system to include part of physical memory in the swap reservation totals. The idea is that if pressure occurs, some things will be pushed out to disk/FS swap -- but when those are full, the algorithm can "swap" the page to itself... therefore some part of RAM counts in the reservation since that amount will have a place to go (where it already is). It isn't all of RAM because you want to leave the virtual memory subsystem some space to move things around to relieve the pressure.

So getting back to the real question -- you can create around 5Gb of additional virtual objects (new processes, malloc within processes, etc.) If that's enough -- don't change anything. If you're running below your expected maximum workload, however -- and if the SAP tool is working off of the maximum workload expectation, it is telling you to add more swap so it will be there to create the additional processes or virtual memory objects you'll need in the future. I don't know SAP internals so I can't tell you if it is right.

[joseph123]

this is my current status
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20480 5 20475 0% 0 - 1 /dev/vg00/lvol2
dev 10240 5 10235 0% 0 - 1 /dev/vg00/lvol9
reserve - 30710 -30710
memory 40925 34562 6363 84%
total 71645 65282 6363 91% - 0 -

i found total used is 5+5+30710+34562 = 65282
if i used full swap will become
20480+ 10240+ 30710+ 34562 = 95992

so ~ the total PCT USEd will become 95992/71645 = 1.3398....
exceed 100% , rite ?

reserve is a system parameter ?
how can i setup it, can i reduce it ?

[Don Morris_1]

No, the reserve is the amount from the above dev/FS lines which is set aside in case of swap.

When/if actual swap occurs, it would shift resources from the Reserved state (accounted for but not specific blocks, no data on dev/FS) to the Used state (accounted for, known specific locations with data on dev/FS).

So a 100% consumption assuming only reservation would look like:

5 + 5 + 30710 + 40925 = 71645

Which, as you'll notice is only different in your memory line. (As 30710 shows you being the sum of 20480 + 10240 [total] minus 5 + 5 [used], you've reserved all of your device/FS swap space. Only memory swap is available at this point for new reservations).

A 100% Used scenario would look like: 20480 + 10240 + 0 + 40925 = 71645. (All reserved swap becomes Used).

As I hope is apparent from the above discussion -- reserve is not a parameter, it is a metric of the workload and represents the virtual address space backed by your device/FS which has not yet had to be swapped out. You can only reduce it by reducing the virtual address space footprint of your workload (run less processes, use fenceline tunables like maxdsiz, maxssiz, shmmax, shmmni, shmseg to enforce smaller virtual objects or fewer objects, etc. and perhaps through tuning in the application space [smaller data set, limiting parameters, whatnot]).

[Dennis Handly]

>I found total used is 5+5+30710+34562 = 65282

The total is given by the "total" line. You don't have to compute it. ;-)

[joseph123]

hi guys
i'm understand the formula now.

but, i still confuse
reserve is not a parameter, it is a metric of the workload and represents the virtual address space backed by your device/FS which has not yet had to be swapped out.

in this case
====================
[root@PRDAP3]/usr/sap/PR1/SYS/profile # swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 3931 165 96% 0 - 1 /dev/vg00/lvol2
dev 36000 7387 28613 21% 0 - 1 /dev/vg00/lvol9
reserve - 3729 -3729
memory 8180 1662 6518 20%
total 48276 16709 31567 35% - 0 -
====================
swap has 40G and used approx 11G
so ~it should reserve 29G
why in this case it juse reserve 3729M ??

and...
with regards virtual address space
is it a system parameter ? can i setup it ?

[Don Morris_1]

Presumably because the current running workload doesn't need more than it reserved.

Virtual address space isn't a system parameter -- it is a measure of the virtual size of all the user processes across the entire system.

For a very simple example, let's say we have 3 processes running in total.

Process A uses: 5 Mb for Text, 15 Mb for Data, 2 Mb for Stack, 30 Mb for Shared data (which is shared with B, but not with C). All counts are Virtual, not physical (i.e. they represent the total size from malloc()/shmget()/mmap() [or just the system setting things up like Stack]).

Process B uses: 3 Mb for Text, 25 Mb for Data, 1 Mb for Stack, 30 Mb for shared with A, 10 Mb for shared with C.

Process C uses: 10 Mb for Text, 200 Mb for Data, 2 Mb for Stack, 10 Mb for shared (with B).

The total virtual address space in use is the virtual size of A (52Mb) plus B (69Mb) plus C (222Mb) minus the duplicates from cross-process sharing (the 30 Mb we counted in B that was counted in A, the 10 Mb we counted in C that was in B -- so 40Mb) or 303Mb.

Swap is required to back all virtual objects which are not backed by other means (which almost always just means "not backed by a File"). So in the above example [assuming the Shared objects are anonymous memory and not shared file mmap for simplicity], we can remove the Text objects from the total.

So our expectation is that the virtual space for each process which needs to be backed by swap is:

A: 15 (Data) + 2 (Stack) + 30 (Shared)
B: 25 (Data) + 1 (Stack) + 30 + 10 (Shared)
C: 200 (Data) + 2 (Stack) + 10 (Shared)

Again removing duplicates -- we see that the expected total swap consumption of the system would be:

15 (Data A) + 25 (Data B) + 200 (Data C) + 2 (Stack A) + 1 (Stack B) + 2 (Stack C) + 30 (Shared A/B) + 10 (Shared B/C) = 285Mb.

This is the virtual address space consumption of the System. (Well, there's some interaction with the kernel allocations -- but I'm trying to keep this simple for explanatory purposes...)

The total virtual address space available (for non-file backed objects) is the Total AVAIL on the swap line.

Whether that is Reserved or Used depends on if _physical_ memory got low enough such that the system had to page out memory. [Your output showing multiple Gb USED suggests there was memory pressure on this box in the past at least.] There isn't more Reserved here because no process needs more. If one of these three processes requests more virtual address space (through malloc or other such methods), the corresponding swap reservation request is made at that time, increasing the amount reserved.

By the way, while it hasn't been updated in some time (the tunable statements are very stale for one thing) -- you may find reading the Memory Management white paper illuminating for the basic concepts of HP-UX virtual memory management:
http://docs.hp.com/en/1218/mem_mgt.html