topic Re: hp-ux swap utilization so high >90% in Operating System - HP-UX

hp-ux swap utilization so high >90%

joseph123 — Tue, 28 Jul 2009 15:22:20 GMT

hi guys

i have a question regarding swap.
i have read some infomation from book or ITRC...but, i still confuse should i increase swap.
==================
# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20480 0 20480 0% 0 - 1 /dev/vg00/lvol2
dev 10240 0 10240 0% 0 - 1 /dev/vg00/lvol9
reserve - 30720 -30720
memory 40925 35298 5627 86%
total 71645 66018 5627 92% - 0 -
==========================
this server is a SAP system.
i use a tool named sappfpar which provided by sap. the result is "the worst case requirement need 37G". (current swap 30G)
from sap point of view, i should increase swap to my system.

but, from OS point of view, we can see there is no memory page out occuring.

so, i'm confuse now. should i increase the swap and is it possible to identify the root cause why the swap utilization is so high?

Re: hp-ux swap utilization so high >90%

Steven E. Protter — Tue, 28 Jul 2009 15:25:30 GMT

Shalom,

When a process starts, it reserves swap. In case it needs to be swapped out.

You are good so long as you are not paging.

vmstat will tell you if you are paging.

If you start paging, increasing swap will let you page and slow you down. If you start paging the solution is either to reduce memory demand or increase memory supply.

SEP

Re: hp-ux swap utilization so high >90%

Patrick Wallek — Tue, 28 Jul 2009 15:28:07 GMT

>>but, from OS point of view, we can see there is no memory page out occuring.

Correct. Your device swap usage percent is 0. That is very good.

>>should i increase the swap

I don't see any reason to do this from the information provided.

>>is it possible to identify the root cause why the swap utilization is so high

There is no swap utilization at this point, other than for reservation purposes.

Re: hp-ux swap utilization so high >90%

joseph123 — Tue, 28 Jul 2009 16:04:58 GMT

Shalom,
so ~ as long as swap usage percent is 0, even thought swap utilization become 100% , then the system is fine ?

but, why our swaw usage is 0, it never used ? and what does reserve mean?

i saw the total used is reserve used + memory used = 30720 + 35298 = 66018

Re: hp-ux swap utilization so high >90%

Don Morris_1 — Tue, 28 Jul 2009 16:19:50 GMT

A system going to 100% swap used+reserved means that new virtual memory allocations will be denied. In practice, that will mean you'll start getting ENOMEM from calls like malloc(), fork(), etc. I wouldn't encourage you to be in that state for very long or to explore it. (Not being in that state is the reason for tunables like maxdsiz/maxssiz in the first place).

Reserved means just that -- these are swap resources which have been claimed for _possible future use_ by a client. HP-UX (like other Enterprise OS's) requires that a virtual memory object upon creation or grow be given the corresponding swap resources so that if memory pressure does occur -- everything can find a place on the swap areas. So... while you don't know exactly what disk block or FS entry your process has, you know it is guaranteed X amount of swap. Correspondingly, when no more swap is available for reservations -- no new virtual address space can be consumed, and that's why the system must start to fail these requests.

There's a little more detail in `man 1M swapinfo`, but that's the basics.

Memory swap is an accounting trick that allows the system to include part of physical memory in the swap reservation totals. The idea is that if pressure occurs, some things will be pushed out to disk/FS swap -- but when those are full, the algorithm can "swap" the page to itself... therefore some part of RAM counts in the reservation since that amount will have a place to go (where it already is). It isn't all of RAM because you want to leave the virtual memory subsystem some space to move things around to relieve the pressure.

So getting back to the real question -- you can create around 5Gb of additional virtual objects (new processes, malloc within processes, etc.) If that's enough -- don't change anything. If you're running below your expected maximum workload, however -- and if the SAP tool is working off of the maximum workload expectation, it is telling you to add more swap so it will be there to create the additional processes or virtual memory objects you'll need in the future. I don't know SAP internals so I can't tell you if it is right.

Re: hp-ux swap utilization so high >90%

joseph123 — Thu, 30 Jul 2009 16:32:59 GMT

this is my current status
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20480 5 20475 0% 0 - 1 /dev/vg00/lvol2
dev 10240 5 10235 0% 0 - 1 /dev/vg00/lvol9
reserve - 30710 -30710
memory 40925 34562 6363 84%
total 71645 65282 6363 91% - 0 -

i found total used is 5+5+30710+34562 = 65282
if i used full swap will become
20480+ 10240+ 30710+ 34562 = 95992

so ~ the total PCT USEd will become 95992/71645 = 1.3398....
exceed 100% , rite ?

reserve is a system parameter ?
how can i setup it, can i reduce it ?

Re: hp-ux swap utilization so high >90%

Don Morris_1 — Thu, 30 Jul 2009 17:36:51 GMT

No, the reserve is the amount from the above dev/FS lines which is set aside in case of swap.

When/if actual swap occurs, it would shift resources from the Reserved state (accounted for but not specific blocks, no data on dev/FS) to the Used state (accounted for, known specific locations with data on dev/FS).

So a 100% consumption assuming only reservation would look like:

5 + 5 + 30710 + 40925 = 71645

Which, as you'll notice is only different in your memory line. (As 30710 shows you being the sum of 20480 + 10240 [total] minus 5 + 5 [used], you've reserved all of your device/FS swap space. Only memory swap is available at this point for new reservations).

A 100% Used scenario would look like: 20480 + 10240 + 0 + 40925 = 71645. (All reserved swap becomes Used).

As I hope is apparent from the above discussion -- reserve is not a parameter, it is a metric of the workload and represents the virtual address space backed by your device/FS which has not yet had to be swapped out. You can only reduce it by reducing the virtual address space footprint of your workload (run less processes, use fenceline tunables like maxdsiz, maxssiz, shmmax, shmmni, shmseg to enforce smaller virtual objects or fewer objects, etc. and perhaps through tuning in the application space [smaller data set, limiting parameters, whatnot]).

Re: hp-ux swap utilization so high >90%

Dennis Handly — Thu, 30 Jul 2009 18:04:00 GMT

>I found total used is 5+5+30710+34562 = 65282

The total is given by the "total" line. You don't have to compute it. ;-)

Re: hp-ux swap utilization so high >90%

joseph123 — Fri, 31 Jul 2009 09:52:46 GMT

hi guys
i'm understand the formula now.

but, i still confuse
reserve is not a parameter, it is a metric of the workload and represents the virtual address space backed by your device/FS which has not yet had to be swapped out.

in this case
====================
[root@PRDAP3]/usr/sap/PR1/SYS/profile # swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 3931 165 96% 0 - 1 /dev/vg00/lvol2
dev 36000 7387 28613 21% 0 - 1 /dev/vg00/lvol9
reserve - 3729 -3729
memory 8180 1662 6518 20%
total 48276 16709 31567 35% - 0 -
====================
swap has 40G and used approx 11G
so ~it should reserve 29G
why in this case it juse reserve 3729M ??

and...
with regards virtual address space
is it a system parameter ? can i setup it ?

Re: hp-ux swap utilization so high >90%

Don Morris_1 — Fri, 31 Jul 2009 10:32:40 GMT

Presumably because the current running workload doesn't need more than it reserved.

Virtual address space isn't a system parameter -- it is a measure of the virtual size of all the user processes across the entire system.

For a very simple example, let's say we have 3 processes running in total.

Process A uses: 5 Mb for Text, 15 Mb for Data, 2 Mb for Stack, 30 Mb for Shared data (which is shared with B, but not with C). All counts are Virtual, not physical (i.e. they represent the total size from malloc()/shmget()/mmap() [or just the system setting things up like Stack]).

Process B uses: 3 Mb for Text, 25 Mb for Data, 1 Mb for Stack, 30 Mb for shared with A, 10 Mb for shared with C.

Process C uses: 10 Mb for Text, 200 Mb for Data, 2 Mb for Stack, 10 Mb for shared (with B).

The total virtual address space in use is the virtual size of A (52Mb) plus B (69Mb) plus C (222Mb) minus the duplicates from cross-process sharing (the 30 Mb we counted in B that was counted in A, the 10 Mb we counted in C that was in B -- so 40Mb) or 303Mb.

Swap is required to back all virtual objects which are not backed by other means (which almost always just means "not backed by a File"). So in the above example [assuming the Shared objects are anonymous memory and not shared file mmap for simplicity], we can remove the Text objects from the total.

So our expectation is that the virtual space for each process which needs to be backed by swap is:

A: 15 (Data) + 2 (Stack) + 30 (Shared)
B: 25 (Data) + 1 (Stack) + 30 + 10 (Shared)
C: 200 (Data) + 2 (Stack) + 10 (Shared)

Again removing duplicates -- we see that the expected total swap consumption of the system would be:

15 (Data A) + 25 (Data B) + 200 (Data C) + 2 (Stack A) + 1 (Stack B) + 2 (Stack C) + 30 (Shared A/B) + 10 (Shared B/C) = 285Mb.

This is the virtual address space consumption of the System. (Well, there's some interaction with the kernel allocations -- but I'm trying to keep this simple for explanatory purposes...)

The total virtual address space available (for non-file backed objects) is the Total AVAIL on the swap line.

Whether that is Reserved or Used depends on if _physical_ memory got low enough such that the system had to page out memory. [Your output showing multiple Gb USED suggests there was memory pressure on this box in the past at least.] There isn't more Reserved here because no process needs more. If one of these three processes requests more virtual address space (through malloc or other such methods), the corresponding swap reservation request is made at that time, increasing the amount reserved.

By the way, while it hasn't been updated in some time (the tunable statements are very stale for one thing) -- you may find reading the Memory Management white paper illuminating for the basic concepts of HP-UX virtual memory management:
http://docs.hp.com/en/1218/mem_mgt.html

Re: hp-ux swap utilization so high >90%

Ganesan R — Fri, 31 Jul 2009 11:03:42 GMT

Hi Joseph,

>>>>
swap has 40G and used approx 11G
so ~it should reserve 29G
why in this case it juse reserve 3729M ??
<<<<<<

Don explanation is well enough to understand how swap system works.

In simple terms, whenever a process starts some amount of swap space will be reserved. Note that only "reserved" not "used". And the "some amount" represents the amount of swap space needed to move the process to swap area in case the physical memory is full. So when the reservation reaches to the maximum available swap then you will not be able to new process. In this situation only you have to increase the swap.

If "USED" column on dev/fs is more, it indicates there is a physical memory pressure so system started using swap area for swap in/out activity. In that case you have to go far increasing physical memory.

You can read this also to understand the swap memory usage..
http://docs.google.com/View?docid=dc7s6mzr_56cgsg9g

Re: hp-ux swap utilization so high >90%

joseph123 — Sun, 09 Aug 2009 15:40:43 GMT

hi guys

[root@SAPDBCI]/ # swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20480 230 20250 1% 0 - 1 /dev/vg00/lvol2
dev 10240 230 10010 2% 0 - 1 /dev/vg00/lvol9
reserve - 30261 -30261
memory 40925 36278 4647 89%
total 71645 66999 4646 94% - 0 -
=========================
this server is start to used swap
is it possible to find out why memory used (36278) is so high?
or identify which process used many memory ?

Re: hp-ux swap utilization so high >90%

Dennis Handly — Sun, 09 Aug 2009 19:25:45 GMT

>this server is start to used swap

Are you looking at the vmstat output for "po"?

>is it possible to find out why memory used (36278) is so high? or identify which process used many memory?

Because you are running too many processes that want to use lots of swap. Or you have configured SAP to use too much.

You can look at the memory used in the ps(1) output. You can also look at "ipcs -ma" for shared memory segments.

Re: hp-ux swap utilization so high >90%

joseph123 — Tue, 22 Sep 2009 09:20:26 GMT

hi guys

i met a problem
from database listener.log
i did not find any other alert from other log.

11-SEP-2009 04:14:51 * (CONNECT_DATA=(SID=PR1)(GLOBAL_NAME=PR1.WORLD)(CID=(PROGRAM=C:\usr\sap\PR1\D05\exe\disp+work.EXE)(HOST=PRDAP5
)(USER=pr1adm))) * (ADDRESS=(PROTOCOL=tcp)(HOST=10.158.4.16)(PORT=2302)) * establish * PR1 * 12500
TNS-12500: TNS:listener failed to start a dedicated server process
TNS-12540: TNS:internal limit restriction exceeded
TNS-12560: TNS:protocol adapter error
TNS-00510: Internal limit restriction exceeded
HPUX Error: 12: Not enough space

==============================
seems like i should modify two thing
1. Modify kernel parameter
2. add swap

with regards 1, i have checked shmmax, this paremeter has been set to the size of the RAM which is 42 GB.
and maxdsiz*, the kernel parameter has been set as per SAP recommendation.
so, i don't think i sould modify any kernel parameter.

so, i read some data from http://docs.google.com/View?docid=dc7s6mzr_56cgsg9g

i suspect i should add swap

but, from our curreut status

# vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
5 7 0 12523101 633358 244 38 2 0 0 0 2 11322 67967 6615 21 17 62

# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20480 862 19618 4% 0 - 1 /dev/vg00/lvol2
dev 10240 861 9379 8% 0 - 1 /dev/vg00/lvol9
reserve - 28996 -28996
memory 40925 36228 4697 89%
total 71645 66947 4698 93% - 0 -

we can see swap just reserve not used, why does our swap not enough?
please give me some advise, thanks

Re: hp-ux swap utilization so high >90%

Dennis Handly — Tue, 22 Sep 2009 09:44:17 GMT

>maxdsiz*, the kernel parameter has been set as per SAP recommendation.

What are the values?

>total 71645 66947 4698 93%

You only have 4.6 Gb free to allocate. Did you do swapinfo at the time of failure?
How much swap that new process going to use?

>we can see swap just reserve not used ...

That's what reserved mean, it can't be used for anything else.

Re: hp-ux swap utilization so high >90%

joseph123 — Tue, 22 Sep 2009 16:12:00 GMT

Shalom,

kernel parameter Current value
maxdsiz 1073741824 1073741824 yes
maxdsiz_64bit 17179869184 17179869184 yes
shmmax 42949672960 42949672960 yes

i did not do swapinfo at that monment, i'm not sure how much swap that the new process will going used.

Re: hp-ux swap utilization so high >90%

Dennis Handly — Wed, 23 Sep 2009 05:21:40 GMT

>I did not do swapinfo at that moment, I'm not sure how much swap that the new process will going used.

(Did you ever mention which OS version?)

Then you could be out of swap. You could use size(1) on the executable and all shlibs to see how much you need at the start.
Of course it could die latter on a big malloc/mmap.

Re: hp-ux swap utilization so high >90%

joseph123 — Wed, 23 Sep 2009 06:44:09 GMT

the os version is 11iv3 B.11.23

Re: hp-ux swap utilization so high >90%

joseph123 — Thu, 24 Sep 2009 16:07:44 GMT

# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 20480 830 19650 4% 0 - 1 /dev/vg00/lvol2
dev 10240 828 9412 8% 0 - 1 /dev/vg00/lvol9
reserve - 29062 -29062
memory 40925 35720 5205 87%
total 71645 66440 5205 93% - 0 -

if i want to create a new swap about 10G
our pe size is 16M
i should type
# lvcreate -l 640 /dev/vg00
if new lv named lvol10
i want enable swap via /etc/fstab

# more /etc/fstab
/dev/vg00/lvol3 / vxfs delaylog 0 1
/dev/vg00/lvol1 /stand hfs defaults 0 1
/dev/vg00/lvol4 /opt vxfs delaylog 0 2
/dev/vg00/lvol5 /tmp vxfs delaylog 0 2
/dev/vg00/lvol6 /usr vxfs delaylog 0 2
/dev/vg00/lvol7 /var vxfs delaylog 0 2
/dev/vg00/lvol8 /home vxfs delaylog 0 2
/dev/vg00/lvol9 ... swap pri=1 0 0
devap1:/export/archive/ERP /usr/sap/archive/ERP nfs rw,suid 0 0

i should edit /etc/fstab
add a swap entry
/dev/vg00/lvol10 ... swap pri=1 0 0

and
# swapon -a

that's all, rite?
i don't need to stop application server which is on the server, rite?
i also don't need restart server, rite?