High swap usage

[nancy rippey]

We have a HP-UX 11.11 server that occasionally runs out of swap. It has 32GB of memory with about 14GB free. Kernel parm swapmem_on = 1 and device swap is 16GB. Below is output of swapinfo -tam:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 16384 0 16384 0% 0 - 1 /dev/vg00/lvol2
reserve - 16085 -16085
memory 26327 10830 15497 41%
total 42711 26915 15796 63% - 0 -


Total swap can reach 99% at which time we'll recycle one application that uses a lot of virtual memory. This brings total swap under 40% but it just starts climbing again. It seems a waste to add more disk swap space, adding memory will have little effect, and the 3rd party mapping application is not going to change.

Are there any other memory management techniques to keep from running out of swap space?

Thanks,
Jim

[TTr]

> we'll recycle one application

Do you know that this application justifiably uses all that memory? It looks like it may have a really bad memory leak. Nothing much you can do about memory management. This app will use all the memory resources.
> we'll recycle one application
How often do you have to do this? If you can not change the app you should look into cron scheduling a restart of this app during off hours before it uses up all the memory.

[V. Nyga]

Hi,

your swap device is still not used (0%).
pseudo_swap is raising.
Have you checked 'vmstat -n' for po (paging out swap)?
Also check patches for memory leaks - they are too much patches if you don't know what kind of program (java, drivers) is causing this leak.
Are you up-to-date with patching?
Also you should have as much device swap as ram.

Volkmar

[Jaime Bolanos Rojas.]

Nancy, it does look like this application has a memory leak, there are some tools that you can use to check on that, for example you can use this document, and take a look at section D:

ftp://eh:spear9@hprc.external.hp.com/memory.htm

It sounds crazy, but another way to check is by incrementing the device swap, and then seeing if by any chance the application will reserve that space too up to reaching 99%.

Regards,

Jaime.

[nancy rippey]

We have to cycle the app about once per month. The main problem seems to be Oracle connections, which according to GPM use 1.6GB of virtual memory each. Sometimes there are dozens of these. I believe this is a C app, and being 3rd party we have no control over this.

vmstat 1 shows po = 0.
I realize that swap is not used but all of it is "reserved". This is jsut the way HP-UX manages memory. I do not agree that we should have as much swap as RAM. That was HP's reason for implenting speudo_swap.

From DocId KBAN00000895

Pseudo swap is HP's solution for large memory customers who do not wish to purchase large amounts of swap space. The justification for purchasing large memory systems is to prevent paging and swapping, therefore; the argument becomes "Why purchase a lot of swap space if the system is not expected to page or swap?"

I think it is unreasonable to add device swap when it is not even being used.

Thoughts?

[Steven E. Protter]

Shalom,

Your application has a memory leak.

http://www.hpux.ws/?p=8

That tool will identify the process.

SEP

[V. Nyga]

Hi again,

you're right, but your 'total' is more used (in %) than your 'memory', and 'dev' is a part of 'total', so increasing 'dev' will increase 'total' and lower the use in %, so you'll meet 99% later.
Check knowledge base for informations about memory leak with oracle.
Are you up-to-date with patches?
Are there recommended settings for kernel parameters and patches for this application?

V.

[nancy rippey]

The mem.mon script at http://www.hpux.ws/?p=8
is interesting but results do not agree with gpm. The Oracle connections showing 6.9MB resident and 1.6GB virtual in gpm are only showing 25712KB resident and 102976KB virtusl in the script. Or am I reading this wrong? (The man page seems a little old).

It would be nice to know which is correct before going back to the vendor to fix their leak.

[Dennis Handly]

>I think it is unreasonable to add device swap when it is not even being used.

Either you add more device swap, or you have to restart or recode your application.

Unfortunately if it has a leak, it only means you can wait longer before restarting it. :-(

>is interesting but results do not agree with gpm. ... Or am I reading this wrong?

gpm is always correct. :-)
This difference isn't important. Do both show it growing?

>TTr: Nothing much you can do about memory management. This app will use all the memory resources.

You can cheat. You can have that evil process do lazy swap allocation and oversubscribe. And if it actually needs it, it is the only one that dies.

[nancy rippey]

I think we'll see about migrating this application to Red Hat.

Thanks to everyone for their responses.