Extending Swap Space / Memory Leak ?

[EDWARD WYNN_1]

Hi,

I have an L1000 with 1Gb of RAM, 1Gb of Swap space on lvol2, I have pseudo-swap enabled. I am getting the following error messages roughly every week when I try to start a new process/login etc:

/usr/lib/dld.sl call to mmap() failed - BSS /usr/lib/libnsl.l

/usr/lib/dld.sl: not enough space.

I suspected that this was a swap problem and so monitored the swap usage daily for the last week, the last 2 days data (before the error occurs and I have to reboot) is as follows:

swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 168320 880256 16% 0 - 1 /dev/vg00/lvol2
reserve - 880256 -880256
memory 748460 508004 240456 68%
total 1797036 1556580 240456 87% - 0 -


# swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 171736 876840 16% 0 - 1 /dev/vg00/lvol2
reserve - 876840 -876840
memory 748460 646932 101528 86%
total 1797036 1695508 101528 94% - 0 -
#

The system is running 3 Oracle database instances - none of which is very large. So, here are my questions:

1. Why does this keep happening ? Is it simply a matter of not enough swap or is something leaking memory ? If there is not enough swap why does it take so long to fail - I would have expected it to fail sooner ?

2. If something is leaking memory can anyone recommend any useful tools to help track this down ?

3. Assuming I need to increase the swap space,
I have a problem in that I only have one disk and therefore one volume group - which already contains the existing swap - I have read in the forum that creating another logical volume for swap on the same volume group is a bad idea. I think that leaves me with 2 alternatives: file system swap - which I am not keen on, or moving lvol3 out of the way so that I can extend lvol2 - has anyone done this or does anyone have any pointers on how I could go about doing this ?

Many thanks in advance,

Eddie

[Stefan Farrelly · Accepted solution]

1. Yes, you keep running out of swap. This is usually because some application processes continue to grow in memory and do not free up memory used when a user disconnects. A common problem (memory leak) with Oracle. One reason a lot of sites I know regularly shutdown their Oracle databases. We do nightly (except for some important ones where we use an online hotbackup instead). Investigate with Oracle as to your version and if their are any patches to fix memory leaks or if they recommend to upgrade to a new version.

2. Heres a simple tool (if you dont have glance) to monitor process sizes;
UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more

This lists processes in order of size in memory. You can grep the oracle or application processes out if you want. Run it regularly and keep any eye on oracle processes or user ones and if you total them up Im sure you will see the culprit.

3. Putting 2 swap areas on the same disk is not such a bad idea. If you have no choice do it. We have lots of large HP servers with multiple swap areas on one disk and we dont suffer problems. Add another swap area, but remember this isnt a substitute for finding the offending memory leak processes. Simply put the new swap area into a new lvol - you dont need to extend lvol2.
How big is your buffer cache ? perhaps its growing over time and thus causing memory pressure and increased swap usage ? Do a> sysdef | grep bufpages to see how big it is. It should only be a max of 300 MB on any server, and even smaller if youre having memory pressure/swap problems. This is controller by dbc_pct_min and dbc_pct_mac. Try setting them lower.

[Edward Polak]

Hello Edward,

As a rule of thumb, always use 1.5 to 2 times the amount of RAM for swap size. 1GB swap for 1GB of RAM will get you in trouble for sure.

Indeed device swap is to be preferred over filesystem swap since it's much faster. If needed you can actually have an additional filesystem swap setup as a safety net, but make sure it has a lower priority than the device swap.

Best regards,
Edward

[Bernie Vande Griend]

Ah, the mysteries of pseudo swap. Basically pseudo swap allows the system to be able to schedule more processes to run than it actually has memory for. But needs to be able to reserve swap space for these processes to ensure that the system will not run out of memory. Edward is right on the money that you must have 1.5 to 2 times swap space so this "reserve" is large enough. Just to make sure that this is the problem and that you aren't actually swapping, check glance or vmstat and look at page outs, they should be 0. If they are 0, then the reserve is the issue and adding another logical volume to vg00 for swap is just fine. If it was actually swapping out to disk, then its better to have your swap areas spread out on other disks. Add a 500MB logical volume to vg00 for swap and your system will probably be happy again.

[EDWARD WYNN_1]

Thanks to all for their contibutions to date, I have now increased the swap space to 2 Gb by adding a new lvol on the same root volume group and the machine is much happier.

I have also implemented a shell script (thanks to Stefen for that) to log the process memory usage. I have found the following:

1. I was suffering from the HP mib2agt memory leak bug, this is now fixed.

2. We have an in-house developed process that has a memory leak. This process is a CORBA server that provides an interface to an Oracle instance.

3. If our in-house process crashes it does not clean up its connections to oracle, this leaves stale processes using memory until the instance is shut down.

Given this new information I was wondering if anyone had any thoughts on:

a) How to monitor, in detail, the memory usage of a single process - not just the total used but when it is allocated/ freed etc.

b) Given that I cannot shutdown the Oracle instance nightly or even weekly for cold backup how can I get rid of the stale oracle processes - or better still how can I avoid them in the first place ? I have read something in the forums that this situation can happen if the process is in the middle of IO when the controller process crashes...

Many thanks in advance, Eddie