Explain please SWAPINFO for me

[Igor I. Shulz]

Time to time my system suffered from lack of paging area:

ksh: no memory: Not enough space

/usr/lib/dld.sl: Call to mmap() failed - BSS /usr/lib/libcl.2
/usr/lib/dld.sl: Not enough space
sh: 18280 Abort(coredump)

/usr/lib/dld.sl: Call to mmap() failed - ZEROES /opt/perl5/lib/5.8.0/PA-RISC2.0/CORE/libperl.sl
/usr/lib/dld.sl: Not enough space
sh: 18472 Abort(coredump)

To prevent such errors I added additional 1GB swap device and for last 1/5 months the errors didn't take place.

But during peak of activity on my system last week I saw this output of `swapinfo -tam`(see attachment for possible better view):

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 1339 2757 33% 0 - 1 /dev/vg00/lvol2
dev 1000 0 1000 0% 0 - 2 /dev/vg00/lvol_swap
reserve - 3757 -3757
memory 13166 12294 872 93%
total 18262 17390 872 95% - 0 -

This is strange: how it could be - total 872 MB of free swap (paging area), though there are at least 1000 MB free space in one of my swap devices ? I'm totally confused...
And what should I do? Should I add more swap space to make these errors ever less possible?

[Geoff Wild]

Check out the following articles:

http://www.met.ca/itrc/index.php?option=com_content&task=view&id=52&Itemid=2

http://www.met.ca/itrc/index.php?option=com_content&task=view&id=38&Itemid=2

Also, from man swapinfo:

-t Add a totals line with a TYPE of total. This line totals
only the paging information displayed above it, not all
paging areas;


Rgds...Geoff

[Igor I. Shulz]

2Geoff:

you are right about -t. It's why I use -a key:

-a Show all device paging areas, including those configured into the kernel but currently disabled

thank you for the links.

[Steven E. Protter]

Shalom,

Look like one swap space is not usable.

dev 1000 0 1000 0% 0 - 2 /dev/vg00/lvol_swap

I would not add swap, I'd add memory. Any time you are paging to swap you are getting killed on the performance side.

SEP

[A. Clay Stephenson]

If you examine the mmap man page, you will find that more than lack of virtual memory (swap) can cause mmap to set errno to ENOMEM. You can also be hitting kernel limits or even ulimit constraints.


I'm guessing that you have about 20GiB of physical memory judging from your swapinfo output; if so, you really don't have all that much swap space and add only a 1GiB swap space to such a system might not be enough. It appears that you had a situation where the total amount of virtual memory remaining was simply not enough to grant the request so the process terminated --- and
that is why you could see no use of the 1GiB swap space and still run out of (virtual) memory.

You really have 2 ways to fix this. 1) Add more physical memory 2) Add more swap space. If you only have brief, predictable periods of heavy memory presssure then you might decide to simply add more swap space and tolerate the performance hit during those periods. Relying on pseudoswap (as you are at the moment) assumes that you really play by the rules and in your case, you are skirting the edge.

[Igor I. Shulz]

2Geoff:
do you mean that it would be better for me (to see more clear pictiure) to use swapinfo with -tdm keys instead of -tam?

2Clay:
yes, I have brief and regular periods of havy activity.

Thank 2 all.

[Geoff Wild]

Here's a swapinfo -tam from my SAP server:

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 524 3572 13% 0 - 1 /dev/vg00/lvol2
dev 22432 0 22432 0% 0 - 2 /dev/vg01/swap
reserve - 15928 -15928
memory 12648 2079 10569 16%
total 39176 18531 20645 47% - 0 -


Notice the total free is less then what is availble on my vg01/swap.

I prefer to use memdetail:

# memdetail
Memory Stat total used avail %used
physical 16128.0 15346.5 781.5 95%
active virtual 17150.4 5085.3 12065.0 30%
active real 12288.8 3374.0 8914.9 27%
memory swap 12647.5 2078.2 10569.3 16%
device swap 26528.0 16453.5 10074.5 62%


I attached it - you can compile with the HP native compiler.

Rgds...Geoff

[A. Clay Stephenson · Accepted solution]

In your case, if the memory pressure is brief and the performance degradation during those times is acceptable then you should probably abandon pseudoswap (swapmem_on=0) and add enough device swap to get you up to about 1.3-1.5X physical memory. Pseudoswap is really intended for those systems which have much more memory than swap (which you do) AND never really use swap to any significant degree. You don't meet the second criterion all the time. In your case, reverting to a traditional swap model will result in safer behavior.

The other obvious fix that costs you nothing is to tune some of your applications or the UNIX buffer cache. For example, if you are running databases then reducing the size of the database caches might completely eliminate the swapping. The extra overhead of having to go to disk more often because of reduced database and/or UNIX buffer caches has a much smaller performance impact (by a factor of at least 100) than paging out.

[A. Clay Stephenson]

Oh, one more thing to look for are large shared memory segments which are no longer attached to any process. Done any kill -9's?

Do an ipcs -ma and look for any segments which have an NATTACH value of zero. These MIGHT be candidates for removal. NATTACH = 0 is a necessary condiotion for removal but not necessarily a sufficient condition. It depends upon the design of the software.