Some idea why pagedevice (swap) can be 100% full

[Frank de Vries]

I have an old hpux 10.20
running. Old but stable.

Untill last monday, we started to get
messages like :
/usr/lib/dld.sl: Call to mmap() failed - DATA /usr/lib/libdld.1
/usr/lib/dld.sl: Not enough space
sh[48]: 3021 Abort(coredump)
sh[74]: test: Specify a parameter with this command.
sed: Memory allocation failed.

A quick look of the swapinfo command
revealed that PCT USED for primary swap was
100% en for secondory swap around 90%
(about 2,3 giga worth)

A quick scan of the processes ps and top revealed that not really a whole lot was running ?

I had to reboot in the end as even a running simple on the command line failed with
an error.

This has got me puzzled.
How can unix not reclaim the space,
and certainly it is well oversized.
We only need a couple of 100's of MB.

It has been running for years ,so
how did the swapdevice get so full ?

Is this typical.
Any ideas welcome, but it would help
if they are thought through.

[Jeff Schussele]

Hi Frank,

It's because you're seeing the swap *reservation* usage.
Every process started *must* reserve it's "memory" space in swap - just in case it does get paged out.
If you either add more physical swap space OR turn swapmem_on actually on you'll immediately be able to start more processes.

HTH,
Jeff

[Chan 007]

Hi,

1. Are you upto date with patches?
2. Have you setup all kernal parameters as per the OS and Applications requirements.

Generally these r the 2 thing we miss in a old system like yours.

Also check if any rouge process running. Trying shutting the application.

Check you SAR, VMSTAT, DMESG and SYSLOG.LOG for more what is happening ...

all the best....007

[Frank de Vries]

Jeff
Eh , yes and no :)

I don't realy want to technically solve it:)
It is already solved.
I am just trying to understand how it
could happen only once in 5 years,
and even then I cannot accept normal
behaviour could fill up 2,3giga of swapspace
while normally it is always 0%.

So what happened on that blue moon ?

The dev part of swapinfo
swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 524288 0 524288 0% 0 - 1 /dev/vg00/lvol2
dev 2048000 0 2048000 0% 0 - 1 /dev/vg00/lvol9

(It is now 0%, but image it is 100%)

And the swap related params:
[root@oradb1:]/var/mail<>>> sysdef | grep sw
allocate_fs_swapmap 0 - - -
maxswapchunks 1500 - 1-16384 -
nswapdev 10 - 1-25 -
nswapfs 10 - 1-25 -
remote_nfs_swap 0 - - -
swapmem_on 1 - - -
swchunk 2048 - 2048-16384 kBytes -
You have mail in /var/mail/root
[root@oradb1:]/var/mail<>>>

So swapmem_on is 1

Makes no sense to me :)

[Patrick Wallek · Accepted solution]

Well, if the % used was 100% for the Pri. Swap and 90% for secondary then SOMETHING caused you to start severely paging out.

Since you have already rebooted it will be almost impossible to determine the exact cause. But there was most definitely something somewhere that used all of your RAM caused you to start severely paging out.

[Jeff Schussele]

Run swapinfo -tam & check the total line.
When that hits 100% you can start *no* further processes.

HTH,
Jeff

[Ted Buis]

Sounds like a very slow memory leak. Check it in another four years to see if you need to reboot it again. Seriously, if this is on the internet, you should be concerned about security. There are many holes that are now common knowledge that were never patched in 10.20, so it can be more vulnerable than newer systems.