Why is available swap 10% when free swap is 65%

[Brett Matheson]

Can anyone please explain to me why we have so much free swap space, yet such a small amount of available space? What is the extra Reserved swap space reserved for ?

Swap partition /dev/disk/dsk1b (default swap):
Allocated space: 1932685 pages (14.75GB)
In-use space: 658337 pages ( 34%)
Free space: 1274348 pages ( 65%)


Total swap allocation:
Allocated space: 1932685 pages (14.75GB)
Reserved space: 1735818 pages ( 89%)
In-use space: 658337 pages ( 34%)
Available space: 196867 pages ( 10%)

Thanks

[Venkatesh BL]

It could be because of the swap 'mode' set on the system. Check out the manpage of 'swapon' for useful info.

Also, searching this forum with the keywords would yield good results.

[Johan Brusche]

Brett,

Use the manpage for sys_attrs_vm and look for the explanation of the sysconfigtab parameter "vm_swap_eager".
Your system is using the default =1. If the environment does not need swap reservation in advance (you have 89% resereved now), then set vm_swap_eager=0 (and reboot).


__ Johan.

[Ralf Puchner]

Reading administration guide is a starter ;-)

[Joerg Schulenburg · Accepted solution]

Deliver Tru64 with eager=0 per default is a better starter! Or do you now a (real) situation where swap reservation in advance has a advantage?
One advantage is clearly, that you have to read the documentation to find out that eager=0 is your friend *sigh*.

[Brett Matheson]

Thanks for that Joerg. That was probably my main question - we were told that it was risky to change the allocation mode for our server (which is running Oracle databases).

[Victor Semaska_3]

The risk in changing from changing immediate mode to deferred mode according to section 3.6.3 of the System Administration Guide:

The operating system terminates a process if it attempts to write a modified virtual page to swap space that is depleted.

In other words, if you use deferred mode and the swap space isn't large enough, the OS will start deleting processes.

Vic

[Joerg Schulenburg]

>>In other words, if you use deferred mode and the swap space isn't large enough, the OS will start deleting processes.

What about immediate mode and the swap space isn't large enough. What happens, if the kernel or a driver needs some memory?
Isn't it that UNIX, Windows or whatever comes into big trouble if all physical memory and swap is consumed?
The immidiate mode does not prevent the trouble. But usable memory in immidiate mode is equal to swap whereas usable memory in deferred mode is physical memory + swap.
So who is coming in trouble first?

Happy swapping!

[Victor Semaska_3]

Joerq,

You should read that section in the SysAdmin Guide. Here's the URL to the doc.s:
http://h30097.www3.hp.com/docs/pub_page/doc_list.html

Later in the section it says:

If you use the immediate mode, swap space depletion prevents you from creating additional modifiable virtual address space. If you use the deferred mode, swap space depletion can result in one or more processes being involuntarily terminated.

So, if I understand it correctly, with immediate mode you won't be able to create any new process if you run out of swap space. With deferred mode, the OS will start deleting *existing* processes. We've had this happen once on a system with deferred mode. A process went 'nuts' and used up a lot of virtual space thus using all the swap space. The OS then started deleting processes like the SSH daemon sshd. Had to reboot to get the system working again.

Vic

[Joerg Schulenburg]

Dear Victor,

Please try this on your machine:
main(){
int i, *p;
for (i=0;;) {
p=(int *)malloc(1024*16);
if(p)printf("%i MB\n",(++i)/64);
else sleep(1);
}
}

In immediate mode on a ES45 16GB, 20GB swap it ends with
VMUBC showing Act=1144K Free=554K Swap.free=2 Swap.used=2500K.
Now try to login, or do call a command like kill. No chance.
The result is like this:
Unable to obtain requested swap space
bash: fork: Operation would block

Ok, its true, nothing is killed. But you cant
do anything on the system, where a lazy-system has no problems at all.
If you really want it, you dont need eager swap. The system (not necessarely the kernel) only have to control the sum of all virtual memory and let fail allocation if a limit (mem+swap) is reached, which is easy to program.
No swap-disk is needed for such a feature but it would also work with lazy swap.
Indeed it would prevent the system to decide which process to kill in the out-of-memory contition. Is that an advantage to come in the out-of-(v)memory condition if there is lot of memory free?
And needs the out of virtual memory condition to be coupled with swap?
I still think that immediate mode was
created in a weak hour of the kernel programmer.

Comments welcome :)

[Alexey Borchev]

>> usable memory in deferred mode is physical memory + swap.

- Little correction:
Imho, if a process was swapeed out, then read back into memory - it uses both mem and swap.

[Joerg Schulenburg]

>> Imho, if a process was swapped out, then read back into memory - it uses both mem and swap.

Is that in general? I mean for all Unixes or is it a Tru64 "feature"?
On Linux I think pages are read in from swap
and swap can be freed if that pages keep in memory, so that other pages can use this swap space. Doing so only a small part of pages
exist in memory and swap at the same time.
But I am not sure.
If I find some time I will start a test linux
in the qemu PC emulator and test it.
After that I can give a more qualified statement.

[Joerg Schulenburg]

Ok, I found time to test swap. Its swap+memory
which is usable. I tested IRIX64 and Linux (k2.6.11), lazy type swap of course.

swaptest.c:
main(){ char *buf;int j,i=0;for(;;){
buf=(char *)malloc(1024*1024); /* alloc memory */
if (buf) { i++; for (j=0;j<1024;j++) buf[j*1024]=i; } /* access pages */
else { sleep(1); }
printf(" %dMB\n",i);
}}

Knoppix-3.8 + 512MB memory + 256MB swap

swaptest killed by the kernel at
473MB (without swap) and 728MB (active swap).
I tested also a SGI-O2100-IRIX6.5-64bit
where it was killed after 14GB (10GB memory +
5GB swap). On a SunOS-5.7 machine malloc failed after 1459MB (728MB memory + 1GB swap).
Nothing is crashed, but it was a
very slow on the IRIX machine.
I cant test it on Tru64 because
I dont have a empty machine in lazy mode at the moment and I dont know if it can be tested with a emulator like qemu.
I am interested, if you can make a test. I can do it on the next crash or
if HP support forces me to reboot again.

So the normal result is memory + swap as
usable memory on a good unix.

May be someone can test it for Windows too?

[Ralf Puchner]

I'm not sure why you are testing the program on different platforms and compare the results. Each unix implementation differs from other depending on the base (BSD/SYSV). Windows swapping is totally different, linux is only "unix" like and depending on the distribution will follow an other path.

Tru64 offers 2 swapping modes, if reading the admin guide you will get an explanation.
Modes:

1. total memory = swap
2. total memory = physical memory + swap

So depending on the needs of your application and the vendor config guides choose one of the two modes!

[Joerg Schulenburg]

>> I'm not sure why you are testing the program on different platforms and compare the results.

It was only to show that Tru64 is the only one, which comes with an eager mode and it
would probably be a good idea to remove that mode from the kernel code.
Lazy mode is clearly the better one, therefore people do switch to this mode.
Kernel (and documentation) would be less complex without eager mode.
I would not weep if eager mode is vanished
from the Unix world.

>> ... if reading the admin guide you will get ...
>> 1. total memory = swap
>> 2. total memory = physical memory + swap

Thanks for reading the tuning guide for us.
Assuming the tuning guide is up to date,
we see that Alexeys "Little correction"
has to be corrected.

[Alexey Borchev]

Probably I was unclear with my 'little correction'
To make experimental conformation or rebtion, we should:
- Hawe a Tru64, with lazy swap mode, for example 1 GB ram + 2 GB Swap.
- Hawe two test processes:
1) 'Workhorse' - it consumes, say, 1/2 of Phys. memory, then wait until 'Run' command. Upon command it reads\writes all the mem and wait again.
2) 'Memeater' - exactly Your swaptest.

Now, the experiment:
1) On idle system, run 'Workhorse'. It will have it's 0.5G mem, and will rezide in phys. mem. Passive state - waiting.
Phys Mem usage = 0.5.GB (workhorse)
Swap Usage = 0
2) Start memeater, let it eat 1 GB.
Phys. mem = 1 GB (eater rezide here)
Swap usage = 0.5 GB (Workhorse swapped out)
3) Memeater is treminated or killed.
Phys. mem ~ 0 GB (of course, not 0, because of Tru64 itself)
Swap usage = 0.5 GB (Workhorse still here)
4) Run Workhorse. It will go through it's mem. Tru64 will read workhorse back into memory.

5a) End state 1 - swap space is de-allocated upon read-in:
Phys Mem usage = 0.5 GB (workhorse)
Swap Usage = 0

5b) OR end state 2 - swap space is still allocated upon read-in:
Phys Mem usage = 0.5 GB (workhorse)
Swap Usage = 0.5 (swap-space allocated for workhorse, as in immediate mode)

My 'little correction' is 5b. I.e. imho, Tru 64 will not de-allocate swap.

Your volutes, please - 5a or 5b?

P.S.
By the way, this scenario DOES occur in 'real life'.

[Joerg Schulenburg]

Alexey,
I could not test your suggestion on Tru64, because I can not disturb users just for playing with swap. Also Tru64 does not like such kind of experiments on NUMA machines.
But I did it with linux and the outcome should be very similar. I used knoppix-3.8 within qemu and 256M RAM. But I translate the outcome to your 1GB sample for simplicity.
I used the memeater as workhouse too and
used kill -SIGSTOP and -SIGCONT to switch between waiting and running state.

experiment:
1) workhouse stopped, mem=0.5GB, swap=0
2) eater, mem=1GB, swap=0.5GB
3) eater killed, mem~0, swap=0.5GB
4) cont workhouse, mem=0.5GB
as you predicted.

End state is 2, but not exactly as you described!
5b) mem=0.5GB, swap~0.5GB

This is not like in immediate mode.
In immediate mode this part of swap belongs
to workhouse. In our case it can be used
by other process pages. Try additional memeater 1GB run and stop it: mem=1GB, swap=~0.7GB, workhouse runs.
Swap was not reserved for workhouse!
So its simply kind of definition. If the system would show swap=0 it would be also
ok. In both cases (5a and 5b) we dont need disk operations to put workhouse back to swap.
In both cases this part of swap can be reused
by other pages. Thats completly different from immediate mode.
Of course Tru64 can behave different.
The result would be the same,

maxmem=swap+phys.mem.

The difference would
be, that workhouse has to be swapped out
on the next low memory+rest_of_swap condition, whereas
linux can decide to swap other pages out.
Performance would differ and dead locks
are more likely.

Right?

Ralf, what does Tru64 in that experiment?

Happy swapping!