Multiple swap files on the same device

[Ken Englander]

Does anyone know if there are issues in HP-UX 11i v2 or v3 when more than one swap file resides on the same volume group. Typically we try to size our systems so they do not actually page, but things don't always end up as planned.

I've been told that on AIX if there is more than one swap file on the same device, it is possible the system will crash when it starts paging.

Thanks!

[Patrick Wallek]

It is not necessarily a good practice to have multiple swap files on the same device, especially if they are the same priority.

If you do have multiple swap areas on the same device it can lead to disk head thrashing if you actually do page. This can lead to a degradation of performanace, but if you are actually paging to any extent then performance will suck anyway.

[James R. Ferguson]

Hi Ken:

If you must put multiple device swap files on the same physical disk, you should probably make their swap priorities different. Using the same swap priority would mean that I/O interleaving would occur. While this is desirable if the swap devices reside on different disks, disk head movement (if I/O actually occured) would futher degrade already degraded performance.

Of course, if you are adding device swap largely to satisfy process swap reservation, and you generally do not swap then all of this is moot.

Regards!

...JRF...

[Ken Englander]

Thanks for the input!

I just re-read some of the HP doc about interleaving swap files and I can see that they generally recommend using separate devices although it does not actually state NOT to put them on the same device.

If I read it right, the lv swap areas need to be the same size to utilize interleaving.

I find there are two issues I've had to handle that make it simpler to put additional swap on the same device. First, the primary swap is preallocated on vg00 and it is difficult to resize it.

It is true that I have a number of systems that use internal disks so head movement could be an issue, however, so many systems utilize SAN technology that I wonder if the head movement issue is actually a problem.

[James R. Ferguson]

Hi (again) Ken:

> I just re-read some of the HP doc about interleaving swap files and I can see that they generally recommend using separate devices although it does not actually state NOT to put them on the same device.

There's always more than one way in Unix...

> If I read it right, the lv swap areas need to be the same size to utilize interleaving.

Yes, but I believe that swap devices of unequal size (even on the same physical volume) will round-robin I/O until the smaller is "exhausted".

> I find there are two issues I've had to handle that make it simpler to put additional swap on the same device. First, the primary swap is preallocated on vg00 and it is difficult to resize it.

Resizing primary swap is best achieved by re-Ignite-ing your server. Primary swap needs contiguous extents and to extend it you would first need to move '/' outward.

> It is true that I have a number of systems that use internal disks so head movement could be an issue, however, so many systems utilize SAN technology that I wonder if the head movement issue is actually a problem.

It's probably not nearly as much concern on a SAN as you say. As I said above, ideally you are only adding swapspace for the purposes of process swap reservation. You should be sure that pseudoswap is enabled ('swapmem_on'=1 in your kernel settings). This allows 75% of physical memory to be counted for swap reservation --- an accounting trick of the kernel. As of 11.31 this is set by default. Prior to 11.31 you need to verify and/or set it. The presence of the "memory" line in 'swapinfo -tam' means that it is set on.

Regards!

...JRF...

[Ken Englander]

Yes, swapmem is on.

I'm still wondering if anyone knows if an actual problem (or crash) can occur when more than one swap area is set up on the same disk. I do understand performance can definitely be considered a problem, but I'm more interested to know if it can crash the system as I was told can happen on AIX.

[Dennis Handly]

>I'm still wondering if anyone knows if an actual problem (or crash) can occur when more than one swap area is set up on the same disk.

It would most likely be illegal to copy the bugs in another OS. :-)
Don't worry about it on HP-UX.

>JRF: As of 11.31 this is set by default.

On 11.31, swapmem_on only comes in black, it can't be turned off. Which implies users should always have it on, on previous OS versions.

[VK2COT]

Hello,

You already got good replies. To emphasize
the critical points:

Avoid having multiple swap devices on the
same physical volume!

SOme good design principles:

a) Primary swap is 4 GB (maximum 8 GB
if there is enough space on boot disks).

b) Add secondary swap devices in the same
size (4 GB) on other disks, or even better,
SAN LUNs (if possible).

c) Set the priority to be the same (that
way you achieve perfect interleaving).

e) If you add more RAM, it is easy to
add extra swap devices, without breaking perfect design rules.

f) If you ever start using swap, your boot disk will not suffer from high I/O because
only a small chunk of swap is used on it.

g) Create dedicated dump device.
If enough space, dedicated dump volume
equals RAM size, or between 1/2 and 3/4
of it.

Dedicated dump device will not shorten the
time required to write from memory to dump
during the crash, but WILL SHORTEN the
reboot time.

Think what happens at crash time typically:

1. Memory image gets dumped into swap/dump area.

2. At reboot, due to risk of swap
overwriting those dump pages, savecrash
"exports" them into the crash dump directory
(typically /var/adm/crash).

That way, you can keep an image in the dump
area until the next crash or run
savecrash at will.

On a server with large amount of RAM, this is
a serious problem because the reboot
can take a loooong time. That is a huge
risk because the golden rule of system
administration is: restore production
services AS SOON AS POSSIBLE and then
analyze the root cause.

For that very reason, in my Operations Acceptance Testing, I check these things and fail servers that are not built properly:

INFO Swap space
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 12517376 0 12517376 0% 0 - 1 /dev/vg00/lvol2
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvswap2
reserve - 1610368 -1610368
memory 12473024 11362304 1110720 91%

AUDIT-INFO Physical volume /dev/disk/disk3_p2 contains swap device /dev/vg00/lvol2
AUDIT-INFO Physical volume /dev/disk/disk3_p2 contains swap device /dev/vg00/lvswap2

AUDIT-WARN /dev/vg00/lvol2 has no mirrors

AUDIT-WARN /dev/vg00/lvswap2 has no mirrors
AUDIT-PASS /dev/vg00/lvswap2 has Mirror Consistency Recovery enabled for additional (non-primary) swap

AUDIT-FAIL Physical volume /dev/disk/disk3_p2 contains multiple device-based paging spaces

Cheers,

VK2COT

http://www.circlingcycle.com.au/Unix-sources/HP-UX-check-OAT.pl.txt

[Ken Englander]

> It would most likely be illegal to copy the bugs in another OS. :-)

Dennis - GREAT answer - I did not think HP did such things, but my associate who works with AIX seemed to think it was logical to make such an assumption.

VK2COT - Thanks for the suggestions about the dump device! I have systems with over 8 GB primary swap, so I'm not sure that rule applies. Also, I would suggest it is not practical to set up sufficient manageable swap with only files of 8 GB in size with the large memory systems we use.

[VK2COT]

Hello Ken,

The size of RAM is almost irrelevant when you
design the device-based swaps.

We have servers with 128 GB and 256 GB of RAM
where the primary swap is only 4 GB. And
you can imagine how long would take
to reboot after crash if classical (shared)
swap/dump design is used.

The other swaps are also 4 GB in size and
reside in many LUNs on SAN. Dump areas
are separate. Simple and easy.

And if swap ever gets used, the boot disk will not suffer the silly I/O bottleneck :)
Plus, I will never ever worry about
resizing the primary swap (for whatever reason). As we know, that is not a pleasant
task (requires a reboot) in HP-UX.

"A friend in need is a friend indeed".
When swap is not used, ANY server (even those
built by grandmothers) will work fine.
What happens when swap really gets used
is my concern and I like to address it
before it happens.

I guess I had worked for too long in
Unix/Linux support as well. So, I know what
hurts in real life. Theory is one thing and
reality the other :)

In the end, the important things is
that there are number of choices and we are
free to form a different opinion :)

Cheers,

VK2COT