Multiple swap files on the same device

[Ken Englander]

Does anyone know if there are issues in HP-UX 11i v2 or v3 when more than one swap file resides on the same volume group. Typically we try to size our systems so they do not actually page, but things don't always end up as planned.

I've been told that on AIX if there is more than one swap file on the same device, it is possible the system will crash when it starts paging.

Thanks!

[Patrick Wallek]

It is not necessarily a good practice to have multiple swap files on the same device, especially if they are the same priority.

If you do have multiple swap areas on the same device it can lead to disk head thrashing if you actually do page. This can lead to a degradation of performanace, but if you are actually paging to any extent then performance will suck anyway.

[James R. Ferguson]

Hi Ken:

If you must put multiple device swap files on the same physical disk, you should probably make their swap priorities different. Using the same swap priority would mean that I/O interleaving would occur. While this is desirable if the swap devices reside on different disks, disk head movement (if I/O actually occured) would futher degrade already degraded performance.

Of course, if you are adding device swap largely to satisfy process swap reservation, and you generally do not swap then all of this is moot.

Regards!

...JRF...

[Ken Englander]

Thanks for the input!

I just re-read some of the HP doc about interleaving swap files and I can see that they generally recommend using separate devices although it does not actually state NOT to put them on the same device.

If I read it right, the lv swap areas need to be the same size to utilize interleaving.

I find there are two issues I've had to handle that make it simpler to put additional swap on the same device. First, the primary swap is preallocated on vg00 and it is difficult to resize it.

It is true that I have a number of systems that use internal disks so head movement could be an issue, however, so many systems utilize SAN technology that I wonder if the head movement issue is actually a problem.

[James R. Ferguson]

Hi (again) Ken:

> I just re-read some of the HP doc about interleaving swap files and I can see that they generally recommend using separate devices although it does not actually state NOT to put them on the same device.

There's always more than one way in Unix...

> If I read it right, the lv swap areas need to be the same size to utilize interleaving.

Yes, but I believe that swap devices of unequal size (even on the same physical volume) will round-robin I/O until the smaller is "exhausted".

> I find there are two issues I've had to handle that make it simpler to put additional swap on the same device. First, the primary swap is preallocated on vg00 and it is difficult to resize it.

Resizing primary swap is best achieved by re-Ignite-ing your server. Primary swap needs contiguous extents and to extend it you would first need to move '/' outward.

> It is true that I have a number of systems that use internal disks so head movement could be an issue, however, so many systems utilize SAN technology that I wonder if the head movement issue is actually a problem.

It's probably not nearly as much concern on a SAN as you say. As I said above, ideally you are only adding swapspace for the purposes of process swap reservation. You should be sure that pseudoswap is enabled ('swapmem_on'=1 in your kernel settings). This allows 75% of physical memory to be counted for swap reservation --- an accounting trick of the kernel. As of 11.31 this is set by default. Prior to 11.31 you need to verify and/or set it. The presence of the "memory" line in 'swapinfo -tam' means that it is set on.

Regards!

...JRF...

[Ken Englander]

Yes, swapmem is on.

I'm still wondering if anyone knows if an actual problem (or crash) can occur when more than one swap area is set up on the same disk. I do understand performance can definitely be considered a problem, but I'm more interested to know if it can crash the system as I was told can happen on AIX.

[Dennis Handly]

>I'm still wondering if anyone knows if an actual problem (or crash) can occur when more than one swap area is set up on the same disk.

It would most likely be illegal to copy the bugs in another OS. :-)
Don't worry about it on HP-UX.

>JRF: As of 11.31 this is set by default.

On 11.31, swapmem_on only comes in black, it can't be turned off. Which implies users should always have it on, on previous OS versions.

[VK2COT]

Hello,

You already got good replies. To emphasize
the critical points:

Avoid having multiple swap devices on the
same physical volume!

SOme good design principles:

a) Primary swap is 4 GB (maximum 8 GB
if there is enough space on boot disks).

b) Add secondary swap devices in the same
size (4 GB) on other disks, or even better,
SAN LUNs (if possible).

c) Set the priority to be the same (that
way you achieve perfect interleaving).

e) If you add more RAM, it is easy to
add extra swap devices, without breaking perfect design rules.

f) If you ever start using swap, your boot disk will not suffer from high I/O because
only a small chunk of swap is used on it.

g) Create dedicated dump device.
If enough space, dedicated dump volume
equals RAM size, or between 1/2 and 3/4
of it.

Dedicated dump device will not shorten the
time required to write from memory to dump
during the crash, but WILL SHORTEN the
reboot time.

Think what happens at crash time typically:

1. Memory image gets dumped into swap/dump area.

2. At reboot, due to risk of swap
overwriting those dump pages, savecrash
"exports" them into the crash dump directory
(typically /var/adm/crash).

That way, you can keep an image in the dump
area until the next crash or run
savecrash at will.

On a server with large amount of RAM, this is
a serious problem because the reboot
can take a loooong time. That is a huge
risk because the golden rule of system
administration is: restore production
services AS SOON AS POSSIBLE and then
analyze the root cause.

For that very reason, in my Operations Acceptance Testing, I check these things and fail servers that are not built properly:

INFO Swap space
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 12517376 0 12517376 0% 0 - 1 /dev/vg00/lvol2
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvswap2
reserve - 1610368 -1610368
memory 12473024 11362304 1110720 91%

AUDIT-INFO Physical volume /dev/disk/disk3_p2 contains swap device /dev/vg00/lvol2
AUDIT-INFO Physical volume /dev/disk/disk3_p2 contains swap device /dev/vg00/lvswap2

AUDIT-WARN /dev/vg00/lvol2 has no mirrors

AUDIT-WARN /dev/vg00/lvswap2 has no mirrors
AUDIT-PASS /dev/vg00/lvswap2 has Mirror Consistency Recovery enabled for additional (non-primary) swap

AUDIT-FAIL Physical volume /dev/disk/disk3_p2 contains multiple device-based paging spaces

Cheers,

VK2COT

http://www.circlingcycle.com.au/Unix-sources/HP-UX-check-OAT.pl.txt

[Ken Englander]

> It would most likely be illegal to copy the bugs in another OS. :-)

Dennis - GREAT answer - I did not think HP did such things, but my associate who works with AIX seemed to think it was logical to make such an assumption.

VK2COT - Thanks for the suggestions about the dump device! I have systems with over 8 GB primary swap, so I'm not sure that rule applies. Also, I would suggest it is not practical to set up sufficient manageable swap with only files of 8 GB in size with the large memory systems we use.

[VK2COT]

Hello Ken,

The size of RAM is almost irrelevant when you
design the device-based swaps.

We have servers with 128 GB and 256 GB of RAM
where the primary swap is only 4 GB. And
you can imagine how long would take
to reboot after crash if classical (shared)
swap/dump design is used.

The other swaps are also 4 GB in size and
reside in many LUNs on SAN. Dump areas
are separate. Simple and easy.

And if swap ever gets used, the boot disk will not suffer the silly I/O bottleneck :)
Plus, I will never ever worry about
resizing the primary swap (for whatever reason). As we know, that is not a pleasant
task (requires a reboot) in HP-UX.

"A friend in need is a friend indeed".
When swap is not used, ANY server (even those
built by grandmothers) will work fine.
What happens when swap really gets used
is my concern and I like to address it
before it happens.

I guess I had worked for too long in
Unix/Linux support as well. So, I know what
hurts in real life. Theory is one thing and
reality the other :)

In the end, the important things is
that there are number of choices and we are
free to form a different opinion :)

Cheers,

VK2COT

[Ken Englander]

VK2COT

I understand your statement about sizing swap as a general view, but we must size it large at 1-2 times the size of memory because of our application and/or running an Oracle database where we typically load as much into memory as possible in the SGA.

How much swap do you have allocated for your large memory systems? Do you actually have dozens of 4 GB LUNs?

[Dennis Handly]

>Ken: but we must size it large at 1-2 times the size of memory because of our application and/or running an Oracle database where we typically load as much into memory as possible in the SGA.

If you want to load as much in memory as possible, you don't want to be in swap. It does no good to have that much swap because if you need it, it will run dog slow.

So what is the "swapinfo -tam" output of your system?

[VK2COT]

Hello,

Dennis already answered your 1 or 2 x RAM
for swap requirement.

If you really need that much swap, simply
add many 4 GB LUNs for swap devices.

Let's say, you have a server with 64 GB RAM.

Set primary swap to 4 GB on boot disks (make
sure swap is mirrored UNLESS BOOT DISK IS
SAN-BASED LUN, which might be RAID-1 or RAID-5 on SAN already).

If you want to use formula 2 x RAM for swap,
than add 31 LUNs of 4 GB for swap devices.

In total, you will have 128 GB swap and
yet, its performance will be perfect.
Plus, you will never cause disk I/O
load on boot disks if swap is used.
And you will not "waste" disk space on
boot disk for silly swap area.

And, if you ever add more RAM, you simply
add more 4 GB devices for swap to
match whatever rule you use. No need to
resize primary swap or set up additional
swaps which do not have same size as primary
swap...

Cheers,

VK2COT

[Ken Englander]

Dennis and VK2COT,

I understand you both to be saying it is not necessary to have as much swap as I am allocating. I'd like to know what it is that I do not understand. HP-UX is requiring this swap space so it can "allocate" enough swap to match our memory requirements even though it is not using it. Are we doing something wrong?

It is my understanding and confirmed by HP consultants who have helped us that we must have sufficient swap reservable.

VK2COT - I would hardly consider managing 31 LUNs in order to set up 128 GB of swap simple. That strikes me as a small nightmare. That is simply my opinion.

Thanks for your input!

[James R. Ferguson · Accepted solution]

Hi (again) Ken:

> It is my understanding and confirmed by HP consultants who have helped us that we must have sufficient swap reservable.

Yes, this is true insofar as sufficient swap must be available to be *reserved* when a process is first spawned whether or not that process ever actually uses swap space. As I originally noted, turning on pseduoswap 'swampmem_on=1' in releases prior to 11.31 (or simply running 11.31 where the kernel behaves intrinsicly as if it were on), means that about 75% of your physical memory is counted "as" swap space for the purposes of meeting swap reservation requirements.

You have indicated that you have pseudoswap enabled. Hence you don't need to waste great amounts of disk apportioning it to device swap. As Dennis noted, you don't ever want to actually swap. Pseudoswap was designed for servers with large amounts of memory. You should be able to use a smaller number of device swap gigabytes than you have physical memory and still run everything you need. As always, YMMV (Your-Mileage-May-Vary), so emperical measurements on your part would be best :-)

Regards!

...JRF...

[Ken Englander]

James and VK2COT - I believe the issue we have relates to lockable memory, that is required by Oracle for the SGA. According to HP, swapmem actually decreases the amount of available lockable memory, but adding more disk swap helps with it. So, we are stuck with having rather large swap areas.

[Dennis Handly]

>I believe the issue we have relates to lockable memory, that is required by Oracle for the SGA.

I wasn't aware you had to be lockable for the SGA? Is it optional?

>we are stuck with having rather large swap areas.

This seems to defeat the purpose of having large memory systems.

[VK2COT]

Hello Ken,

Yes, your Oracle DB requirements seem to be
counter-productive in regards to RAM and swap
sizing. I have seen some other weird
applications that also commanded high amount
of swap (even though they did not use it).

Nevertheless, the effort to set up multiple
swaps on SAN is small in comparison to
the benefits. After all, how often do we add
or remove device swaps? It is typically
done via simple Shell scripts at the time
when the server is built and not revisited
during lifetime of the server (if the design
is good).

Spend a bit more time on this and other
issues when the server is built and then enjoy life without panics and performance
issues later on :)

Make computers work on boring and repetitive
tasks (they never complain) :)

Of course, as other said, there is no
bullet-proof solution that work in all
environments. The magic in our jobs is
to adapt and listen!

Cheers,

VK2COT

[VK2COT]

Hello,

One important item I forgot to mention (that I teach in HP-UX courses).

There is a significant difference in number of devices that could be used for swaps in HP-UX. Take a look at this kernel parameter:

HP-UX 11.11, maximum value nswapdev is 25

HP-UX 11.23, maximum value nswapdev is 25

HP-UX 11.31, maximum value nswapdev is 1024

If, for example, you need to set up 128 GB
of swap on HP-UX 11.23, you could not use 4
GB for primary swap and each additional LUN
(you would need 32 device-based swaps). So, primary swap of, say, 8 GB would be more appropriate (and hence, 8 GB for additional
LUNs).

If you use HP-UX 11.31, having 32
device-based swaps is easy and fully supported.

So, the moral of the story is: 4 GB for the primary swap needs to be taken with a grain
of salt. It depends on which version of the
HP-UX O/S is used too...

Cheers,

VK2COT

[Ken Englander]

VK2COT - You make a valid point about only setting up swap one time, but setting up dozens of LUNs does not make much sense if we never end up swapping anyway. Also, I have to consider that my recommendations can affect literally hundreds of systems. The impact of such a recommendation can be major in terms of time required.

Dennis - I may have inadvertently left out the fact that our goal is to use async IO and I think that is what requires the lockable memory. Oracle will run without it.