concern about file system swap configuration

[Marc Ahrendt]

i am rethinking a configuration. currently i have used LVM to mirror all the LVs on a boot disk to another boot disk, except for a newly added LV. this newly added LV i setup to be used for file system swap and put half the LEs on one of the boot disks and the other half on the other boot disk. i did not mirror it becuase this swap will never actually be used to page to/from. it is just used to make the OS see more available swap.

now my concern is when a boot drive fails. how will i be able to fix this swap LV, since is not mirrored, without interrupting the running OS. the system has OnlineJFS, but i am not sure that is going to be good enough to rebuild this file system. my concern is that i cannot unmount the file system, because i cannot turn off this swap while the OS is still running. i am wondering if i am going to have to reconfigure the swap to be mirrored, but the cost for that is using double the PEs ...which seems a waste since this file system swap will never actually be used.

any advice would be appreciated, marc

[Steven E. Protter]

I've seen and agreed with a lot of advice that says not to mirror swap. I actually do it on practice systems, but here is an idea to help.

Instead of mirroring swap, have two swap areas on two disks with different priorities. If you lose the boot disk, you still boot but have a bit less usable swap, which should be okay except on a seriously overworked server.

The theory on not mirroring swap is that its kind of a waste of i/o. There are those that say mirror everything and thats a valid strategy too.

You can get away with either strategy. If disk space is short not mirroring swap can stretch things a bit and let you go with two smaller swap areas instead of one big mirrored one. That is also a potential performance benefit.

SEP

[Devender Khatana]

Hi,

Approximately how much is the size for this file system. The OS will give error while activating swap on this file system if any one of the disks is not available as in both cases the file system will have missing PEs. That should not be any problem as there should not be any impact apart from this in system fuctioning and the system should be functional normally with reduced swap space.

Still the recommended configuration will be to have this swap LVOL mirrored across two disks but this will atleast require one reboot.

HTH,
Devender

[James R. Ferguson]

Hi Marc:

In my view, a boot disk should contain only primary swap space and the standard mount points (/usr, /var, /opt, /tmp, /, /stand). Any secondary swap (and my preference is device swap) should be on a separate disk for performance and configured at a higher priority (0) than that of your primary swap (priority 1, lower).

That said, when I mirror a boot disk, I mirror everything. I want to be able to bring a system up easily and cleanly.

You should generally enable pseudo swap by setting the kernel parameter 'swapmem_on' to one (1). This allows the kernel to perceive that you have 75% more physical memory than you have, counting that 75% of the physical space for the requirments of starting new processes. No actual, paging (in or out) occurs, though. This is a good way, particularly on large memory configurations, to save physical disk space while meeting swap reservation policies.

You noted that you have Online JFS. That being the case, you would not need to unmount a filesystem in order to expand it.

I'm not clear from you post if your boot disk consists of more than one physical disk, either. You noted that you configured your "...newly added LV...setup to be used for file system swap and put half the LEs on one of the boot disks and the other half on the other boot disk.". If your boot disk truly consists of multiple physical volues, I suggest that you re-Ignite your server and configure a boot disk with one physical volume whose volume group's logical volumes are all completely mirrored to another physical disk. Declare another volume group(s) for any "application" data and any secondary swap (device or filesystem) as necessary. Configurations like that make operating system upgrades, disaster recovery much easier to accomplish.

Regards!

...JRF...

[A. Clay Stephenson · Accepted solution]

If you are going for high availability then ANY swap space should be mirrored (including LVOL's which house filesystem swap space). You've already discovered the reason. You are not going to be able to replace failed disks "on the fly" otherwise; it matters naught whether the space is actually being used. My rules are very simple: 1) ALL swapspace should be mirrored; 2) dump space should never be mirrored; 3) Swap space has nothing to do with dump space. 4) Only in extremely rare cases (those being systems with very small amounts of physical memory) should 100% equivalent dump space to memory be required; in almost all cases ~ 25% will more than suffice.

[Marc Ahrendt]

thx Steve, yet my concern about your point is that mirroring this file system swap is "not" a waste of I/O ...its not going to be used on my particular system (the priority is set to 10 and if all the device swap is used up then this system would be thrashing so badly that any additional swap would have no benefit anyway), also i cannot afford reboots (i need to repair failed drives on-the-fly)

thx Devendar, yet reduced swap on a system after a reboot can have a serious impact on a system ...in particular mine requires alot of Java processes that are sadly big memory hogs that reserve much swap at startup, also my main concern is not to reboot to repair a mirrored boot disk

James, yes i do use one PV for the boot LVs then mirror them to another PV ....thus was my question about "cheating" with this secondary file system swap with a priority of 10 (by cheating i mean not mirroring it in order to make it double the size but hoping that if the main boot PV failed i would still be able to rebuild it on-the-fly by resync'ing the LVs ...and maybe use fsadm to address the unmirrored/corrupt file system swap without losing any swap and OS uptime)

Clay, you got the main point of my question ...but i did not give you a 10 because i was hoping someone would tell me why/how i cannot repair this swap file system on-the-fly. i do not care that the file system gets corrupted since i am 100% certain that the OS will never page in/out to this file system, since i have 3GB of device swap at priority 1 and 4GB of RAM with the bogus 75% psuedo swap turned on as well. i was hoping someone would say that on a boot PV failure i would be able to resync the LVs as commonly done, but then for the now corrupt LV used for file system swap (since it is not mirrored) it would not impact/reduce "swapinfo -tam" output. the OS would not know that it was corrupt and that maybe (here is what i wanted to know since i am unsure about) i could repair/rebuild the file system without unmounting it (since i cannot unmount it while its being used as swap)

[A. Clay Stephenson]

There is no "magic bullet" to replacing a disk with filesystem swapspace unless it is mirrored --- and it doesn't matter if the space is actually in use. The OS is simply not designed to run without swap space that it has been told exists. You made an unwise (or possibly uninformed decision) and now you must live with the consequences of that decision.

[Steven E. Protter]

Some things can't be done on the fly.

To remove swap, you normally have to boot the system.

If you do a cold OS install and try to set swap less then half main memory, you get a warning.

If during this repair process swap drops below that level your system might halt suddenly.

The goal of system setup in today's era of less expensive hardware(ie memory) is to have a system that NEVER swaps.

Swap space will get reserved each time a process starts, thats why you need some on even a huge,idle system with 32 GB of ram. If you are paging regularly, as in not just reserving swap but taking processes from main memory to swap an d vice versa, you have a capacity issue that can be solved by more servers or more memory in the server.

I would agree with your initial analysis that mirroring swap may be considered a waste as in it uses twice the PE's.

Good Luck,

SEP

[Marc Ahrendt]

Clay and Steve, i am with you on your explanations ...as i know once you add swap it can only be removed by a reboot. thx again for the additional feedback!