Pseudo-swap or device swap with Apache?

[Michael Simone]

I'm running 11iv2 on an rx7620 Itanium system, with Apache 2.0.51 (mod_perl, etc). This is a web server which handles a lot of traffic, often in bursts. During the heaviest bursts, I've been running into a swap problem. The server is configured to use pseudo (memory swap), as well as device swap (4Gb). Under heavy load, it was using 80% of RAM for swap(~10Gb), plus all of the device swap. In a pinch, I added 4Gb of filesystem swap in some space I had free.

My question is, is it optimal to be using pseudo-swap in the first place, or should I try configure device/filesystem swap to be more than possibly needed? I wish I'd learned about this before turning this server up, but like most things, you learn the hard way.

My Apache config allows up to 256 processes--looking at them in Glance, they appear to use about 40Mb of memory each (not sure how accurate that RSS column is). So that would tell me that at max capacity, Apache alone needs 10+Gb of memory, so wouldn't it make sense to put swap on disk primarily?

I can't tell from other posts whether the swapmem_on parameter SHOULD be on or off. Any recommendations for this type of environment? I can certainly find the disk space to give it 20+Gb of device swap.

[Jeff Schussele]

Hi Michael,

First you should dispell the notion that pseudo-swap is actually "used". It's only purpose is for reservation space. Reservation is like a placeholder. Every process, when it starts, must make a reservation in swap equal to it's memory size. IF you try to start more total processes, memory-wise, than you have swap space (including pseudo), the system will not allow them to start.
Only actual swap space is ever used to actually activate/deactivate processes.
The fact that you have pseudo-swap confirms that swapmem_on is set to 1. If it was 0 you'd have no pseudo swap.
Now to your problem - the fact that you do indeed swap under load indicates you need to do one of two things:
1) Add more RAM
2) Decrease the RAM demand from the current processes - which may include lowering process count.
You only swap when all available RAM is already in use.
You should check just how much disk buffer cache you're using - they're the dbc_max_pct & dbc_min_pct kernel parms. They default to 50 & 2 which means half your RAM may be used simply to cache disk I/O. Normally 400 to 800MB of cache is plenty. This alone may get you out of the woods.

HTH,
Jeff

[Michael Simone]

Okay, that helps a bit. If the pseudo-swap isn't ever used, but is reserved for each process spawned, is the 10Gb of RAM I'm seeing occupied right now actually not being used?? So that memory isn't available for real use since it's occupied by reservations. Right?

So assuming I can't add more RAM (would love to, but not in the budget), wouldn't it make sense to turn off swapmem_on and let any swapping that needs to occur happen to disk? Yes, I'm swapping under heavy load, but if my processes can actually access that memory (Apache is the only active application on this box), they won't need to swap MUCH. And when they do, it'll do it to disk. If I currently have 8Gb of REAL swap (device and filesystem) enabled, and I'm using about 50% of that now under max load, then I actually have enough swap. My problem is that by having swapmem_on, I'm also allocating all that RAM which never gets used for swap?

Hope I'm getting this.

[Jeff Schussele]

Well...let's first look at just *how* that memory's being used. Run up glance & look at the memory bar. Everything between the "S" is system/kernel usage. Everything between "U" is actual user programs like your Apache. Everything between "B" is disk buffer cache.
How big is the "B" section? If it's close to half then a lot of it is essentially being wasted.
Now let's move on to how swap is actually used.
When the system reserves all the device swap & has to start reserving to pseudo-swap the kernel locks those pages into memory and that's not a good thing because then they cannot be swapped out to disk when the system comes under memory pressure. This impacts performance to some extent.
Now IF you set swapmem_on to 0 then - as I stated earlier - you'll ONLY be able to start 8GB worth of processes because they must reserve and all they can use is the 8GB of device swap. So you'll essentially have no access to the other 2GB of RAM. It'll be useless.
SO what you can do now is up your device swap to at *least* 10GB so you can use all your RAM.
But you definitely need to see just what's using memory first & if it's the buffer cache scale it back to that 400-800MB range.

HTH,
Jeff

[Michael Simone]

I'm definitely not wasting memory on buffer cache--learned that one a long time ago. I have db_min/db_max set to 5/15.

Nearly all the memory being consumed during that time was "U", so therefore the actual user processes (Apache).

The system actually has 12Gb of RAM, so I would want to configure at least 12Gb of device swap (preferable to filesystem swap, correct?).

If I did that, would it make sense to set swapmem_on to 0, or am I going to take a performance hit? Guess that's hard to say.

The system finally came down off the plateau late last night as traffic died down, and swap usage is back down. About 2.5Gb of RAM is being "used", and only 115Mb of the device swap. What I noticed this morning though was that disk utilization was pegged at 100%, almost all "V" (Virtual Memory). Apache had been recycling processes all night (have it set to die at 5000 requests), and it was down to just 50 processes to handle the lighter traffic, so I don't fully understand why it would have been swapping those processes. But stopping and restarting Apache cleared up that situation. Maybe just a leftover from hitting that condition earlier in the day.

I guess I'd like to know what the conventional wisdom is IF I have enough disk space to create more than enough device swap. This system is currently using just its internal disks for all my volumes, but I have access to an XP array over dual fiber channel, so I could create the swap there. And if I did so, would it then make sense to turn off swapmem_on, so RAM is not allocated? Or am I still not getting that concept?

[Ted Buis]

There is a good discussion of this in the book "hp-ux 11i internals" by Chris Cooper and Chris Moore starting on page 194. What you want to do is eliminate page-outs, by adding more RAM. If that can't be done you might do better without pseudo-swap, because then all memory is exposed to being paged out rather than just a small section of RAM that gets hit unfairly and arbitrarily. Remember that psuedo-swap is similar to locking processes in memory, except that it is done rather arbitrarily, not according to a plan.

[Steven E. Protter]

Must be a lot of apache processes runnning.

Let me say that what you are looking for is actual paging to disk. This slams performance and should be avoided by adding memory.

If you reserve swap but never actually use it, then worry not and move on to other issues.

swapinfo -tam

Run sar scripts(attached) and look for acual paging. Or vmstat.

SEP

[Michael Simone]

Like I said, Apache maxes out at 256--thought that was the conventional maximum for a busy server. Obviously, it would benefit me to add more RAM, which would eliminate paging to disk, and I'll try fighting that budget battle. But in the meantime, it sounds like my best option would be:

a) Create device swap of more than 12Gb, using what I think will be my fastest disk device (probably the XP over dual fiber)

b) Turn off pseudo-swap (swapmem_on=0) so that it's free to page out what it needs and doesn't lock anything into memory.

If I have more swap than memory, isn't it best to just let it swap when necessary, knowing that if and when I hit that point, I'm going to take a hit? Seems like having pseudo-swap on when you HAVE enough swap to cover your RAM is unnecessary.

[Ted Buis]

Device swap has much better performance than file system swap, so you are correct. However, you can also spread your swap space across multiple disks to improve swap performance, however, if you can afford to do that, you could likely afford more RAM.

My read of the Internals book is that if you must page out, then psuedo-swap will lower overall system performance. It is intended for the case where you don't want to have as much device swap as RAM, and becasue you do have large RAM, you don't do page-outs often.

[Bill Hassell]

Apache is a highly interactive program with long "think" times between individual requests. This is actually an ideal situation for limited memory and lots of occupied swap space. First, processes are not swapped out but rather small pages are paged out to the swap area to make room for a newly activated process. If multiple processes need to run continuously (constant CPU cycles) and they all will not fit into RAM, then page thrashing will occur where competing processes are constantly paged out and back in.

Apache is different. Most of the 256 processes are idle and activated only when an incoming request is routed to the process. And many of these processes have been deactivated due to the inactivity and portions of their RAM areas have been paged to the swap area. When the paged out process wakes up again, the OS moves the needed pages back to RAM (possibly moving some pages from another deactivated process to swap) and starts the process. This may take a second or two and then the process runs full speed.

Now whether this activity creates visible delays is a matter of perception by users. With today's disks, spreading swap areas among different disks for performance reasons has very little effect unless the paging rate (vmstat's po column) is thousands of pages per minue. And by using the XP for a swap area, you will have the benefit of enormous buffering.

Pseudo swap should have no visible performance effect. Instead, turning pseudo swap off will mean that you MUST have more swap space than RAM, perhaps 2-3x. The reason is that every process must have it's data area reserved in the swap area regardless of whether it is ever used. If swap is equal to memory, then almost no swap space will be used and process virtual memory will be the size of swap. When swapmem_on=1, virtual memory is swap + 3/4 RAM. If you can grab 20-30Gb from the XP, leaving swapmem_on=0 should not be a problem. But try the setting both ways.