Decision to pseudo swap or not on HP-UX 11.23 Itanium

[Bryan Rooney]

Swapinfo not looking very good on the HP-UX 11.23 Itanium server running Oracle 10g R2.
Memory = 29647 MB (28.952148 GB)
Number of CPUs = 8
ia64 hp server rx8640

# swapinfo -ta
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
reserve - 1533284 -1533284
memory 30358576 25510316 4848260 84%
total 34552880 27043600 7509280 78% - 0 -

I've read the forum articles about decreasing the SGA, increasing physical RAM, and/or increasing physical disk swap space.
My question is if I allocate enough physical disk swap to cover my process creation needs then can I turn off pseudo swap with no ill affects? Am I missing some of the benfits that spseudo swap has to offer, beyond the swap reservation offering? HP memory white paper seems to indicate that the kernel competes with pseudo swap for system RAM such that memory allocated for psuedo swap is not available for the kernel to use. I'd rather I have some unusable disk space verses unusable RAM.

Additional info:
# ipcs -ma
IPC status from /dev/kmem as of Tue Sep 18 21:03:58 2007
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c018a --rw-rw-rw- root root root root 0 348 871 871 17:40:34 17:40:34 17:40:28
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 61760 871 871 17:40:30 17:40:34 17:40:28
m 2 0x4120928c --rw-rw-rw- root root root root 1 8192 871 883 17:40:30 17:40:28 17:40:28
m 196611 0x0c6629c9 --rw-r----- root root root root 2 17842600 2152 1413 15:47:07 15:53:31 17:41:49
m 4 0x06347849 --rw-rw-rw- root root root root 2 65544 2103 2158 17:41:51 17:41:49 17:41:48
m 32773 0x491808f0 --rw-r--r-- root root root root 0 22908 2108 2158 21:03:00 21:03:00 17:41:50
m 10584070 0xa35fe53c --rw-r----- oracle oinstall oracle oinstall 22 421486592 6869 24557 21:00:01 21:00:01 17:47:05
m 7 0xae1eacc8 --rw-r----- oracle oinstall oracle oinstall 52 21491658752 7604 24553 21:00:25 21:01:01 17:47:44

Slightly different form:
ID KEY NATTACH SIZE ATIME DTIME CTIME
0 0x411c018a 0 348 09-11|17:40 09-11|17:40 09-11|17:40
1 0x4e0c0002 1 61760 09-11|17:40 09-11|17:40 09-11|17:40
2 0x4120928c 1 8192 09-11|17:40 09-11|17:40 09-11|17:40
196611 0x0c6629c9 2 17842600 09-18|15:47 09-18|15:53 09-11|17:41
4 0x06347849 2 65544 09-11|17:41 09-11|17:41 09-11|17:41
32773 0x491808f0 0 22908 09-18|21:03 09-18|21:03 09-11|17:41
10584070 0xffffffffa35fe53c 22 421486592 09-18|21:00 09-18|21:00 09-11|17:47
7 0xffffffffae1eacc8 52 21491658752 09-18|21:00 09-18|21:01 09-11|17:47

How can I determine the proper amount of additional physical disk swap space to cover completely my process creation needs such that I have the possibility of turning off the pseudo swap?
I read that not enough is bad (my current situation) and that to much is bad as well.
Anyone know how to find the sweet spot?
Is there an easy calculation that can be made from the swapinfo output?

Thanks,
Bryan

[A. Clay Stephenson · Accepted solution]

First, you have to understand that pseudoswap (swapmem_on=1) is not swap; it doesn't compete with anything. It's nothing more and nothing less than a bit a kernel math hand-waving to allow 75% of physical memory to count as process reservation space. That's it!

Consider a system with 32GiB of physical memory (RAM) and 8GiB of actual (device or filesystem) swap space. If pseudoswap is not enabled (swapmem_on=0) then no more than 8GiB's worth of process space can be used eventhough you still have ~24GiB's of memory left. If you try to run more processes then you will get "Can't fork" errors. Now on that same system is pseudoswap is enabled, you can count 0.75 x 32GiB (RAM) + 8GiB actual swap space = 32GiB as process reservation space and you can fully utilize all of your physical memory. Pseudoswap is really intended for those systems which have less swap space than physical memory; if you have swap space >= RAM then pseudoswap serves no purpose and should be turned off as the system will actually run a little safer.

There really is no "sweet spot" for swap; it has no performance impact (other than limiting the number and sizes of processes which can be run). What you should really do is configure about 25% of RAM as primary swap (or a small 1GiB or primary swap space and enough additional swap to bring you up to 25% of RAM) and enable pseudoswap. You then monitor swap usage and add additional swap until you are satisfied. Your goal should be to never swap to any significant degree at all because the performance impact of swapping is so large.

[Theodore Pardike]

Bryan,

I believe this to be a very complex and controversial subject.
We are primarily a Sybase shop, so we have followed their recommendations in regards to Pseudoswap.

An old quote I have from a Sybase document:
"Pseudoswap is not a wise thing to use with the SQL Server and is
not something that Sybase recommends. Reader's digest
version is with psuedoswap enabled you may not have enough lockable
memory available for async IO and/or shared memory of the server
The SWAPMEM_ON parameter should be set to 0. There have been
cases where the system appears to have sufficient memory but the server
boots using standard UNIX I/O instead of asynchronous I/O,
due to the SWAPMEM_ON parameter having been set to 1.

Under HP-UX lockable shared memory defaults to 75% of physical memory.
This percentage may be adjusted via the kernel parameter UNLOCKABLE_MEM.
Its default value is 0, signifying that 25% of physical memory is
unlockable."

Also, see Don Morris's comment in this thread.
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1156937

I have set my swap to 1-1/2 times ram.
We have 16 GB RAM and 24 GB of swap.

Don't know if this helps in your situation, but thought I would throw it out there.

Ted

[Tim Nelson]

With 28GB of RAM I would configure at least 30-36GB of device swap. Your 4GB allocation does no good. HPUX allocates/reserves swap space up front so in order to use all 28GB of RAM you must have at a minumum 28GB of device swap.

Only 28GB of device swap is needed BUT if you approach the 28GB mark of used phyical you will experience "cannot fork" errors. Having a handfull of extra device swap will allow you to function without error until you can either make application adjustments or purchase more memory.

No need to turn off psuedo swap. As the system needs memory for the applications the psuedo allocations are reduced until none remain. Psuedo swap is the magic math that is currently allowing you to run more than 4GB of processes.

[whiteknight]

Hi Bryan,

Extracted from the whitepaper

Pseudo swap is used to increase the amount of reservable virtual memory. This is useful
when you can't configure as much device swap as you need. For example, say you have
more physical memory installed than you have disks available to use as swap: in this case, if pseudo swap is not turned on, youâ ll never be able to allocate all the physical memory you have.
Legend had it that if you had plenty of disk swap reservable (way more than
physmem), then also enabling pseudo swap could slow performance.

Doug - The author, spent a good few days trying to confirm this with benchmarks on some test systems and could not find any effect of pseudo swap on performance, unless your system is trying to reserve more
swap than you have device swap available to cover. So: pseudo swap can slow down
performance only when it "kicks in".
When your total reserved swap space increases beyond the amount available for device swap, if you do not have pseudo swap enabled, programs will fail ("out of memory"). If your total swap reservation exceeds available device swap and you do have pseudo swap enabled, then programs will not fail, but the kernel will start locking their pages into physical memory. If this happens, the number for "Used" memory swap shown in glance will go up quickly. We realize this is a real
head-spinner.

Rule of thumb: if you have enough device swap available to cover the amount you will reserve, then you don't need to worry about how this parameter is set. If
you need to set it because you're short on device swap, then beware that it could affect performance.

Bottom line is to try and configure enough swap disk to cover your expected workload.

WK
please assign points

[Bryan Rooney]

Thanks everyone for your replies:

As quoted from WK:
"If your total swap reservation exceeds available device swap and you do have pseudo swap enabled, then programs will not fail, but the kernel will start locking their pages into physical memory. If this happens, the number for "Used" memory swap shown in glance will go up quickly."

This to me means that the kernel and swap can complete for RAM under certain situations. Am I understanding ths incorrectly? I don't have any issue with keeping pseudo swap on if it will cover me in situation where more reservation occurs that I excpet.

[Theodore Pardike]

Bryan,

Oh yea...

If you plan to test this out, run "dmesg" before and after enabling/disabling
pseudoswap.

Here are the results of an old system we used to have:

swapmem_on from 1 to 0

Before:

Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 1179648 Kbytes, lockable: 842660 Kbytes, available: 977624 Kbytes

After:
Memory Information:
physical page size = 4096 bytes, logical page size = 4096 bytes
Physical: 1179648 Kbytes, lockable: 943012 Kbytes, available: 977624 Kbytes

You will see that lockable memory does increase after disabling pseudoswap.

Ted

[Steven E. Protter]

Shalom Bryan,

It would be nice to see some swapinfo -tam output to help you more.

Quick fix: What is dbc_pct_max ? It defaults to 50 and reducing it to say 10 can get you a lot of memory back cheap and easy.

To determine the calculations on swap needs, you may want to take a look at data on swap over time.

http://www.hpux.ws/system.perf.sh

This script measures various performance parameters including swap. I'm trying to shoehorn a vmstat command into the thing because people like that better than the one I'm currently using (HP wrote and I later enhanced the script).

Once you have some data you can make some caluclations no swap needs.

Doc:
http://h41156.www4.hp.com/education/upload/th/en/h4262sb.pdf
http://docs.hp.com/en/TKP-90203/TKP-90203.pdf


SEP

[Bryan Rooney]

Steven -

Thank you for your reply.
I included the swapinfo output in the original message. If you specifically want the MB version:
# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 1510 -1510
memory 29647 24945 4702 84%
total 33743 26455 7288 78% - 0 -

# kctune dbc_max_pct
Tunable Value Expression Changes
dbc_max_pct 5 5 Immed

We're using Oracle ASM against raw disk via Async I/O so can afford to set it low.

Top shows:
Memory: 8547408K (7390236K) real, 9050592K (7832500K) virtual, 7285684K free

[A. Clay Stephenson]

Top is about as useless as a bucket of spit for displaying memory usage. It only knows about memory used by processes and nothing about kernel data structures such as buffer cache. In your particular case, you are not yet at 25% of RAM for actual device/filesystem swap space so at a minimum, you should add enough secondary swap to get you there. The absolute bottom line is that if you are concerned about running out of virtual memory then forget about pseudoswap and add enough actual swap space to at least equal memory. Again, pseudoswap is only intended for those systems which have LESS swap than memory --- just as you currently do. I have many systems with much less swap than memory and they run just fine --- and never swap. I bought all that memory so that I would never have to swap.

If you do add swap, make certain that every dab of it is mirrored.

[Bryan Rooney]

Clay -

Thank you for your reply.
Thanks for your comment on the mirroring.
I came from AIX land where the swap was not mirrored by its mirrirong command, at least on the versions I worked on. I checked the native swap and it is mirrored. I'll make sure the secondary swap space is mirrored as well. Thanks again for the heads up on that.
Here's a snip of what I saw for primary swap:

# lvdisplay -v /dev/vg00/lvol2 |more
--- Logical volumes ---
LV Name /dev/vg00/lvol2
VG Name /dev/vg00
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 4096
Current LE 128
Allocated PE 256
Stripes 0
Stripe Size (Kbytes) 0
Bad block off
Allocation strict/contiguous
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c0t6d0s2 128 128
/dev/dsk/c4t6d0s2 128 128

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c0t6d0s2 00010 current /dev/dsk/c4t6d0s2 00010 current
00001 /dev/dsk/c0t6d0s2 00011 current /dev/dsk/c4t6d0s2 00011 current
00002 /dev/dsk/c0t6d0s2 00012 current /dev/dsk/c4t6d0s2 00012 current
.
.
.

[Bryan Rooney]

In my case the solution is to increase disk swap to match my physical RAM as I have enough disk space, at a minimum, to match the RAM.
As was noted in one of the replies, in the case of a mirrored system, the swap needs to be mirrored as well.