swap survey

[Dan Copeland]

I know this is a highly debated topic and it really depends on individual applications, but I'll ask anyway.

Of you that have a similar setup, what are you allocation for swap:

64 way SD w/ 750 MHz cpus and 128 GB mem
running one big oracle database

That's the scenario when we upgrade next month from a 32 way w/ 64 gb mem and 64 gb swap. We've always followed swap = mem but when you're talking 128 gb of swap (64 of which would have to be expensive emc disk in our case) it's worth asking the question again.

tia,
Frank

[Patrick Wallek]

There is no way I'd have 128GB of swap. In your scenario, I would HOPE that you never need to swap with 128GB of RAM.

I would make sure that swapmem_on is set to 1 (on) and from there I would define, maybe 4GB of swap. If you need to install Oracle or SAP, and the installer actually checks how much swap is set up, set up the appropriate amount temporarily, install your stuff, then delete the swap.

[Hai Nguyen_1]

Frank,

If there is no change in oracle memory usage then I guess you can keep your primary swap at 64GB. Keep in mind that you can always increase/create your swap space while your system is up and running without any downtime (assuming that you have available disk space for new swap space.)

You may want to adjust the percentages of kernel parameters dbc_max_pct and dbc_min_pct as well since HP states that you never need more than 300MB of Dynamic Buffer Cache (dbc).

Hai

[Paula J Frazer-Campbell]

Frank are you seeing any swapping on this system or can memory handle the workload.

I think with that much memory I would be tempted to do a trial and error test and perhaps start with 1/2 swap to mem. - monitor and then make your final decision.

Paula

[James Odak]

im thinking for the cost of that SD, a small VA SAN is just adrop in the bucket in cost

heh
but seriously, personaly i would leave the 64GB swap alone and review the kernel prarams for dynamic buffering and swapmem just to make sure you have enough but overkill on virtual memory being used for swap space

[A. Clay Stephenson]

If you have enough memory that you are reasonably certain that you are never going to swap then I would
allocate 1GB of primary swap,
no more than 12GB of additional swap (possibly even the dreaded filesystem swap at low priority, make that probably) and enable pseudoswap. That would be it. That let's you use all the virtual address space; there's no point in talking about optimum swap space layout for performance because when you start talking about swapping you ain't performing.

I do insist that any disk swap be mirrored (or at least in a highly reliable array).

The only time I ever configure more swap than that on hi-performance boxes is when the dumb install program for an application actually checks that I have 2-3x swap space. As soon as the install is done, I remove the unneeded swap space.

My rule is go ahead and make maxswapchunks large. Use a small amount of primary swap, enable pseudoswap, and monitor the system for a while. It's then so easy to add additional swap that you really shouldn't worry about it.

[Sridhar Bhaskarla]

HI Frank,

I agree with Patrick.

This is exactly the same reason why we have swapmem_on parameter.

It will come very handy for the systems that are having enormous physical memory.

Go ahead and turn on the parameter with 64 GB of device swap.

-Sri

[Sridhar Bhaskarla]

I disagree with 12 GB of device swap.

There should be a reason why you are going with 128GB of massive memory.

with swapmem_on parameter you will get 75% of physical memory = 96GB + 12GB for virtual memory.

However, when your application is completely up and running, there is a good chance that the memory reserved for swap from the physical memory will be taken out of the calculation (what you see as "used" of memory entry in swapinfo -t)and you may end up with a less virtual memory.

So, thought 2x memory is not valid anymore, I would recommend atleast 1/2 mem if you are enabling swapmem parameter.

-Sri

[Dan Copeland]

We never see any swapping. Heck, mem util hangs around 80-90% at most so only 5-15% of the disk space we allocated for swap is even being reserved (psuedo swap is enabled).

The harm of only having 16 gb of swap on a box w/ 128 gb of mem is only having 112 gb of reservable space for memory allocation, correct? Anything else?

Why is a mirror disk a must for swap? What happens of you reserved space on a disk that crashes?

tia,
Frank

[Pete Randall]

I've got to know, Frank: how much does a 64 way SD with 128 GB of RAM cost? It's probably equal to my IT budget for the next 10 years.

Pete

[Patrick Wallek]

If you have space reserved on a disk that crashes, you're screwed. No new procecesses will be able to start since it can't reserve any space. Processes that are currently running may hang, or your system may crash.

swap should always be mirrored. If you have a seperate dump area configured, this can be unmirrored, since it is not system critical.

[Sridhar Bhaskarla]

Hi (Again),

The discussion about 'whether you are going to swap' is completely out of question here. You will never with 128GB memory.

The question is about the virtual memory that you can have. With swapmem_on parameter, the utilization of physical memory impacts the amount of virtual memory you can have on the system.

So, unless you know how much memory you really use it, you have to be careful.


-Sri

[Dan Copeland]

Pete,

well we icod all but 8 cpu's on the expansion cabinet--so we didn't pay for them yet. But by the end of the year...

and you have to do the same for the failover superdome...

plus upgrade the dr superdome (not equal size 40% less capacity)...

you might be looking at your 15-20 year budget...

Frank

[Dan Copeland]

Sri,

Wait. I need clarification.

If my system was using 100% of 64 gb of memory, I still have 48 gb of swap space that I can reserve from "psuedo swap" plus any device swap I may have. Is this not true?

[Pete Randall]

Frank,

Yeah, I just ran some numbers (2001 prices - so it's probably far worse) and I couldn't believe what I was seeing. I think we're talking lifetime budget here. I'd love to just see the thing.

Pete

No points, please

[Sridhar Bhaskarla]

Hi Frank,

Please post your "swapinfo -t" output on the server where the mem is 100% used. I would like to see device+memory portions of it.

Thanks,
Sri

[Sridhar Bhaskarla]

HI Frank,

Look at the Doc ID: KBRC00010136.

In particular, I would like you to go through the answer of the first question "How is the used portion of pseudo swap determined...".

You will find it interesting.

-Sri

[A. Clay Stephenson]

Ooop, I misread your question - you are going to 128GB of memory. In order to address all your VAS, you need pseudoswap enabled which will allow you to use 3/4 of your physical memory + 32GB of device or filesystem swap. That will allow you to fully utilize your memory.

I would still use only 1-2GB of primary swap (you must have some on the boot disk and mirror it), then 32GB or so of device or filesystem swap. The reason that you mirror swap is so that a disk failure will not bring the machine down. Even if swap is not being used at all, if the swap device fails the machine is toast. If swap is mirrored and the disks are hot-plug you can replace them 'on the fly' and never need to bring the machine down.

[Brian M Rawlings]

Frank: great question, and thought-provoking responses. Regarding the swap HW, if you end up needing more storage (it doesn't look like you will), I agree that using expensive Symmetrix space is a last alternative. I would talk to HP and your reseller about adding a couple of, say, DS2100 JBOD chassis with 15,000 RPM drives, and mirroring them. Some config work would be needed to determine which Cells and/or I/O Chassis they would best be connected to, but the cost is trivial next to a couple of Symm disks...

Just a thought. Good luck with this one, sounds like fun!

--bmr

[Steven E. Protter]

HP people have told me that up to 16 GB of physical memory, swap should be twice physical memory.

With 128 G of main memory, you can get away with a lot less. Than that rule.

Thats a lot of disk space to be allocating for swapping that's simply not going to happen.

No matter how big the Oracle database is, there are limits on how much memory its going to be able to use.

From what I've seen on Oracle's web site, you need some swap space, but 64 GB might be too much. Take a look at swap usage on your current server and check usage.

If you've got 50G of swap and swapinfo shows you never use more than 10% then swap is too big.

I'm assuming this is an application mirgration, so you can do some analysis on the existing server and see what it tells you.

Here is an old fashioned data collection script that uses sar. It lets you measure and collect data in the background for a time period that you fix.

A hard look at the output will help you make an intelligent decision.

Steve

[Steven E. Protter · Accepted solution]

The promised attachement.