swap usage in V3

[Davis Paul]

Hi all,
Below is the swapinfo output.

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 2634 5558 32% 0 - 1 /dev/vg00/lvol2
dev 16384 2635 13749 16% 0 - 1 /dev/vg00/lvol10
dev 8192 2634 5558 32% 0 - 1 /dev/vg00/lvol11
reserve - 15251 -15251
memory 15555 4296 11259 28%
total 48323 27450 20873 57% - 0 -

This is in V3 system in which pseudo swap is enabled by default. But the above swapinfo out put shows that using swap space. How to instruct my system to use physical memory? This is oracle database system.


Thanks
Davis Paul.

[Don Morris_1]

You _are_ using physical memory. Overusing it, in fact such that the system had to push 7903Mb out to the swap device. (Used on the dev lines is actual disk space written to, not reservation -- it means you really honestly paged memory out).

You need to add more RAM or reduce the load. Reducing the load may be on the user side (did you set the SGA too large? Running too many DB connections?) or may be on the kernel side (what's filecache_min / filecache_max? Are you using direct I/O for the oracle transactions [in which case, the filecache is less important and should be reduced]? Are you using VxFS and have you checked the value of vx_ninode (see such links as http://docs.hp.com/en/5992-3364/apbs02.html )?

Your memory line does indicate not that much of this should be kernel dynamic memory [so the vx_ninode may be a factor, but not enough of one]. Those allocations will be considered locked and hence borrow from memory swap. With only 4296Mb of memory swap consumed, it isn't 7Gb of kernel consumption that's your problem -- so you should check file cache and user space loads first (and likely the latter -- too big of an SGA or too many clients).

[Michael Steele_2]

Hi

It is better to have everything in primary swap and yours is into 24 gb of 2ndary swap. Not good. However, 8GB is a lot of primary swap. You can play around with this formula and see if changing maxswapchunk helps.

Use this formula for to determine primary swap.

MaxSwap=maxswapchunks*1024*swchunk

Let say I increased maxswapchunk to 2048. BTW swchunk is set at 2048 (no need to change this). With that..
(2048*1024*2048)/1024=4194304 KB
ie 4096MB So the max amount of swap I can configure is 4GB.

vxinnode is not the problem here, his RAM is 16 GB. The forumla provided is for 3 GB.

More than likely your application, is this SAP, has set up a huge amount of cache within the SGA and it needs to be reduced.

I would look at the application being uses and see about reducing its cache, sga, shared memory consumption.

What is po in vmstat? Is is above 0? If so then you are certainly paging.

Adding more memory would make this a faster box if you can't reduce your 2ndary swap.

[Patrick Wallek]

>>It is better to have everything in primary swap

My personal opinion --- Baloney!

It does not matter HOW you lay out swap. If you start paging out to any significant degree, performance will SUCK!

To paraphrase A. Clay Stephenson: Worrying about swap layout is like rearranging the deck chair on the Titanic. It's basically pointless.

[Michael Steele_2]

Really? Since when was disk faster than RAM?

[Michael Steele_2]

Now really Patrick why? I always get confused on this arguement with Clay. He use to go against an HP white paper too as well and now you support him.

[Dennis Handly]

>Michael: Really? Since when was disk faster than RAM?

That's exactly Clay's and Patrick's points. You don't want to swap at all. By the time it decides to use primary or secondary swap, it is already orders of magnitude too slow.

[Michael Steele_2]

WELL REREAD THE THREAD - THE SUBJECT IS TWO SECONDARY SWAP LVOLS TOTALLING 24 GB!!

[Patrick Wallek]

>>WELL REREAD THE THREAD - THE SUBJECT IS TWO SECONDARY SWAP LVOLS TOTALLING 24 GB!!

No, the subject is "swap usage in V3."

The main question is: "How to instruct my system to use physical memory?" Which has been answered.

Unless you're reading something different than I am.

[Bill Hassell]

As mentioned before, whether you have 20 separate swap areas or one large one, when it gets used, something could not fit into RAM and processes had to be kicked out into swap space. 20 years ago, when RAM was measured in MB, having fast swap was really important because you couldn't afford to have 1000 MB of RAM and you lived with swapping. Today, the internal disks on most servers are much, much slower than fibre disk arrays so technically, using primary swap on vg00 is not a good idea -- assuming you have to page out a lot. In that case, a small primary area (2-4 GB) with a large LUN from a high speed array would be a better choice. But that's because additional RAM is too expensive for your system and therefore maximizing swap speed is the primary objective.

Of course, disk speeds are meaningless if you are not using swap space except for reservation. You don't want processes to ever be paged out if performance is a important.

Now that said, you might have a database engine that fits nicely into RAM but there are 500 interactive users all around the world. They type some data into the system every few minutes but their programs are idle (waiting to do something) for at least 50% of the workday. In that case, those programs can indeed be paged out and then recalled when necessary. There is a slight delay (a second or two) as each program is brought back but no delay while it is in use.

[Michael Steele_2]

Patrick

there is 8gb of primary swap in ram and 16gb of 2ndary swap on disk. You're comment is "..all swap is baloney...".

Don't you want to provide some kind of arguement that puts all the 2ndary swap into the primary swap by buying more ram.

Cuz I don't see that explanation at all.

[Davis Paul]

Hi All,

Thanks for your quick response. My question is not yet answered.

In the swapinfo output above only 32% of physical memory is used.System is using swap even free physical memory is available. I need my system to use maximum physical memory then swap.Is it possible?.how?..

[Patrick Wallek]

>>there is 8gb of primary swap in ram and 16gb of 2ndary swap on disk.

What?!?!?! "swap in ram"? What the heck are you talking about? There is no such thing!

The PRIMARY SWAP space (/dev/vg00/lvol2 - 8GB) is on disk. The other swap space (/dev/vg00/lvol10 and lvol11 - 24 GB total), which is what I assume you term secondary, is also on disk.

All 3 device swap areas above are also set at a priority of '1' so they will all get used equally.

If you are talking about 'pseudo swap' as swap space in RAM, then that is not swap at all. It is a calculation to make the system **think** there is more swap than there actually is.

>>In the swapinfo output above only 32% of physical memory is used.

If you are looking at the 'memory' line of swapinfo, that is incorrect. The memory line indicates how much 'pseudo swap' is used. It is not an indication of how much physical RAM is used.

>>I need my system to use maximum physical memory then swap.Is it possible?.how?..

It will do that all by itself. There is no parameter to set to tell the system to use 75% RAM before swapping or 100% RAM before swapping.

With the snapshot posted above, it is possible that RAM usage decreased so that you are not using 100% RAM, but until the pages that have been swapped out to disk are requested again, they will remain on disk.

[Michael Steele_2]

dev 8192 2634 5558 32% 0 - 1 /dev/vg00/lvol2

So this isn't swap. Right. That's your explanation, right? lvol2 is not swap? Right?

God has HP support gone downhill. What a joke.

[Patrick Wallek]

>>dev 8192 2634 5558 32% 0 - 1 /dev/vg00/lvol2

>>So this isn't swap. Right. That's your >>explanation, right? lvol2 is not swap? Right?

NO, that is NOT my explanation. If you go back and READ what I wrote, you will see that.

LVOL2 is definitely swap. BUT it is DEVICE swap. It is on DISK. It is NOT in RAM.

I am still VERY CONFUSED by your "swap in RAM" statement. That makes absolutely no sense to me.

>>God has HP support gone downhill. What a joke.

You think I work for HP? You are mistaken!

[Michael Steele_2]

My personal opinion --- Baloney!

It does not matter HOW you lay out swap.

Did you say all swap is baloney or not?

[Patrick Wallek]

>>My personal opinion --- Baloney!

>>>It does not matter HOW you lay out swap.

>>>>Did you say all swap is baloney or not?

No, I did not say "all swap is baloney". What I said, or tried to say, is that having all swap as PRIMARY SWAP, ie. one large swap volume, is baloney.

You said: "It is better to have everything in primary swap"

That is what I consider baloney.

[Michael Steele_2]

Just so you understand, Patrick, you need lvol2 to boot up, right? And you can't boot up without swap, right? So I guess its not all baloney is it? You need swap to boot up.

Yes or no?

[Patrick Wallek]

>>you need lvol2 to boot up, right?

Yes

>>And you can't boot up without swap, right?

Yes.

>>So I guess its not all baloney is it? You need swap to boot up.

I never said you do not need swap. I never said to get rid of LVOL2.

Again, you said: ""It is better to have everything in primary swap"

That is the statement I disagree with.

For all I care you **COULD** have 24 GB of primary swap (LVOL2) or you could 1 GB of Primary swap (LVOL2) and 23 GB of secondary swap. I don't care. BUT, worrying about what layout will give the **best performance** when paging out is what I have a problem with.

And you still have not explained your "8gb of primary swap in ram" statement.

[Michael Steele_2]

It is better to have everything in primary swap and yours is into 24 gb of 2ndary swap. Not good. However, 8GB is a lot of primary swap. You can play around with this formula and see if changing maxswapchunk helps.


Keep it up Patrick.

[Davis Paul]

Dear Michael Steele,
There is no maxswapchunks parameter in 11.31 hp-ux system. I cannot play with the formula.

Davis Paul.

[Michael Steele_2 · Accepted solution]

Hi

Sorry, gave you 11.11 information. In 11.23 and 11.31 refer to swchunk

http://docs.hp.com/en/B2355-60130/swchunk.5.html

Thanks for catching that.

[Davis Paul]

Dear Michael Steele,

# kctune |grep swchunk
swchunk 65536 65536
My question is still remains unanswered. Above is the swchunk value set as maximum. Can you tell me how can I use maximum physical memory ? Here in the swapinfo output reserve memory is "reserve - 15251 -15251". Why this much memory is reserved ?

[Don Morris_1]

Because it was required for the virtual objects your applications/workloads created.

Read over http://docs.hp.com/en/5965-4641/ch01s09.html (or any of the many, many other times this question has come up). You want the sections on "swap reservation" and "pseudo-swap".

As I said before, you are not only using all of physical memory -- you are *over subscribed* on physical memory (which is why you're consuming actual disk swap).

[Michael Steele_2]

swchunk = number of blocks that will be placed on one swap device


Minimum: 2048 blocks

Maximum: 65536 blocks

Note that the minimum (or default) value of swchunk therefore allows 4,096 TB of swap space.

The total bytes of swap space manageable by the system is swchunk * 1 KB * 2,147,483,648 (the system maximum number of swap chunks in the swap table).

############################

Your swchunk is currently 65526, the MAX, and it is mis configured since you are stating you now have 128 TB of swap.

Swap= 2,147,483,648 * 1024 * 65536 = 144115188075855872 / 1024 ** 5 = 128 TB of swap

############################

The default swchunk value is 2048 and it allows for 4 TB of swap.

Swap= 2,147,483,648 * 1024 * 2048 = 4503599627370496 / 1024 ** 5 = 4 TB of swap default

############################

By default then, without changing anything, your swap will allow for 4 TB of swap. Well this is insane. You want as much in RAM as possible in order to eliminate disk I/O.

a) change your current swchunk back to 2048.
b) Get as much swap into RAM as possible.
c) Eliminate all 2ndary swap.

Note: This is what you have now - 2ndary swap is written in round robin interleave fashion and not similtaneously or in parallel. I.e., Going up one block of primary swap, then one block of 2ndary, then a 3rd block of 2ndary - Coming down one block of 2ndary, then another block of 2ndary then a 3rd block of primary.

Verses the misunderstanding that one block of primary and one block of 2ndary and a 2nd block of 2ndary are all written similtaneously in parallel. This is not how your swap writes. It does not write in parallel!

So even this algorithm tells you that the use of 2ndary swap is less efficient.
##########################
total 48323 27450 20873 57% - 0 -
Right now you are using 27 GB of swap with 48 GB configured.
Sigh.
Run this command for me and total up the VSZ column, the first column and post or attach both please. I can't believe this total will come close to 27 GB but we'll see.
UNIX95=ps -ef -o vsz,pid,comm | sort -rn

d) Total virtual memory consumed by all processes.