Swap Utilization 90% + all the time

[Darren Etheridge_2]

We have an oracle box that runs very well but the Swap Util is 90% or more all the time, but it never uses the physical swap space.
This machine runs 6 instances of Oracle with plenty of disk space. I have attaced the kernel parameters... Any and all help would be greatly appreciated.

Darren

[PAL:/]swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 2048 0 2048 0% 0 - 1 /dev/vg00/lvol9
reserve - 6144 -6144
memory 9795 9290 505 95%
total 15939 15434 505 97% - 0 -

CPU Util S SR RU | 28% 22% 64%
Disk Util F F | 57% 26% 100%
Mem Util S SU UB B | 90% 90% 91%
Networkil U UR R | 97% 96% 98%
--------------------------------------------------------------------------------
MEMORY REPORT Users= 3
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 249 123887 51.8 62.4 1725.2
Page In 25 48244 5.2 24.3 1007.8
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 1.0mb 0.0 0.5 172.7
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 67 0.0 0.0 10.4
VM Writes 2 780 0.0 0.3 11.2

Total VM : 11.5gb Sys Mem : 1.93gb User Mem: 6.96gb Phys Mem: 11.9gb
Active VM: 4.33gb Buf Cache: 1.79gb Free Mem: 1.25gb Page 1 of 1

[Alex Lavrov.]

It looks that you still have free memory, so there is no need to use swap ...

[Darren Etheridge_2]

So you are recommending that I turn swap off in the kernel?

[Alex Lavrov. · Accepted solution]

No .... to be honest I don't see any problem right now. Do you have performance problems?

[Darren Etheridge_2]

Not yet.. but I keep getting advisory messages:

RED ALERT 05/05/05 11:08:22 Global swap space is nearly full

[A. Clay Stephenson]

Note that your page outs are zero; you are not under memory presssure. The "kernel" swapping you are referring to is pseudoswap and in your case, if you set swapmem_on=0, you will then only be able to run 6GB's of processes because that is all the swap space you will have. On systems with more physical memory than swap space ( a common situation today), pseudoswap allows the kernel to count 75% of physical memory + device/filesystem swap when calculating total virtual memory. Pseudoswap is nothing more and nothing less than kernel "bookkeeping"; it's not swap at all.

[Alex Lavrov.]

Daren, it means that when it will reach 100%, physical swap will be used. It's a normal condition, so you can ignore the "RED ALERT".

As Mr. Stephenson stated, "total" is a pseudo-swap, it's nothing. It's there to fool applications :)

[Darren Etheridge_2]

Thanks... I feel better now :)

[Don Morris_1]

I wouldn't completely ignore this.. your total reserved swap is at 97%.. that means that you only have 505Mb of virtual address space left (keep in mind that HP-UX reserves swap for all virtual address space [text "reserves" from the backing file, though]... if you fail the reservation, the allocation fails). If you get up to 100% then you're unlikely to be able to start new processes (and memory allocations in existing ones will fail). If your machine is truly in a steady state, you're ok.. if not, consider adding some more swap to give yourself additional virtual address space.

[Darren Etheridge_2]

Add swap space in the kernel or allocate physical disk swap space?

[Alex Lavrov.]

Well, in order not to repeat others, here you go, some usefull discussions about swap size:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=26968

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=456768

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=763197

http://docs.hp.com/en/TKP-90202/ch08s01.html#d0e2720

[Don Morris_1]

Add physical swap space (disk or filesystem). You can't get the kernel to revise the pseudo-swap calculations without adding or removing physical memory which means you either have to reboot or build a time machine (since OLA isn't in HP-UX yet).

[Darren Etheridge_2]

Well my Oracle system just crashed and gave everyone "Out of Memory/Space errors", so I had to do some changes and see what happens. UGGG

[Alex Lavrov.]

That's weird, because according to your post, you had almost zero paging, which indicates that you have enough memory to keep all the pages in ....

[Alex Lavrov.]

What Oracle version do you run on your system? Can it be 32bit Oracle on 64bit OS?

If so, you can get these errors if yo don't use memory windows ....

[Darren Etheridge_2]

64 bit

[Andrew Cowan]

I've seen this behaviour with Oracle before. I think its related to way in which is tries to create a pre-warmed SGA? Oracle seems to pin large amounts of paging/swap space in case it needs it for a query. I'm afraid that I have never found any solution other than to allocate a large amount of swap, even though it never seems to get used.

The best way in Oracle to avoid massive memory drain is to avoid queries that involve full table scans, by judicious use of indexes.

[Don Morris_1]

It isn't paging that's the problem. The physical usage is low enough (and most of it is pinned/locked in memory anyway) that vhand doesn't need to actually page out anything. The virtual usage is consuming all of the available virtual address space == (pseudoswap + swap).

If you do a brk(), create a shared mmap/sysV segment, whatever of X Mb... the system will reserve X Mb of swap so it knows that if the time ever comes for vhand to do something with it, it will have the room to do so. If that reservation fails, the allocation request is also failed (with ENOMEM) which is what happened to Oracle, apparently. You can screw up a system without ever using any actual RAM by just starting programs that make huge malloc's... hence the max*siz tunables, process rlimits to prevent malicious users (or naive undergrads).

Adding more swap space increases the virtual address space for such allocations.

[Andrew Cowan]

Don, your answer is very interesting, however what puzzles me (considering your explanation) is why the system is show "Page-ins" and "Page_Faults" and yet no "Page-outs".

I'm definitely not a kernel expert, however I was always taught that it is normal for a system to "Page-out", however if it "pages-in" constantly then you have a memory problem? Also the number of "Page Faults" suggests that VHAND has "Paged-out" because the requested data no-longer resides in RAM. Any ideas?

[Ted Buis]

There is a great discussion of Pseudo-swap in HP-UX 11i Internals by Chris Moore and Chis Cooper (or is it Cooper and More). There are positive and negatives to psueudo-swap, but the bottom like from my understand is that it is great as long as you never need to use physical device swap, because the RAM that is assigned for this pseudo-swap cannot be paged out ever to device swap. Thus the small number of pages that are left that can be swapped to disk suffer an unfair burden. Assignment of these pages is arbitrary, so this isn't at all good if paging is done. The good side of Pseudo-swap is you don't waste disk space that never gets used if you have sufficient RAM. However if you start getting over 90% utilization of your total virtual address space, then you can either buy more RAM, add addition disk device swap space, or both. Once you run out of virtual address space, no new processes can be launched, and processes that need more either hang or die. If the system needs more space and can't get it then it hangs or crashes.