topic Re: Swap Utilization 90% + all the time in Operating System - HP-UX

Swap Utilization 90% + all the time

Darren Etheridge_2 — Thu, 05 May 2005 10:34:20 GMT

We have an oracle box that runs very well but the Swap Util is 90% or more all the time, but it never uses the physical swap space.
This machine runs 6 instances of Oracle with plenty of disk space. I have attaced the kernel parameters... Any and all help would be greatly appreciated.

Darren

[PAL:/]swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
dev 2048 0 2048 0% 0 - 1 /dev/vg00/lvol9
reserve - 6144 -6144
memory 9795 9290 505 95%
total 15939 15434 505 97% - 0 -

CPU Util S SR RU | 28% 22% 64%
Disk Util F F | 57% 26% 100%
Mem Util S SU UB B | 90% 90% 91%
Networkil U UR R | 97% 96% 98%
--------------------------------------------------------------------------------
MEMORY REPORT Users= 3
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 249 123887 51.8 62.4 1725.2
Page In 25 48244 5.2 24.3 1007.8
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 1.0mb 0.0 0.5 172.7
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 67 0.0 0.0 10.4
VM Writes 2 780 0.0 0.3 11.2

Total VM : 11.5gb Sys Mem : 1.93gb User Mem: 6.96gb Phys Mem: 11.9gb
Active VM: 4.33gb Buf Cache: 1.79gb Free Mem: 1.25gb Page 1 of 1

Re: Swap Utilization 90% + all the time

Alex Lavrov. — Thu, 05 May 2005 11:18:23 GMT

It looks that you still have free memory, so there is no need to use swap ...

Re: Swap Utilization 90% + all the time

Darren Etheridge_2 — Thu, 05 May 2005 11:20:27 GMT

So you are recommending that I turn swap off in the kernel?

Re: Swap Utilization 90% + all the time

Alex Lavrov. — Thu, 05 May 2005 11:24:56 GMT

No .... to be honest I don't see any problem right now. Do you have performance problems?

Re: Swap Utilization 90% + all the time

Darren Etheridge_2 — Thu, 05 May 2005 11:27:10 GMT

Not yet.. but I keep getting advisory messages:

RED ALERT 05/05/05 11:08:22 Global swap space is nearly full

Re: Swap Utilization 90% + all the time

A. Clay Stephenson — Thu, 05 May 2005 11:28:35 GMT

Note that your page outs are zero; you are not under memory presssure. The "kernel" swapping you are referring to is pseudoswap and in your case, if you set swapmem_on=0, you will then only be able to run 6GB's of processes because that is all the swap space you will have. On systems with more physical memory than swap space ( a common situation today), pseudoswap allows the kernel to count 75% of physical memory + device/filesystem swap when calculating total virtual memory. Pseudoswap is nothing more and nothing less than kernel "bookkeeping"; it's not swap at all.

Re: Swap Utilization 90% + all the time

Alex Lavrov. — Thu, 05 May 2005 11:34:00 GMT

Daren, it means that when it will reach 100%, physical swap will be used. It's a normal condition, so you can ignore the "RED ALERT".

As Mr. Stephenson stated, "total" is a pseudo-swap, it's nothing. It's there to fool applications :)

Re: Swap Utilization 90% + all the time

Darren Etheridge_2 — Thu, 05 May 2005 11:43:49 GMT

Thanks... I feel better now :)

Re: Swap Utilization 90% + all the time

Don Morris_1 — Thu, 05 May 2005 12:33:50 GMT

I wouldn't completely ignore this.. your total reserved swap is at 97%.. that means that you only have 505Mb of virtual address space left (keep in mind that HP-UX reserves swap for all virtual address space [text "reserves" from the backing file, though]... if you fail the reservation, the allocation fails). If you get up to 100% then you're unlikely to be able to start new processes (and memory allocations in existing ones will fail). If your machine is truly in a steady state, you're ok.. if not, consider adding some more swap to give yourself additional virtual address space.

Re: Swap Utilization 90% + all the time

Darren Etheridge_2 — Thu, 05 May 2005 12:54:42 GMT

Add swap space in the kernel or allocate physical disk swap space?

Re: Swap Utilization 90% + all the time

Alex Lavrov. — Thu, 05 May 2005 13:20:57 GMT

Well, in order not to repeat others, here you go, some usefull discussions about swap size:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=26968

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=456768

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=763197

http://docs.hp.com/en/TKP-90202/ch08s01.html#d0e2720

Re: Swap Utilization 90% + all the time

Don Morris_1 — Thu, 05 May 2005 14:40:32 GMT

Add physical swap space (disk or filesystem). You can't get the kernel to revise the pseudo-swap calculations without adding or removing physical memory which means you either have to reboot or build a time machine (since OLA isn't in HP-UX yet).

Re: Swap Utilization 90% + all the time

Darren Etheridge_2 — Thu, 05 May 2005 14:43:07 GMT

Well my Oracle system just crashed and gave everyone "Out of Memory/Space errors", so I had to do some changes and see what happens. UGGG

Re: Swap Utilization 90% + all the time

Alex Lavrov. — Thu, 05 May 2005 15:36:13 GMT

That's weird, because according to your post, you had almost zero paging, which indicates that you have enough memory to keep all the pages in ....

Re: Swap Utilization 90% + all the time

Alex Lavrov. — Thu, 05 May 2005 15:42:07 GMT

What Oracle version do you run on your system? Can it be 32bit Oracle on 64bit OS?

If so, you can get these errors if yo don't use memory windows ....

Re: Swap Utilization 90% + all the time

Darren Etheridge_2 — Thu, 05 May 2005 16:02:32 GMT

64 bit

Re: Swap Utilization 90% + all the time

Andrew Cowan — Fri, 06 May 2005 02:27:27 GMT

I've seen this behaviour with Oracle before. I think its related to way in which is tries to create a pre-warmed SGA? Oracle seems to pin large amounts of paging/swap space in case it needs it for a query. I'm afraid that I have never found any solution other than to allocate a large amount of swap, even though it never seems to get used.

The best way in Oracle to avoid massive memory drain is to avoid queries that involve full table scans, by judicious use of indexes.

Re: Swap Utilization 90% + all the time

Don Morris_1 — Fri, 06 May 2005 07:54:23 GMT

It isn't paging that's the problem. The physical usage is low enough (and most of it is pinned/locked in memory anyway) that vhand doesn't need to actually page out anything. The virtual usage is consuming all of the available virtual address space == (pseudoswap + swap).

If you do a brk(), create a shared mmap/sysV segment, whatever of X Mb... the system will reserve X Mb of swap so it knows that if the time ever comes for vhand to do something with it, it will have the room to do so. If that reservation fails, the allocation request is also failed (with ENOMEM) which is what happened to Oracle, apparently. You can screw up a system without ever using any actual RAM by just starting programs that make huge malloc's... hence the max*siz tunables, process rlimits to prevent malicious users (or naive undergrads).

Adding more swap space increases the virtual address space for such allocations.

Re: Swap Utilization 90% + all the time

Andrew Cowan — Fri, 06 May 2005 09:14:42 GMT

Don, your answer is very interesting, however what puzzles me (considering your explanation) is why the system is show "Page-ins" and "Page_Faults" and yet no "Page-outs".

I'm definitely not a kernel expert, however I was always taught that it is normal for a system to "Page-out", however if it "pages-in" constantly then you have a memory problem? Also the number of "Page Faults" suggests that VHAND has "Paged-out" because the requested data no-longer resides in RAM. Any ideas?

Re: Swap Utilization 90% + all the time

Ted Buis — Fri, 06 May 2005 13:53:01 GMT

There is a great discussion of Pseudo-swap in HP-UX 11i Internals by Chris Moore and Chis Cooper (or is it Cooper and More). There are positive and negatives to psueudo-swap, but the bottom like from my understand is that it is great as long as you never need to use physical device swap, because the RAM that is assigned for this pseudo-swap cannot be paged out ever to device swap. Thus the small number of pages that are left that can be swapped to disk suffer an unfair burden. Assignment of these pages is arbitrary, so this isn't at all good if paging is done. The good side of Pseudo-swap is you don't waste disk space that never gets used if you have sufficient RAM. However if you start getting over 90% utilization of your total virtual address space, then you can either buy more RAM, add addition disk device swap space, or both. Once you run out of virtual address space, no new processes can be launched, and processes that need more either hang or die. If the system needs more space and can't get it then it hangs or crashes.