General
cancel
Showing results for 
Search instead for 
Did you mean: 

swap utilisation is ver high but not memory utilisation

SOLVED
Go to solution
amithp
Frequent Advisor

swap utilisation is ver high but not memory utilisation

Hi We have a system where swap utilisation is very high but not the memory utilisation.

Can sombody give explanation what exactly is happening.

following is the glance output of the system
B3692A GlancePlus C.03.86.00 02:31:08 tstqat14 9000/800 Current Avg High
------------------------------------------------------------------------------------------------------------------------------
CPU Util S SU U | 32% 32% 35%
Disk Util F | 1% 1% 1%
Mem Util S SU UB B | 82% 82% 82%
Swap Util U UR R | 96% 96% 96%
------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 19
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name (1600% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
uprt_lsrrt 15457 1 168 qatwrk73 0.0/ 0.1 52.2 0.0/ 0.0 210.7mb 1
java 4709 4708 168 apiadm77 0.2/ 0.1 908.1 0.0/ 0.0 206.0mb 76
uprt_lsrrt 18133 1 168 qatwrk73 0.0/ 0.2 76.5 0.0/ 0.1 196.2mb 1
pcs_pr_fp_f 13323 13186 154 qatwrk73 0.0/ 0.0 8.7 0.0/ 0.0 193.5mb 1
pcs_pr_futt 26951 26935 154 qatwrk73 0.0/ 0.0 18.4 0.0/ 0.0 177.5mb 1
oracleQAT14 26977 1 154 qatwrk73 0.0/ 0.0 21.7 0.0/ 0.0 174.3mb 1
oracleQAT14 26978 1 154 qatwrk73 0.0/ 0.0 23.9 0.0/ 0.0 172.2mb 1
pcs_pr_futt 26952 26935 154 qatwrk73 0.0/ 0.0 22.9 0.0/ 0.0 169.5mb 1
pcs_pr_futt 26950 26935 154 qatwrk73 0.0/ 0.0 20.1 0.0/ 0.0 167.7mb 1
BlRrtChk 23489 1 168 qatwrk75 0.0/ 0.0 15.0 0.0/ 0.0 150.0mb 1
BlRrtChk 23176 1 168 qatwrk77 0.0/ 0.1 8.6 0.0/ 0.0 148.4mb 1
uprt_rqrck 15925 1 168 qatwrk73 0.0/ 0.0 2.5 0.0/ 0.0 119.4mb 1
csmUFldr 10179 1 155 qatoln79 0.0/ 0.0 5.8 0.0/ 0.0 118.7mb 1
uprt_rqrck 15399 1 168 qatwrk73 0.0/ 0.0 2.8 0.0/ 0.0 118.0mb 1
oracleQAT14 26976 1 154 qatwrk73 0.0/ 0.0 24.2 0.0/ 0.0 114.9mb 1
BlPrep 23292 1 168 qatwrk77 0.0/ 0.1 8.4 0.0/ 0.1 113.9mb 1
oracleQAT14 13383 1 154 qatwrk73 0.0/ 0.0 8.0 0.0/ 0.0 113.8mb 1
BlPrep 23568 1 168 qatwrk75 0.0/ 0.0 10.4 0.0/ 0.0 111.4mb 1
BlCalc 23210 1 168 qatwrk77 0.0/ 0.1 8.3 0.0/ 0.0 111.3mb 1
BlCalc 23453 1 168 qatwrk75 0.0/ 0.0 9.7 0.0/ 0.0 110.3mb 1


glance with "w" is

Swap Device Type Avail Used Priority
--------------------------------------------------------------------------------
/dev/vg00/lvswap device 32.0gb 0mb 1
/dev/vg00/lvswap01 device 15.6gb 0mb 1
/dev/vgqat/lvswap1 device 15.6gb 0mb 1
/dev/vgoravl/lvswap2 device 29.3gb 0mb 1
/dev/vgoravl/lvswap1 device 29.3gb 0mb 1
pseudo-swap memory 49.8gb 43.1gb -1

Swap Available: 175800 Swap Used: 43789m Swap Util (%): 96 Reserved: 168549

THE vmstat OUTPUT IS

[root@tstqat14:/root]# vmstat 1 11
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
7 1 0 24337943 2386739 144 15 2 0 0 0 0 4610 27743 2556 7 2 91
7 1 0 24337943 2386739 1188 152 0 0 0 0 0 3646 100130 12382 38 16 46
7 1 0 24337943 2385578 1337 157 0 0 0 0 0 3600 108431 12773 36 17 47
7 1 0 24337943 2385064 1360 167 0 0 0 0 0 3560 106497 12998 37 13 50
7 1 0 24337943 2384677 1351 172 0 0 0 0 0 3547 102886 12785 36 9 55
7 1 0 24337943 2383355 1424 173 0 0 0 0 0 3532 105187 12710 36 10 54
7 1 0 24337943 2383033 1316 169 0 0 0 0 0 3529 110449 12775 38 12 50
7 1 0 24337943 2382776 1580 184 0 0 0 0 0 3479 110376 13038 37 8 55
7 1 0 24337943 2381938 1494 181 0 0 0 0 0 3467 106625 12994 36 7 57
7 1 0 24337943 2382135 1247 149 0 0 0 0 0 3462 102812 12783 41 13 47
10 2 0 20551946 2375753 1876 276 0 0 0 0 0 3450 113606 13197 40 11 49

So there is no page in's or out also. but still swap util is high!!!!

Please help me in telling what is happening.
6 REPLIES
Don Morris_1
Honored Contributor
Solution

Re: swap utilisation is ver high but not memory utilisation

Swap utilization is high because it is all reserved, not because there's any data on the devices. (You have 175800Mb of swap, and 168549Mb of it is reserved). [Note that none of the devices show anything but 0Mb in the Used column of glance/"w", and this matches the observation of no pageouts].

Processes must reserve swap when non-file backed virtual objects are created (barring use of lazy swap options via chatr, mmap flags, etc.). This is to ensure the system has swap blocks available if and when a pageout must occur. So your workload has a virtual requirement which is almost at your available swap space taking into account both device swap and memory swap.

If things remain this way -- you do have about 7Gb of virtual address space remaining, so this may simply be a well-tuned load for the box. If it pushes to 100%, however, you would start to see failures from malloc, fork, mmap, etc. with ENOMEM so you'd want to either add more swap before that point or throttle down the workload.
amithp
Frequent Advisor

Re: swap utilisation is ver high but not memory utilisation

Thanks Don.

But i didnt understand few things in your reply

>>non-file backed virtual objects are created (barring use of lazy swap options via chatr, mmap flags, etc.)

Please elaborate.

Also are there lots of processes that have reserverd virtual memory?
Do we need to add swap space so that more ana more processes could be started.

Also if more and more processes are started is pages in be very high?

Dennis Handly
Acclaimed Contributor

Re: swap utilisation is ver high but not memory utilisation

Please provide the simpler "swapinfo -tam" output.

>I didn't understand few things in your reply
>>non-file backed virtual objects are created (barring use of lazy swap options via chatr, mmap flags, etc.)

The first part says that if you request virtual memory for shared memory, heap or private mmap, you need to reserve swap space. Shared mapped files don't need it.
The second part says if you are tricky and use lazy swap, then it doesn't reserve it.

>are there lots of processes that have reserved virtual memory?

There may be few that reserve lots of memory, it's the sum of the requirements.

>Do we need to add swap space so that more and more processes could be started.

Yes, either that or more memory.

>Also if more and more processes are started is pages in be very high?

You can't start that much more, only 7 Gb as Don said.
amithp
Frequent Advisor

Re: swap utilisation is ver high but not memory utilisation

Thanks Dennis.
And What is "lazy swap" ?
Dennis Handly
Acclaimed Contributor

Re: swap utilisation is ver high but not memory utilisation

>And what is "lazy swap"?

It says "I know what I'm doing and I don't care if there isn't any swap reserved when I touch that page and it's ok to abort".

It basically doesn't reserve swap space when you allocate virtual memory, only when you touch a page. If you don't have swap, your application will abort.
amithp
Frequent Advisor

Re: swap utilisation is ver high but not memory utilisation

Thank you Don and dennis I got my answer