Unexplained increasing swap utilisation

[Glenn L. Stewart]

Three of our HA SAP/Oracle servers recently had a large increase in utilisation in swap usage post upgrade from 10.20 to 11.00

prdapp2 /etc/named.data > swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 1261 787 62% 0 - 1 /dev/vgroot/lvol2
dev 512 463 49 90% 0 - 1 /dev/vgroot/swap2
reserve - 703 -703
memory 1535 284 1251 19%
total 4095 2711 1384 66% - 0 -
Swap utilisation 62% & 90% as above. Swap on prdapp2 and prdapp3 would normally be close to zero. This is quite unusual.
Also related - MC/Service Guard cluster server: prdapp4
Recently prdapp4 started to use /var/tmp to swap (as shown by swapinfo -atm)
root@prdapp4:/ # swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 200 4 196 2% 0 - 1 /dev/vg00/lvol2
dev 2500 1438 1062 58% 0 - 0 /dev/vg00/lvol12
dev 2500 545 1955 22% 0 - 0 /dev/vg00/lvol11
localfs 758 2 756 0% none 0 1 /var/tmp
reserve - 561 -561
memory 1515 387 1128 26%
total 7473 2937 4536 39% - 0 -
(although none of this is used).
Would like help determining the root cause and suggestions as to reducing the utilisation of swap.

Thanks in advance

[Magdi KAMAL]

Hi Glenn,

I got this situation after installing hp-ux 11.0. First let me know about your total physical memory and swap space ?

There is a rule to respect in very heavy transactionnal server ( OLTP ) :

Swap space = 4 * physical memory.

Magdi

[Glenn L. Stewart]

It might be because of the upgrade - highly likley. Thanks for at least cinfirming you has similar problems.
How did you solve these problems?

One the APP servers we have 2GB Physical, with approx 3GB swap (divided half between two lvols)

Thanks

Glenn

[Magdi KAMAL]

Hi again Glenn,

If you have 2GB memory, I would say that your swap ( 3GB ) is really too small.

You may configer it to be between 6 and 8 GBs ( from three to four times the amount of physical memory).

it's a good idea to have two device to reduce competing disk I/Os for swapping.

Have a look on that book :

O'Reilly
Essential System Administration
2nd edition Revised and Updated

Aeleen Frisch

Page 315 "Managing System Resources".


Magdi

[Larry Basford]

dev 512 0 512 0% 0 - 1 /dev/vg00/lvol2
reserve - 512 -512
memory 2368 1358 1010 57%
total 2880 1870 1010 65

Our Oracle server.

As long as swap never goes to the disk it sould be OK. If it goes to disk you need more memory.

We saw an increase in network traffic just changing to 11.0
Have you also noticed this ?

[Glenn L. Stewart]

On top of my original question, wondering why swap in/outs are high given that a low percentage of physical is used.

# swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 2048 1261 787 62% 0 - 1 /dev/vgroot/lvol2
dev 512 439 73 86% 0 - 1 /dev/vgroot/swap2
reserve - 748 -748
memory 1535 207 1328 13%
total 4095 2655 1440 65% - 0 -

Physical used 13%
Swap used 62% & 86%

# vmstat -s | head
170368 swap ins
170368 swap outs
524 pages swapped in
353 pages swapped out
630974013 total address trans. faults taken
246995526 page ins
1154871 page outs

Also curious as to the accurate meaning of SZ and VSZ in the UNIX95 version of ps
Example:
# export UNIX95=1
# ps -ef -o pid -o comm -o sz -o vsz
PID COMMAND SZ VSZ
12879 dw.sapPRD_D00 6962 139596
14391 dw.sapPRD_D00 10001 43372
5829 dw.sapPRD_D00 5846 29628
17450 dw.sapPRD_D00 5117 30140
18604 dw.sapPRD_D00 11460 50188
27260 dw.sapPRD_D00 12013 60300
29659 dw.sapPRD_D00 5336 26220
16245 dw.sapPRD_D00 5256 28572
17044 dw.sapPRD_D00 7923 37612
15161 dw.sapPRD_D00 7835 37164
10711 dw.sapPRD_D00 10731 47036
14228 dw.sapPRD_D00 10605 46908
5813 dw.sapPRD_D00 1947 21388
7089 dw.sapPRD_D00 4704 27580
3477 dw.sapPRD_D00 4998 27692
925 dw.sapPRD_D00 6163 35788
24202 dw.sapPRD_D00 10921 49148
1025 dw.sapPRD_D00 6511 30348
5337 dw.sapPRD_D00 8702 38492
2662 dw.sapPRD_D00 6891 32956
5814 se.sapPRD_D00 38 296
2728 dw.sapPRD_D00 7147 32268

(these look like the offending processes)

Typically I will assign points for fair to great answers (unlike some people these days)

Thanks in advance for any information you all can provide