swapping

[Animesh Chakraborty]

Hi,
One of our N class server having 2GB memory ,2 CPUs,20 GB swap running oracle databese for sap application.
vmstat -s out put shows :
#vmstat -s
9570476 swap ins
9570493 swap outs
57802 pages swapped in
1007722 pages swapped out
1277702576 total address trans. faults taken
365441200 page ins
2069089 page outs
26259892 pages paged in
12705088 pages paged out
336176349 reclaims from free list
353105123 total page reclaims
28387 intransit blocking page faults
966096162 zero fill pages created
670100863 zero fill page faults
127564167 executable fill pages created
5845660 executable fill page faults
0 swap text pages found in free list
64270503 inode text pages found in free list
2846 revolutions of the clock hand
66557963 pages scanned for page out
74681319 pages freed by the clock daemon
3099650326 cpu context switches
8318653950 device interrupts
2115269423 traps
62176108870 system calls
423296504 Page Select Size Successes for Page size 4K
24222619 Page Select Size Successes for Page size 16K
423296504 Page Select Size Failures for Page size 16K
1350976449 Page Allocate Successes for Page size 4K
37330318 Page Allocate Successes for Page size 16K
7241 Page Allocate Successes for Page size 64K
301 Page Allocate Successes for Page size 256K
20 Page Allocate Successes for Page size 1M
2 Page Allocate Successes for Page size 64M
20620128 Page Allocate Failures for Page size 4K
21404189 Page Allocate Failures for Page size 16K
35214 Page Allocate Failures for Page size 64K
49696 Page Allocate Failures for Page size 256K
50298 Page Allocate Failures for Page size 1M
50338 Page Allocate Failures for Page size 4M
50338 Page Allocate Failures for Page size 16M
50338 Page Allocate Failures for Page size 64M
1502867 Page Demotions for Page size 16K
7138 Page Demotions for Page size 64K
273 Page Demotions for Page size 256K
15 Page Demotions for Page size 1M

I am suspecting memory leak.
What else should I check for this high paging.

Thanks in advance
Animesh

[Stefan Farrelly]

Cor, 2 GB of RAM and 20 GB of swap !! Is this right ?

How about some output from swapinfo -mt and;
vmstat 1 10

This will show swap/paging usage a bit better.....

[Animesh Chakraborty]

Hi,
#swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 1097 2999 27% 0 - 1 /dev/vg00/lvol2
dev 16000 1103 14897 7% 0 - 1 /dev/vg00/swap01
reserve - 1664 -1664
memory 1534 856 678 56%
total 21630 4720 16910 22% - 0 -

#vmstat 1 10
procs memory page u
r b w avm free re at pi po fr de sr ind
3 0 0 325865 2717 27 6 2 1 7 0 6 5982
1 1 0 327517 2626 27 3 2 0 1 0 0 19478
1 1 0 327517 2671 25 3 2 0 3 0 0 18447
1 1 0 327517 2671 28 4 2 0 4 0 0 17746
1 1 0 327517 2671 22 3 2 0 3 0 0 17270
1 1 0 327517 2671 18 2 1 0 2 0 0 16888
0 0 0 79580 2671 15 1 0 0 1 0 0 16526
0 0 0 79580 2671 22 3 1 0 3 0 0 16114
0 0 0 79580 2671 18 2 0 0 2 0 0 15885
0 0 0 79580 2671 15 1 0 0 1 0 0 15877

[Stefan Farrelly]

So you do have 20GB of swap, why with only 2GB of RAM ? You only need swap=ram size.

Total device swap usage = 2999+14897 (or 18MB) which basically means at some point (not necessarily right now) you had 18MB of processes swapped to disk. This means at some point memory pressure was so tight that you were 18MB of RAM short of what you needed.

vmstat free column (2671 - * 4096) = 10MB free right now, which is ok. The pi/po rate is very small which means very little paging going on right now which is good.

So, youre running at 99% memory usage now and at some point went just over 100%. You need to find out what is causing the memory pressure. Whats your unix buffer cache set to ?
(sysdef | grep dbc) This can be reduced which should freeup lots of memory. How much shared memory is Oracle using ?

If you run the following command (its all on one line) it will list processes by their memory size, then we can see if anything big hogging memory which can be stopped;

UNIX95= ps -e -o vsz=Kbytes -o ruser -o pid,args=Command-Line | sort -rnk1 | more

[Animesh Chakraborty]

Hi Stefen,
Thanks for the reply.
I think PHSS_23670 is required for mib2agt.


dbc_max_pct 50
dbc_min_pct 5
***********
967080 root 1968 /usr/sbin/mib2agt
79600 sg1adm 24956 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
56864 sg1adm 25748 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
46392 orasg1 927 oracleSG1 (DESCRIPTION=(LOCAL=NO)(SDU=32768))
43584 sg1adm 11167 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
43216 sg1adm 19477 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
43200 sg1adm 24954 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
42976 sg1adm 22964 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
42528 sg1adm 25156 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
41040 sg1adm 922 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM
GS0
40328 orasg1 27825 oracleSG1 (DESCRIPTION=(LOCAL=NO)(SDU=32768))
40040 orasg1 25753 oracleSG1 (DESCRIPTION=(LOCAL=NO)(SDU=32768))
38600 orasg1 19484 oracleSG1 (DESCRIPTION=(LOCAL=NO)(SDU=32768))
37136 sg1adm 24955 dw.sapSG1_DVEBMGS00 pf=/usr/sap/SG1/SYS/profile/SG1_DVEBM

Thanks
Animesh

[Stefan Farrelly · Accepted solution]

Cor, mib2agt is using tons of MB ! for now you can shut it down - /sbin/init.d/SnmpMib2 stop

We disable this process from running completely on all our servers, it does not cause a problem. Edit /etc/rc.config.d/SnmpMib2 and change the following to 0;

SNMP_MIB2_START=0 # Start the MIB2 SNMP subAgent.
export SNMP_MIB2_START

That will stop it completely.

Also, your buffer cache is set far too high. Reduce dbc_mac_pct from 50 down to around 15 and reboot. You will then have tons of memory free and shouldnt have to worry about swapping/paging anymore.

[Santosh Nair_1]

You might also consider lowering dbc_max_pct from 50 to 10. This is the maximum amount of physical memory that can be set aside for disk buffer cache. 50 percent is the default, but this tends to be fairly hight and so most ppl lower this down to 10 or even 5 percent.

-Santosh

[Trond Haugen]

Hi,

When setting the dbc_max_pct do also considder that dynamic buffer cache isn't always the best strategy. For manny datbases a fixed buffer cache is better and I normally see that for Oracle installations.
The book "HP-UX tuning and performance" says that the "growth and shrinking of the cache may in fact cause performance degradation".

[Roger Baptiste]

Animesh,

Adding further to the
excellent suggestions folks
in this thread have given
so far:

Check the SGA size of the
database. Make sure it is
not set too high. The DBA
should be able to help you
in figuring the SGA size.

Also, don't forget to
reduce the dbc_max size
to 10%. It will need a
kernel regeneration and reboot.

Install Glance on your
system, if you don't have
it. Helps a lot in tracking
performance related issues.

-raj