High swap space usuage

[Bruno Bossier_1]

I have a 2-node Serviceguard cluster (11.16) with extensions for RAC (latest version) and running an Oracle 9i RAC (cluster interconnect via Hyperfabric using UDP).

Swap space usuage is constantly very high, over 90%. I have been looking for specific processes who might cause this, but could not find any, even after stopping Oracle RAC, Serviceguard, ... swap space usuage stays very high. After we reboot both machines, all is OK, but after while usuage increases again to over 90%.

Any ideas ?

Thanks,
Bruno

[Bharat Katkar]

Bruno,

Could u go thr' this cookbook attached!!
Hope that helps.

Regards,

[Fred Ruffet]

Are you using raw devices or FS to store Datafiles ?

Regards,

Fred

[Bruno Bossier_1]

We are using raw devices. All shared storage is configured using Veritas Cluster Volume Manager.

As for the cookbook, I will look into it, but it's gonna be heavy reading :-)

Bruno

[Fred Ruffet]

can't you see processes using a huge amount of memory ? using
UNIX95=true ps -A -o vsz,pid,args
and
ipcs -mb

have you possibility to start with just one node to see if it acts the same ?

Regards,

Fred

[Steve Steel]

Hi

The unix95 shown is a good start

Here is a script of 39 lines

1 #RSZ = Resident set size
2 #VSZ = Virtual set size
3 #
4 #RSZ is a subset of VSZ
5 #-the portion of the processes memory which is 'resident'
6 #in physical memory (ie not paged out).
7 #
8 #VSZ is the total size, including resident and non-resident pages.
9 #
10 #These values are totals, obtained by adding up the memory utilisation of the
11 #processes various memory segments (of which there can be lots).
12 #################################################################################
13 uname -a
14 echo -------------------------------------
15 /bin/rm /tmp/$PPID 2>/dev/null
16 if [ "$LOGNAME" = "root" ]
17 then
18 echo memory from kernel
19 name=$(file /stand/vmunix|grep 64)
20 if [ "$name" = "" ]
21 then
22 echo phys_mem_pages/D | adb /stand/vmunix /dev/kmem 2>/dev/null
23 else
24 echo phys_mem_pages/D | adb64 /stand/vmunix /dev/kmem 2>&1|grep -i page
25 fi
26 fi
27
28 echo " "
29 echo swap
30 echo -----------------
31 swapinfo -tm
32 echo " "
33 echo memory usage per process
34 echo "----------------------------------------------"
35 UNIX95= ps -e -o ruser,vsz,sz,pid,args > /tmp/$PPID
36 head -n 1 /tmp/$PPID
37 tail -n +2 /tmp/$PPID|
38 sort -rnk2
39 /bin/rm /tmp/$PPID 2>/dev/null


Also look at

Memory Usage - â What is using all of the memory?â

by:eric.herberholz@hp.com

Last modified: October 18, 2004

Full document is available at external ftp site: ftp://eh:spear9@hprc.external.hp.com/memory.htm
The "Table of Contents" is available in ITRC doc id MEMORYKBAN00000975


You can reach the ftp site shown and there is good info and tools available


Steve

[Bruno Bossier_1]

OK, thanks to all. We have also a call open with HP support, but will in parallel investigate using this script.

Bruno

[Bill Hassell]

Swap usage means low memory or very high memory mapped file usage. Now the ps listing (UNIX95= ps -e -o ...) should total up the current size of process usage, but also check on shared memory usage:

ipcs -bmop

You may find huge shared memory segments, which is OK and really improves performance unless you have critically small RAM. If swap thrashing is occcuring (processes competing for memory) then vmstat's metric po (page out) will indicate this. Single digits OK, double digits so-so, triple digits for long periods = very bad performance and lack of RAM.