Memory issue,server rebooted-Oracle RAC on DL580/RHEL5u1

[bGacesa]

Hi,

The problem relates to a two DL580-R04(4xXeon EM64T, 64GB mem) servers running RHEL 5/Oracle 10g R3 within RACluster.

The memory (64GB) and swap(10GB) gets filled up on second server usually while RMAN is performing backup, and after backup is complete memory and swap are left full!

Is this related to:
Jun 10 08:27:23 db2 kernel: warning: many lost ticks.
Jun 10 08:27:23 db2 kernel: Your time source seems to be instable or some driver is hogging interupts
Jun 10 08:27:23 db2 kernel: rip __do_softirq+0x53/0xd5
Jun 10 08:41:38 db2 kernel: tnslsnr[11623]: segfault at 0000000000000018 rip 0000003385a6d92e rsp 00007fff493e2600 error 4
Jun 10 08:58:19 db2 logger: Oracle clsomon failed with fatal status 13.
Jun 10 08:58:21 db2 logger: Oracle CRS failure. Rebooting for cluster integrity.

because we are getting our servers rebooted every day!?

What tools/procedure any of you know to investigate further this?

Here is
ps -e -o 'vsz pid ruser cpu time args' |sort –nr ; free ; cat /proc/meminfo ; vmstat

second server(db2 kernel: Linux version 2.6.18-53.1.14.el5 ) output:

total used free shared buffers cached
Mem: 66008828 65688228 320600 0 7525120 49425444
-/+ buffers/cache: 8737664 57271164
Swap: 10289144 9831608 457536

MemTotal: 66008828 kB
MemFree: 320968 kB
Buffers: 7525152 kB
Cached: 49425540 kB
SwapCached: 4363448 kB
Active: 27767896 kB
Inactive: 34905880 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 66008828 kB
LowFree: 320968 kB
SwapTotal: 10289144 kB
SwapFree: 457536 kB
Dirty: 728 kB
Writeback: 132 kB
AnonPages: 1359268 kB
Mapped: 21109612 kB
Slab: 655864 kB
PageTables: 2152800 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 43293556 kB
Committed_AS: 83404984 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 9068 kB
VmallocChunk: 34359729111 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB


procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 9831512 320968 7525176 49425732 1 4 511 21 13 23 6 1 94 0 0


46260656 13505 oracle - 00:01:20 ora_lgwr_bccs2
46255512 13687 oracle - 00:00:05 ora_arc1_bccs2
46255508 13685 oracle - 00:00:03 ora_arc0_bccs2
46254808 13503 oracle - 00:00:23 ora_dbw1_bccs2
46254600 4992 oracle - 00:00:15 oraclebccs2 (LOCAL=NO)
46253200 13501 oracle - 00:00:21 ora_dbw0_bccs2
46250948 13660 oracle - 00:38:44 ora_p000_bccs2
46250912 13662 oracle - 00:39:07 ora_p001_bccs2
46250468 12390 oracle - 00:01:12 oraclebccs2 (LOCAL=NO)
46250468 10473 oracle - 00:00:49 oraclebccs2 (LOCAL=NO)
46250408 13481 oracle - 00:03:23 ora_lmd0_bccs2
46250404 13495 oracle - 00:01:48 ora_lms3_bccs2
46250404 13487 oracle - 00:01:46 ora_lms1_bccs2
46250400 13491 oracle - 00:01:50 ora_lms2_bccs2
46250400 13483 oracle - 00:03:37 ora_lms0_bccs2
46248852 13670 oracle - 00:37:25 ora_p003_bccs2
46248656 13475 oracle - 00:00:05 ora_diag_bccs2
46248588 19442 oracle - 00:00:58 oraclebccs2 (LOCAL=NO)
46248464 10204 oracle - 00:00:10 oraclebccs2 (LOCAL=NO)
46248436 17888 oracle - 00:00:45 oraclebccs2 (LOCAL=NO)
46248428 13609 oracle - 00:01:12 oraclebccs2 (LOCAL=NO)
46246672 4814 oracle - 00:01:18 oraclebccs2 (LOCAL=NO)
46246604 13507 oracle - 00:00:12 ora_ckpt_bccs2
46245404 26120 oracle - 00:14:34 ora_p014_bccs2
46245388 26155 oracle - 00:15:13 ora_p028_bccs2
46244800 13664 oracle - 00:37:51 ora_p002_bccs2
46244768 26130 oracle - 00:15:17 ora_p018_bccs2
46244748 26137 oracle - 00:15:16 ora_p021_bccs2
46244640 13479 oracle - 00:00:56 ora_lmon_bccs2
46244524 14662 oracle - 00:53:43 ora_j000_bccs2
46244364 21983 oracle - 00:00:24 oraclebccs2 (LOCAL=NO)
46244352 4805 oracle - 00:00:23 oraclebccs2 (LOCAL=NO)
46244332 28252 oracle - 00:00:18 oraclebccs2 (LOCAL=NO)
46244332 21496 oracle - 00:00:13 oraclebccs2 (LOCAL=NO)
46244332 17885 oracle - 00:00:46 oraclebccs2 (LOCAL=NO)
46244332 15697 oracle - 00:00:21 oraclebccs2 (LOCAL=NO)
46244324 7731 oracle - 00:00:05 oraclebccs2 (LOCAL=NO)
46244324 18814 oracle - 00:02:19 oraclebccs2 (LOCAL=NO)
46242704 26124 oracle - 00:15:44 ora_p015_bccs2
46242688 26161 oracle - 00:15:12 ora_p031_bccs2
46242660 13711 oracle - 00:00:00 ora_qmnc_bccs2
46242492 13519 oracle - 00:00:14 ora_mmon_bccs2
46242416 17892 oracle - 00:00:11 oraclebccs2 (LOCAL=NO)
46242324 13517 oracle - 00:00:07 ora_cjq0_bccs2
46242312 19468 oracle - 00:00:23 oraclebccs2 (LOCAL=NO)
46242308 10202 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242304 13647 oracle - 00:00:14 oraclebccs2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
46242284 25647 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242284 13509 oracle - 00:00:19 ora_smon_bccs2
46242276 13788 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242272 27693 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242272 22652 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242272 22489 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242268 4133 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242264 30802 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242264 23678 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242264 21753 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242264 18469 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242264 17829 oracle - 00:00:20 oraclebccs2 (LOCAL=NO)
46242264 17462 oracle - 00:00:02 oraclebccs2 (LOCAL=NO)
46242260 30191 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242260 26942 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242260 17775 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242260 11304 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242244 26096 oracle - 00:16:30 ora_p004_bccs2
46242236 26100 oracle - 00:16:05 ora_p006_bccs2
46242224 30470 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242212 13955 oracle - 00:03:18 oraclebccs2 (LOCAL=NO)
46242212 12837 oracle - 00:00:31 oraclebccs2 (LOCAL=NO)
46242208 5459 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242204 9938 oracle - 00:00:22 oraclebccs2 (LOCAL=NO)
46242204 6401 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242204 496 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242204 24428 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242204 23766 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242204 22150 oracle - 00:00:01 oraclebccs2 (LOCAL=NO)
46242204 17956 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242204 15827 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242204 11318 oracle - 00:00:15 oraclebccs2 (LOCAL=NO)
46242200 5951 oracle - 00:00:18 oraclebccs2 (LOCAL=NO)
46242196 29119 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242196 22775 oracle - 00:00:31 oraclebccs2 (LOCAL=NO)
46242196 21355 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242196 13239 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46242112 26102 oracle - 00:15:50 ora_p007_bccs2
46241752 15054 oracle - 00:00:33 oraclebccs2 (LOCAL=NO)
46241688 26133 oracle - 00:15:40 ora_p019_bccs2
46241676 26139 oracle - 00:15:29 ora_p022_bccs2
46241676 26135 oracle - 00:15:36 ora_p020_bccs2
46241492 13574 oracle - 00:00:00 ora_rbal_bccs2
46241164 26126 oracle - 00:15:19 ora_p016_bccs2
46241160 26153 oracle - 00:15:17 ora_p027_bccs2
46241160 26151 oracle - 00:14:57 ora_p026_bccs2
46241160 26149 oracle - 00:15:08 ora_p025_bccs2
46240656 26128 oracle - 00:15:12 ora_p017_bccs2
46240472 13551 oracle - 00:00:50 ora_lck0_bccs2
46240316 26098 oracle - 00:16:05 ora_p005_bccs2
46240304 13642 oracle - 00:00:01 oraclebccs2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
46240260 13654 oracle - 00:00:01 oraclebccs2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
46240044 13515 oracle - 00:00:00 ora_reco_bccs2
46239248 26108 oracle - 00:15:28 ora_p010_bccs2
46239228 26145 oracle - 00:14:44 ora_p023_bccs2
46239024 13477 oracle - 00:00:00 ora_psp0_bccs2
46238644 26104 oracle - 00:15:43 ora_p008_bccs2
46238600 26112 oracle - 00:15:26 ora_p012_bccs2
46238600 26110 oracle - 00:15:29 ora_p011_bccs2
46238588 26159 oracle - 00:15:20 ora_p030_bccs2
46238588 26147 oracle - 00:15:24 ora_p024_bccs2
46238584 26157 oracle - 00:15:13 ora_p029_bccs2
46238188 24152 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238160 18158 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238156 23464 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238148 26465 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238124 26106 oracle - 00:15:30 ora_p009_bccs2
46238100 25464 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238096 26973 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238092 8947 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238092 27216 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238092 27113 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238092 26375 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238092 14399 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238088 31073 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238088 26118 oracle - 00:15:39 ora_p013_bccs2
46238084 8017 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238084 32433 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238084 21092 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238080 23918 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46238072 4803 oracle - 00:00:35 oraclebccs2 (LOCAL=NO)
46237892 13570 oracle - 00:00:05 ora_asmb_bccs2
46237068 11676 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46237060 4253 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46237056 18379 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46236952 24738 oracle - 00:00:00 ora_pz99_bccs2
46236936 13920 oracle - 00:00:00 ora_q000_bccs2
46236864 13473 oracle - 00:00:20 ora_pmon_bccs2
46236592 13525 oracle - 00:00:00 ora_s000_bccs2
46236204 13523 oracle - 00:00:00 ora_d000_bccs2
46236028 18454 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46235932 24748 oracle - 00:00:00 ora_pz98_bccs2
46235556 3093 oracle - 00:00:00 ora_o001_bccs2
46235552 17883 oracle - 00:00:00 oraclebccs2 (LOCAL=NO)
46235420 13521 oracle - 00:00:10 ora_mmnl_bccs2
46235420 13499 oracle - 00:00:22 ora_mman_bccs2
46235380 13922 oracle - 00:00:00 ora_q001_bccs2
669244 10556 root - 00:00:58 /opt/oracle/crs/bin/crsd.bin reboot
614636 17741 oracle - 00:02:01 /opt/oracle/oracle/10.2.0/db1/jdk/bin/java -server -Xmx256M -XX:MaxPermSize=96m -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -DORACLE_HOME=/opt/oracle/oracle/10.2.0/db1 -Doracle.home=/opt/oracle/oracle/10.2.0/db1/oc4j -Doracle.oc4j.localhome=/opt/oracle/oracle/10.2.0/db1/db2_bccs2/sysman -DEMSTATE=/opt/oracle/oracle/10.2.0/db1/db2_bccs2 -Doracle.j2ee.dont.use.memory.archive=true -Djava.protocol.handler.pkgs=HTTPClient -Doracle.security.jazn.config=/opt/oracle/oracle/10.2.0/db1/oc4j/j2ee/OC4J_DBConsole_db2_bccs2/config/jazn.xml -Djava.security.policy=/opt/oracle/oracle/10.2.0/db1/oc4j/j2ee/OC4J_DBConsole_db2_bccs2/config/java2.policy -Djava.security.properties=/opt/oracle/oracle/10.2.0/db1/oc4j/j2ee/home/config/jazn.security.props -DEMDROOT=/opt/oracle/oracle/10.2.0/db1/db2_bccs2 -Dsysman.md5password=true -Drepapi.oracle.home=/opt/oracle/oracle/10.2.0/db1 -Ddisable.checkForUpdate=true -Djava.awt.headless=true -jar /opt/oracle/oracle/10.2.0/db1/oc4j/j2ee/home/oc4j.jar -config /opt/oracle/oracle/10.2.0/db1/oc4j/j2ee/OC4J_DBConsole_db2_bccs2/config/server.xml
360384 10445 oracle - 00:00:04 /opt/oracle/crs/bin/evmd.bin
311064 13559 oracle - 00:00:10 /opt/oracle/oracle/10.2.0/db1/bin/racgimon startd bccs
289048 11178 oracle - 00:02:33 asm_lms0_+ASM2
289044 11176 oracle - 00:00:05 asm_lmd0_+ASM2
281272 11194 oracle - 00:00:00 asm_gmon_+ASM2
280904 11184 oracle - 00:00:00 asm_dbw0_+ASM2
278152 11174 oracle - 00:00:02 asm_lmon_+ASM2
277892 30284 oracle - 00:00:01 oracle+ASM2 (LOCAL=NO)
277796 11192 oracle - 00:00:00 asm_rbal_+ASM2
277796 11186 oracle - 00:00:00 asm_lgwr_+ASM2
277128 11168 oracle - 00:00:01 asm_diag_+ASM2
276728 10922 oracle - 00:06:40 /opt/oracle/crs/bin/ocssd.bin
276616 11202 oracle - 00:00:01 asm_lck0_+ASM2
275372 11164 oracle - 00:00:01 asm_pmon_+ASM2
274332 13572 oracle - 00:00:01 oracle+ASM2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
274328 13455 oracle - 00:00:00 asm_o000_+ASM2
274292 1914 oracle - 00:00:00 oracle+ASM2 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
274200 11190 oracle - 00:00:00 asm_smon_+ASM2
274200 11188 oracle - 00:00:00 asm_ckpt_+ASM2
274196 11182 oracle - 00:00:00 asm_mman_+ASM2
274196 11171 oracle - 00:00:00 asm_psp0_+ASM2
174196 11266 oracle - 00:00:00 /opt/oracle/asm/bin/racgimon daemon ora.db2.ASM2.asm
138784 11219 oracle - 00:00:39 /opt/oracle/oracle/10.2.0/db1/bin/tnslsnr LISTENER_DB2 -inherit
122364 11290 oracle - 00:00:00 /opt/oracle/crs/opmn/bin/ons -d
109096 2359 miroslav - 00:00:00 sort -nr
100972 31715 root - 00:00:00 su - oracle
100972 11950 root - 00:00:00 su - oracle
100972 11640 programi - 00:00:00 su root
95468 7888 root - 00:00:01 automount
85112 31648 root - 00:00:00 sshd: root@pts/3
85112 12383 root - 00:00:00 sshd: root@pts/1
85112 11567 programi - 00:00:00 sshd: programi@pts/0
84984 11534 root - 00:00:00 sshd: programi [priv]
84980 2116 miroslav - 00:00:00 sshd: miroslav@pts/2
84980 1994 root - 00:00:00 sshd: miroslav [priv]
79552 19385 oracle - 00:01:00 /opt/oracle/oracle/10.2.0/db1/bin/emagent
74744 9115 root - 00:00:00 crond
68964 9443 nobody - 00:00:00 /opt/oracle/oracle/osb/.etc.linux86_64/obhttpd -DSSL -d/opt/oracle/oracle/osb/apache
68964 9442 nobody - 00:00:00 /opt/oracle/oracle/osb/.etc.linux86_64/obhttpd -DSSL -d/opt/oracle/oracle/osb/apache
68964 9441 nobody - 00:00:00 /opt/oracle/oracle/osb/.etc.linux86_64/obhttpd -DSSL -d/opt/oracle/oracle/osb/apache
68964 9440 nobody - 00:00:00 /opt/oracle/oracle/osb/.etc.linux86_64/obhttpd -DSSL -d/opt/oracle/oracle/osb/apache
68964 9439 nobody - 00:00:00 /opt/oracle/oracle/osb/.etc.linux86_64/obhttpd -DSSL -d/opt/oracle/oracle/osb/apache
68964 9424 root - 00:00:00 /opt/oracle/oracle/osb/.etc.linux86_64/obhttpd -DSSL -d/opt/oracle/oracle/osb/apache
66832 9052 root - 00:00:00 sendmail: accepting connections
66008 11951 oracle - 00:00:00 -bash
66008 11568 programi - 00:00:00 -bash
66004 31716 oracle - 00:00:00 -bash
66004 2117 miroslav - 00:00:00 -bash
66000 31665 root - 00:00:00 -bash
66000 12470 root - 00:00:00 -bash
66000 11723 root - 00:00:00 bash
65544 2358 miroslav - 00:00:00 ps -e -o vsz pid ruser cpu time args
65224 8897 root - 00:00:00 rpc.rquotad
63776 10773 oracle - 00:00:00 /bin/sh -c cd /opt/oracle/crs/log/db2/cssd/oclsomon; ulimit -c unlimited; /opt/oracle/crs/bin/oclsomon || exit $?
58840 9406 root - 00:00:00 rhnsd --interval 240
58176 11030 oracle - 00:00:00 /opt/oracle/crs/bin/evmlogger.bin -o /opt/oracle/crs/evm/log/evmlogger.info -l /opt/oracle/crs/evm/log/evmlogger.log
57584 8813 root - 00:00:01 /usr/sbin/sshd
57580 9060 smmsp - 00:00:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
52764 9618 programi - 00:00:00 /opt/oracle/oracle/10.2.0/db1/bin/tnslsnr EXTPROC_LISTENER -inherit
50792 10801 oracle - 00:00:05 /opt/oracle/crs/bin/oclsomon.bin
48616 7690 root - 00:00:00 rpc.idmapd
46272 9167 root - 00:00:06 /opt/oracle/oracle/osb/.etc.linux86_64/observiced -b
45988 9237 root - 00:00:01 /opt/oracle/oracle/osb/.etc.linux86_64/obscheduled
45852 9739 root - 00:00:00 /bin/su -l oracle -c sh -c 'ulimit -c unlimited; cd /opt/oracle/crs/log/db2/evmd; exec /opt/oracle/crs/bin/evmd '
35768 10772 root - 00:00:00 /sbin/runuser -l oracle -c /bin/sh -c 'cd /opt/oracle/crs/log/db2/cssd/oclsomon; ulimit -c unlimited; /opt/oracle/crs/bin/oclsomon || exit $?'
32636 9422 68 - 00:00:06 hald
32204 8773 root - 00:00:11 /usr/local/bin/qlremote
26268 9615 root - 00:00:00 /usr/ecc/exec/mstragent -s
23732 9432 root - 00:00:00 /usr/libexec/hald-addon-cpufreq
23316 8834 ntp - 00:00:00 ntpd -u ntp:ntp -p /var/run/ntpd.pid -g
21632 9423 root - 00:00:00 hald-runner
21232 7719 dbus - 00:00:00 dbus-daemon --system
20888 9151 xfs - 00:00:00 xfs -droppriv -daemon
18684 9198 root - 00:00:00 /usr/sbin/atd
12616 1570 root - 00:00:00 /sbin/udevd -d
12272 9448 68 - 00:00:00 hald-addon-keyboard: listening on /dev/input/event0
12272 9437 68 - 00:00:00 hald-addon-keyboard: listening on /dev/input/event2
12272 9431 68 - 00:00:00 hald-addon-acpi: listening on acpi kernel interface /proc/acpi/event
11004 9741 root - 00:02:54 /bin/sh /etc/init.d/init.cssd fatal
10876 10593 root - 00:00:00 /bin/sh /etc/init.d/init.cssd daemon
10876 10567 root - 00:00:00 /bin/sh /etc/init.d/init.cssd oclsomon
10748 9744 root - 00:00:00 /bin/sh /etc/init.d/init.crsd run
10696 7565 root - 00:00:02 irqbalance
10316 1 root - 00:00:06 init [3]
10180 9459 root - 00:00:06 hald-addon-storage: polling /dev/hda
10176 9457 root - 00:00:04 hald-addon-storage: polling /dev/hdb
10112 17726 oracle - 00:00:01 /opt/oracle/oracle/10.2.0/db1/perl/bin/perl /opt/oracle/oracle/10.2.0/db1/bin/emwd.pl dbconsole /opt/oracle/oracle/10.2.0/db1/db2_bccs2/sysman/log/emdb.nohup
8480 7848 root - 00:00:00 /usr/bin/hidd --server
8080 9001 root - 00:00:00 rpc.mountd
8024 7641 rpcuser - 00:00:00 rpc.statd
8016 7602 rpc - 00:00:00 portmap
7196 9107 zabbix - 00:00:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
7140 9106 zabbix - 00:02:32 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
7140 9105 zabbix - 00:02:32 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
7140 9102 zabbix - 00:02:32 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
7124 9101 zabbix - 00:05:19 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
7124 9094 zabbix - 00:00:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
6420 9077 root - 00:00:00 gpm -m /dev/input/mice -t exps2
5876 7545 root - 00:00:00 syslogd -m 0
5632 9559 root - 00:00:00 /usr/ecc/exec/mstragent
4044 9727 root - 00:00:00 /usr/sbin/smartd -q never
3772 7548 root - 00:00:00 klogd -x
3760 9734 root - 00:00:00 /sbin/mingetty tty4
3760 9732 root - 00:00:00 /sbin/mingetty tty2
3760 14198 root - 00:00:00 /sbin/mingetty tty1
3756 9737 root - 00:00:00 /sbin/mingetty tty6
3756 9736 root - 00:00:00 /sbin/mingetty tty5
3756 9733 root - 00:00:00 /sbin/mingetty tty3
3756 2357 root - 00:00:00 /bin/sleep 1
2828 11289 oracle - 00:00:00 /opt/oracle/crs/opmn/bin/ons -d
VSZ PID RUSER CPU TIME COMMAND
0 9 root - 00:00:00 [ksoftirqd/2]
0 8 root - 00:00:00 [migration/2]
0 8998 root - 00:00:00 [nfsd]
0 8997 root - 00:00:00 [nfsd]
0 8996 root - 00:00:00 [nfsd]
0 8995 root - 00:00:00 [nfsd]
0 8994 root - 00:00:00 [nfsd]
0 8993 root - 00:00:00 [nfsd]
0 8992 root - 00:00:00 [nfsd]
0 8991 root - 00:00:00 [nfsd]
0 8990 root - 00:00:00 [rpciod/15]
0 8989 root - 00:00:00 [rpciod/14]
0 8988 root - 00:00:00 [rpciod/13]
0 8987 root - 00:00:00 [rpciod/12]
0 8986 root - 00:00:00 [rpciod/11]
0 8985 root - 00:00:00 [rpciod/10]
0 8984 root - 00:00:00 [rpciod/9]
0 8983 root - 00:00:00 [rpciod/8]
0 8982 root - 00:00:00 [rpciod/7]
0 8981 root - 00:00:00 [rpciod/6]
0 8980 root - 00:00:00 [rpciod/5]
0 8979 root - 00:00:00 [rpciod/4]
0 8978 root - 00:00:00 [rpciod/3]
0 8977 root - 00:00:00 [rpciod/2]
0 8976 root - 00:00:00 [rpciod/1]
0 8975 root - 00:00:00 [rpciod/0]
0 8974 root - 00:00:00 [lockd]
0 8973 root - 00:00:00 [nfsd4]
0 845 root - 00:00:00 [kseriod]
0 843 root - 00:00:00 [khubd]
0 840 root - 00:00:00 [cqueue/15]
0 839 root - 00:00:00 [cqueue/14]
0 838 root - 00:00:00 [cqueue/13]
0 837 root - 00:00:00 [cqueue/12]
0 836 root - 00:00:00 [cqueue/11]
0 835 root - 00:00:00 [cqueue/10]
0 834 root - 00:00:00 [cqueue/9]
0 833 root - 00:00:00 [cqueue/8]
0 832 root - 00:00:00 [cqueue/7]
0 831 root - 00:00:00 [cqueue/6]
0 830 root - 00:00:00 [cqueue/5]
0 829 root - 00:00:00 [cqueue/4]
0 828 root - 00:00:00 [cqueue/3]
0 827 root - 00:00:00 [cqueue/2]
0 826 root - 00:00:00 [cqueue/1]
0 825 root - 00:00:00 [cqueue/0]
0 7 root - 00:00:00 [watchdog/1]
0 7768 root - 00:00:00 [kjournald]
0 6 root - 00:00:00 [ksoftirqd/1]
0 6949 root - 00:00:00 [kondemand/15]
0 6948 root - 00:00:00 [kondemand/14]
0 6947 root - 00:00:00 [kondemand/13]
0 6946 root - 00:00:00 [kondemand/12]
0 6945 root - 00:00:00 [kondemand/11]
0 6944 root - 00:00:00 [kondemand/10]
0 6943 root - 00:00:00 [kondemand/9]
0 6942 root - 00:00:00 [kondemand/8]
0 6941 root - 00:00:00 [kondemand/7]
0 6940 root - 00:00:00 [kondemand/6]
0 6939 root - 00:00:00 [kondemand/5]
0 6938 root - 00:00:00 [kondemand/4]
0 6937 root - 00:00:00 [kondemand/3]
0 6936 root - 00:00:00 [kondemand/2]
0 6935 root - 00:00:00 [kondemand/1]
0 6934 root - 00:00:00 [kondemand/0]
0 66 root - 00:00:00 [khelper]
0 65 root - 00:00:00 [events/15]
0 6584 root - 00:00:04 [kjournald]
0 6581 root - 00:00:00 [kjournald]
0 6570 root - 00:00:00 [kjournald]
0 6557 root - 00:00:00 [kjournald]
0 6546 root - 00:00:00 [kjournald]
0 6537 root - 00:00:04 [kjournald]
0 6529 root - 00:00:00 [kjournald]
0 64 root - 00:00:00 [events/14]
0 63 root - 00:00:00 [events/13]
0 62 root - 00:00:00 [events/12]
0 620 root - 00:00:00 [kacpid]
0 61 root - 00:00:00 [events/11]
0 619 root - 00:00:00 [kblockd/15]
0 618 root - 00:00:00 [kblockd/14]
0 617 root - 00:00:00 [kblockd/13]
0 616 root - 00:00:00 [kblockd/12]
0 615 root - 00:00:00 [kblockd/11]
0 614 root - 00:00:00 [kblockd/10]
0 613 root - 00:00:00 [kblockd/9]
0 612 root - 00:00:00 [kblockd/8]
0 611 root - 00:00:00 [kblockd/7]
0 610 root - 00:00:00 [kblockd/6]
0 60 root - 00:00:00 [events/10]
0 609 root - 00:00:00 [kblockd/5]
0 608 root - 00:00:00 [kblockd/4]
0 607 root - 00:00:00 [kblockd/3]
0 606 root - 00:00:00 [kblockd/2]
0 605 root - 00:00:00 [kblockd/1]
0 604 root - 00:00:00 [kblockd/0]
0 5 root - 00:00:00 [migration/1]
0 59 root - 00:00:00 [events/9]
0 58 root - 00:00:00 [events/8]
0 585 root - 00:00:00 [kthread]
0 57 root - 00:00:00 [events/7]
0 56 root - 00:00:00 [events/6]
0 5684 root - 00:00:00 [MpxTestDaemon ]
0 5644 root - 00:00:00 [MpxTestDaemon]
0 5643 root - 00:00:00 [MpxDispatchDaem]
0 5642 root - 00:00:00 [MpxProactiveDae]
0 5641 root - 00:00:00 [MpxGrDaemon]
0 5640 root - 00:00:00 [MpxPeriodicCall]
0 5639 root - 00:00:00 [MpxResumeIoDaem]
0 5637 root - 00:00:00 [MpxAsyncIoDaemo]
0 55 root - 00:00:00 [events/5]
0 5554 root - 00:00:00 [emcprequestd]
0 5553 root - 00:00:00 [emcpdefd]
0 5552 root - 00:00:00 [emcpd]
0 54 root - 00:00:00 [events/4]
0 5457 root - 00:00:00 [kmpathd/15]
0 5456 root - 00:00:00 [kmpathd/14]
0 5455 root - 00:00:00 [kmpathd/13]
0 5454 root - 00:00:00 [kmpathd/12]
0 5453 root - 00:00:00 [kmpathd/11]
0 5452 root - 00:00:00 [kmpathd/10]
0 5451 root - 00:00:00 [kmpathd/9]
0 5450 root - 00:00:00 [kmpathd/8]
0 5449 root - 00:00:00 [kmpathd/7]
0 5448 root - 00:00:00 [kmpathd/6]
0 5447 root - 00:00:00 [kmpathd/5]
0 5446 root - 00:00:00 [kmpathd/4]
0 5445 root - 00:00:00 [kmpathd/3]
0 5444 root - 00:00:00 [kmpathd/2]
0 5443 root - 00:00:00 [kmpathd/1]
0 5442 root - 00:00:00 [kmpathd/0]
0 53 root - 00:00:00 [events/3]
0 52 root - 00:00:00 [events/2]
0 51 root - 00:00:00 [events/1]
0 50 root - 00:00:00 [events/0]
0 4 root - 00:00:00 [watchdog/0]
0 49 root - 00:00:00 [watchdog/15]
0 48 root - 00:00:00 [ksoftirqd/15]
0 47 root - 00:00:00 [migration/15]
0 46 root - 00:00:00 [watchdog/14]
0 45 root - 00:00:00 [ksoftirqd/14]
0 44 root - 00:00:00 [migration/14]
0 43 root - 00:00:00 [watchdog/13]
0 42 root - 00:00:00 [ksoftirqd/13]
0 41 root - 00:00:00 [migration/13]
0 40 root - 00:00:00 [watchdog/12]
0 3 root - 00:00:00 [ksoftirqd/0]
0 39 root - 00:00:00 [ksoftirqd/12]
0 38 root - 00:00:00 [migration/12]
0 37 root - 00:00:00 [watchdog/11]
0 36 root - 00:00:00 [ksoftirqd/11]
0 35 root - 00:00:00 [migration/11]
0 34 root - 00:00:00 [watchdog/10]
0 33 root - 00:00:00 [ksoftirqd/10]
0 3385 root - 00:00:00 [ata_aux]
0 3384 root - 00:00:00 [ata/15]
0 3383 root - 00:00:00 [ata/14]
0 3382 root - 00:00:00 [ata/13]
0 3381 root - 00:00:00 [ata/12]
0 3380 root - 00:00:00 [ata/11]
0 3379 root - 00:00:00 [ata/10]
0 3378 root - 00:00:00 [ata/9]
0 3377 root - 00:00:00 [ata/8]
0 3376 root - 00:00:00 [ata/7]
0 3375 root - 00:00:00 [ata/6]
0 3373 root - 00:00:00 [ata/5]
0 3372 root - 00:00:00 [ata/4]
0 3371 root - 00:00:00 [ata/3]
0 3370 root - 00:00:00 [ata/2]
0 3369 root - 00:00:00 [ata/1]
0 3368 root - 00:00:00 [ata/0]
0 32 root - 00:00:00 [migration/10]
0 31 root - 00:00:00 [watchdog/9]
0 30 root - 00:00:00 [ksoftirqd/9]
0 2 root - 00:00:00 [migration/0]
0 29 root - 00:00:00 [migration/9]
0 28 root - 00:00:00 [watchdog/8]
0 27 root - 00:00:00 [ksoftirqd/8]
0 26 root - 00:00:00 [migration/8]
0 25 root - 00:00:00 [watchdog/7]
0 24 root - 00:00:00 [ksoftirqd/7]
0 23 root - 00:00:00 [migration/7]
0 22 root - 00:00:00 [watchdog/6]
0 21 root - 00:00:00 [ksoftirqd/6]
0 20 root - 00:00:00 [migration/6]
0 19 root - 00:00:00 [watchdog/5]
0 18 root - 00:00:00 [ksoftirqd/5]
0 17 root - 00:00:00 [migration/5]
0 16 root - 00:00:00 [watchdog/4]
0 15 root - 00:00:00 [ksoftirqd/4]
0 1536 root - 00:00:00 [kauditd]
0 1509 root - 00:00:02 [kjournald]
0 1506 root - 00:00:00 [ksnapd]
0 14 root - 00:00:00 [migration/4]
0 1437 root - 00:00:00 [fc_dl_2]
0 1436 root - 00:00:00 [fc_wq_2]
0 1435 root - 00:00:00 [scsi_wq_2]
0 1434 root - 00:00:00 [qla2xxx_2_dpc]
0 1433 root - 00:00:00 [scsi_eh_2]
0 1432 root - 00:00:00 [fc_dl_1]
0 1431 root - 00:00:00 [fc_wq_1]
0 1430 root - 00:00:00 [scsi_wq_1]
0 1429 root - 00:00:00 [qla2xxx_1_dpc]
0 1428 root - 00:00:00 [scsi_eh_1]
0 13 root - 00:00:00 [watchdog/3]
0 1376 root - 00:00:00 [scsi_eh_0]
0 12 root - 00:00:00 [ksoftirqd/3]
0 11 root - 00:00:00 [migration/3]
0 1182 root - 00:00:00 [kpsmoused]
0 10 root - 00:00:00 [watchdog/2]
0 1035 root - 00:00:00 [aio/15]
0 1034 root - 00:00:00 [aio/14]
0 1033 root - 00:00:00 [aio/13]
0 1032 root - 00:00:00 [aio/12]
0 1031 root - 00:00:00 [aio/11]
0 1030 root - 00:00:00 [aio/10]
0 1029 root - 00:00:00 [aio/9]
0 1028 root - 00:00:00 [aio/8]
0 1027 root - 00:00:00 [aio/7]
0 1026 root - 00:00:00 [aio/6]
0 1025 root - 00:00:00 [aio/5]
0 1024 root - 00:00:00 [aio/4]
0 1023 root - 00:00:00 [aio/3]
0 1022 root - 00:00:00 [aio/2]
0 1021 root - 00:00:00 [aio/1]
0 1020 root - 00:00:00 [aio/0]
0 1019 root - 00:01:03 [kswapd0]
0 1018 root - 00:00:05 [pdflush]
0 1017 root - 00:00:00 [pdflush]

[Ivan Ferreira]

>>> The memory (64GB) and swap(10GB) gets filled up on second server usually while RMAN is performing backup, and after backup is complete memory and swap are left full!

The output of the free command report that most is used as buffer/cache. This is normal, check other threads for memory usage in Linux.

>>> because we are getting our servers rebooted every day!?

Probably is caused by the RMAN process as it is very resource intensive, but is not because of memory problems.


>> Jun 10 08:58:19 db2 logger: Oracle clsomon failed with fatal status 13.
Jun 10 08:58:21 db2 logger: Oracle CRS >> failure. Rebooting for cluster integrity.

These kind of problems normally are related to timeout accesing the ocr or voting disks. Maybe your I/O is so heavy during the backup that you get timeouts.

Try to identify the real source of the CRS failure, then ivestigate the operating system.