topic Re: memory in Operating System - Linux

memory

dawn_jose85 — Mon, 31 Jan 2011 07:24:25 GMT

server got hang . when connected through the console , in the screen it is showing error as below continuously . The error is executing as like in a loop .

cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0
cpu 10 hot:High 166,batch1 used 0

lowmem_reserve[]:0 0 12877 12877
Mode 0 Normal free:476588 min127879
inactive:62354598 present :1368978
lowmem_reserve[]:0 0 0 0
Node 0 HighMem free:0kB min128kB
Node 0 DMA:3*4kB 3*8kB 4*16kB 3*32kB
Node 0 HighMem:empty
400547 pagecache pages
swap cache: add 8213770,delete 821809
Free swap =0kB
Total swap=240026632kB

The os
Linux ipl333 2.6.18-92.1.18.el5 #1 SMP Wed Nov 12 09:19:49 EST 2008 x86_64

After reboot the server came UP .

Pleae help me to trouble shoot this issue . What is the root cause for this issue

Re: memory

Alzhy — Mon, 31 Jan 2011 15:28:34 GMT

Difficult to ascertain.

Do yo have SAR collections running and capturing historicals?

If you do check mem and swap utilisation historicals:

sar -r -f /var/log/sa/saNN

If not, it is a very good practice to have SAR (install sysstat package) and set up sar collection:

- yum install sysstat
- edit /etc/cron.d/sysstat (to suite your collections, I have it gathering stats every 5 minutes)

Cheers!

Re: memory

dawn_jose85 — Tue, 01 Feb 2011 05:22:56 GMT

Hi,

is this occured due to memory overflow.
When this happend , the system was showing swap=0kB.
In the output screen console the error was in running mode , means as it is in a loop.

Re: memory

Alzhy — Tue, 01 Feb 2011 14:33:05 GMT

Very likely good sir.
That is why it is suggested you set up performance collection as outlined so you can definitely pinpoint and correlate if this was indeed a Virtual Memory issue.

What application is running?
How much memory does your system have?
How much SWAP space do you have?
If a DB Instance is running:
-- how much total SGA is defined
-- how many Oracle connections was designed

Cheers!

Re: memory

dawn_jose85 — Wed, 02 Feb 2011 07:06:09 GMT

will this be helpful to find out the root cause of the previous error .

Re: memory

dawn_jose85 — Wed, 02 Feb 2011 12:40:53 GMT

Hi ,
There are the outputs u r asking ..
What application is running? ---

Mediation services are running in this server.
How much memory does your system have?

ipl323root#free -m
total used free shared buffers cached
Mem: 15923 15823 99 0 602 2741
-/+ buffers/cache: 12480 3443
Swap: 23440 3989 19450

ipl323root#cat /proc/meminfo
MemTotal: 16305936 kB
MemFree: 109416 kB
Buffers: 616468 kB
Cached: 2807560 kB
SwapCached: 128284 kB
Active: 10526700 kB
Inactive: 4996364 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 16305936 kB
LowFree: 109416 kB
SwapTotal: 24002632 kB
SwapFree: 19917276 kB
Dirty: 3004 kB
Writeback: 0 kB
AnonPages: 11831056 kB
Mapped: 1509908 kB
Slab: 206156 kB
PageTables: 236984 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 32105424 kB
Committed_AS: 21981976 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 275596 kB
VmallocChunk: 34359461883 kB
HugePages_Total: 49
HugePages_Free: 49
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
ipl323root#

If a DB Instance is running:
-- how much total SGA is defined

#kernel parameters for Oracle
kernel.shmall=7416046
kernel.shmmax=27001002393
kernel.shmmni=4096
kernel.sem=510 32000 100 256
fs.file-max=65536
net.ipv4.ip_local_port_range=32768 65000
net.core.rmem_default=262144
net.core.rmem_max=262144
net.core.wmem_default=262144
net.core.wmem_max=262144
kernel.core_pattern=/var/core/core_%u_%g_%e
kernel.core_uses_pid=0

-- how many Oracle connections was designed

6 are designed and they are running parallelly.

Re: memory

Alzhy — Wed, 02 Feb 2011 14:36:56 GMT

Absolutely.
So next step will be to look at which application is possibly introudcing the memory bloat if that is the case good sir.

Re: memory

dawn_jose85 — Thu, 03 Feb 2011 06:05:15 GMT

How can i find out , in this case . applications are same for the server ,those are running continuosly...

How to check which application is consuming more memory?

Re: memory

Alzhy — Fri, 04 Feb 2011 14:27:29 GMT

You can explore "ps" and size up each process' memory footprint -- i.e. RSS or SZ as a first start sir.

Heed my advice -- get SAR installed and collecting stats daily sir. It will be very valuable.