Out of Memory: Linux

[Pradep]

dear all,

[root@jupiter root]# uname -a
Linux jupiter 2.4.21-27.ELsmp #1 SMP Wed Dec 1 21:59:02 EST 2004 i686 athlon i386 GNU/Linux

Out of memory error and server hangs.
Please help how can i troubleshoot with what commands.

Mar 16 06:25:54 jupiter kernel: ENOMEM in journal_get_undo_access_Rsmp_6317abf4, retrying.
Mar 16 06:25:54 jupiter kernel: Mem-info:
Mar 16 06:25:54 jupiter kernel: Zone:DMA freepages: 2858 min: 0 low: 0 high: 0
Mar 16 06:25:54 jupiter kernel: Zone:Normal freepages: 638 min: 1279 low: 4544 high: 6304
Mar 16 06:25:54 jupiter kernel: Zone:HighMem freepages: 260 min: 255 low: 11776 high: 17664
Mar 16 06:25:54 jupiter kernel: Free pages: 3756 ( 260 HighMem)
Mar 16 06:25:54 jupiter kernel: ( Active: 103988/388960, inactive_laundry: 58345, inactive_clean: 58338, free: 3756 )
Mar 16 06:25:54 jupiter kernel: aa:0 ac:0 id:0 il:0 ic:0 fr:2858
Mar 16 06:25:54 jupiter kernel: aa:1393 ac:34 id:27 il:130 ic:0 fr:638
Mar 16 06:25:54 jupiter kernel: aa:61259 ac:41302 id:388847 il:58327 ic:58338 fr:260
Mar 16 06:25:54 jupiter kernel: 2*4kB 0*8kB 0*16kB 3*32kB 1*64kB 0*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 2*4096kB = 11432kB)
Mar 16 06:25:54 jupiter kernel: 16*4kB 11*8kB 10*16kB 4*32kB 3*64kB 3*128kB 2*256kB 0*512kB 1*1024kB 0*2048kB 0*4096kB = 2552kB)
Mar 16 06:25:54 jupiter kernel: 12*4kB 2*8kB 3*16kB 3*32kB 13*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1040kB)
Mar 16 06:25:54 jupiter kernel: Swap cache: add 5627, delete 4224, find 571/1401, race 0+0
Mar 16 06:25:54 jupiter kernel: 14709 pages of slabcache
Mar 16 06:25:54 jupiter kernel: 566 pages of kernel stacks
Mar 16 06:25:54 jupiter kernel: 0 lowmem pagetables, 1027 highmem pagetables
Mar 16 06:25:54 jupiter kernel: Free swap: 12559624kB
Mar 16 06:25:54 jupiter kernel: 983040 pages of RAM
Mar 16 06:25:54 jupiter kernel: 753648 pages of HIGHMEM
Mar 16 06:25:54 jupiter kernel: 18715 reserved pages
Mar 16 06:25:54 jupiter kernel: 443126 pages shared
Mar 16 06:25:54 jupiter kernel: 1403 pages swap cached
Mar 16 06:25:54 jupiter kernel: Out of Memory: Killed process 10420 (java).
Mar 16 06:25:54 jupiter kernel: Out of Memory: Killed process 10420 (java).
Mar 16 06:25:54 jupiter kernel: Out of Memory: Killed process 10421 (java).
Mar 16 06:25:54 jupiter kernel: Out of Memory: Killed process 10422 (java).
Mar 16 06:25:54 jupiter kernel: Out of Memory: Killed process 10423 (java).
Mar 16 06:25:54 jupiter kernel: Out of Memory: Killed process 10424 (java).

[Steven E. Protter]

Shalom,

Reduce memory consumption prior to running your commands.

This is a legitimate out of memory condition.

Check free output before you start.

Maybe you can shut down some daemons and server processes to save memory.

SEP

[Pradep]

Hi Steven,

can u please help explain about above mem condition description ?

Free swap: 12559624kB

Swap is still free. why it was not used ?

[dirk dierickx]

maybe the swapiness setting was set to 0 (zero), thus the swap will never be used.

other then that, i can't think of a reason why swap would not be used.

[Pradep]

Hi,

i tried
echo 60 > /proc/sys/vm/swappiness
-bash: /proc/sys/vm/swappiness: No such file or directory

what can i do ?
can touch the swappiness file first or it has to be something registered with kernel/vm somewhere/somehow ?

regards

[Vitaly Karasik_1]

IIRC, swapiness parameter was introduced only in 2.6 kernels.

As for as I see, you run i386 kernel, so probably you encountered 3GB per process limit?
Can you send us "free" command output and some stats about your memory-intensive processes - for example, from "top" sorted using "M".

[Pradep]

free
total used free shared buffers cached
Mem: 3857300 1192536 2664764 0 221852 572528
-/+ buffers/cache: 398156 3459144
Swap: 12578812 3192 12575620





TOP SORTED by M

249 processes: 245 sleeping, 2 running, 2 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 0.0% 0.0% 1.2% 0.0% 0.0% 0.0% 98.5%
cpu00 0.0% 0.0% 0.3% 0.1% 0.0% 0.0% 99.4%
cpu01 0.0% 0.0% 2.1% 0.0% 0.1% 0.0% 97.6%
Mem: 3857300k av, 1194116k used, 2663184k free, 0k shrd, 221712k buff
757112k actv, 153224k in_d, 2436k in_c
Swap: 12578812k av, 3192k used, 12575620k free 572532k cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME CPU COMMAND
18047 root 25 0 153M 152M 39100 S 0.0 4.0 0:47 0 java
18050 root 15 0 153M 152M 39100 S 0.0 4.0 1:00 0 java
18051 root 16 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18052 root 16 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18053 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18054 root 15 0 153M 152M 39100 S 0.0 4.0 0:07 1 java
18055 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18056 root 15 0 153M 152M 39100 S 0.0 4.0 1:41 1 java
18059 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18060 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18061 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18062 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18070 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18074 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18077 root 15 0 153M 152M 39100 S 0.0 4.0 0:01 0 java
18080 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18081 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18082 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18083 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18084 root 24 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18110 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18111 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18113 root 16 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18117 root 25 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18120 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18121 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18122 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18123 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 1 java
18133 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18134 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18135 root 15 0 153M 152M 39100 S 0.0 4.0 0:00 0 java
18136 root 15 0 153M 152M 39100 S 0.0 4.0 0:01 0 java
18137 root 15 0 153M 152M 39100 S 0.0 4.0 0:04 0 java

[Vitaly Karasik_1]

OK, so it is not one huge process, but many small Java instances. Accroding to this snapshot, situation with RAM usage seems OK now.

I guess, for some reason number of java (tomcat, jboss?) instances was increased and they took all RAM.

It is still interesting why system crashed when you had free swap; but from practical point of view I suggest to investigate why number of java instances were increased.
I suggest to activate some monitoring solution (script-based or real NMS) for alerting when java is starting eat too much ram.

[Matti_Kurkela]

Note that this is a 32-bit OS, running a kernel 2.4.21.

The amount of swap and the presence of HIGHMEM seems to indicate this machine has more than 1 GB of RAM.

By the output, the number of free pages is quite small compared to the number of active pages. This seems to be most severe in the HighMem zone.

My *guess* would be that there might be so much memory fragmentation that when the system requires a large contiguous block of memory for some reason, it cannot find one.

In any case, you're running kernel version 2.4.21-27.ELsmp, which has plenty of known bugs. By the kernel version number, I'm guessing you are running RedHat Enterprise Linux 3. I would recommend to update to the latest errata kernel (2.4.21-57.ELsmp or newer) and see if the problem can be reproduced.

MK

[Pradep]

thanks,
can u give me procedure to upgrade kernel ?