topic Re: kernel: Out of Memory in Operating System - Linux

kernel: Out of Memory

Danesh Qureshi — Thu, 22 Feb 2007 10:09:19 GMT

I am running RHEL AS 3 (Taroon) on a HP DL380 G3 server. The server has 2GB of RAM.

In the /var/log/messages I've noticed a lot of Out of Memory errors. Also the server sometimes hangs and we need to reboot to get round this.

Please can anyone help me to resolve the Out Memory and system hang problems?

Thank you.

Feb 22 13:36:17 leda kernel: Out of Memory: Killed process 24766 (java).
Feb 22 13:36:17 leda kernel: Out of Memory: Killed process 25746 (java).
Feb 22 13:36:17 leda kernel: Out of Memory: Killed process 28559 (java).
Feb 22 13:36:17 leda kernel: Out of Memory: Killed process 30900 (java).

Re: kernel: Out of Memory

Ivan Ferreira — Thu, 22 Feb 2007 10:40:15 GMT

This problem you should troubleshoot at application level.

See:
http://www-128.ibm.com/developerworks/java/library/j-leaks/
http://www.onjava.com/pub/a/onjava/2001/08/22/optimization.html

Re: kernel: Out of Memory

George Liu_4 — Thu, 22 Feb 2007 15:04:02 GMT

You may use "free" to check the memory usage when you see the problem.

It seems java applicaiton is a culprit on your case.

Re: kernel: Out of Memory

dirk dierickx — Fri, 23 Feb 2007 01:44:43 GMT

Take a look at the following java options:

-Xms set initial Java heap size
-Xmx set maximum Java heap size

this way you can limit the memory usage of java programs.

Re: kernel: Out of Memory

Sheriff Andy — Fri, 23 Feb 2007 12:32:38 GMT

Java is such a memory consumer. Any way to limit the amount of users connected?

What is your swap set to as well?

Re: kernel: Out of Memory

Danesh Qureshi — Fri, 23 Feb 2007 13:09:50 GMT

Unfortunatley I cannot limit the number of users connected.

I have attached results of free and vmstat.

Please have a look at the output of these two commands.

Re: kernel: Out of Memory

Rick Beldin — Mon, 26 Feb 2007 09:52:25 GMT

You are seeing the effects of the Linux kernel Out Of Memory (OOM killer). When the kernel reaches low memory conditions, it takes decisive action and attempts to free memory for continued operation by looking for processes that are big consumers of memory. The algorithm takes into acount a number of factors, including age of the process, the time it last run, whether or not it is root and so on. This is merely a *symptom* of a system that is either badly overloaded or suffering from serious defects in memory.

You should also see information from the kernel at the time of the 'killed' message that indicate how much memory was available
at the time.

Recommendations would be to:

- update to the RHEL3 U8 kernel (2.4.21-47)
as there have been subtle adjustments to
the oom-killer and the vm

- consideration of using ulimit to limit
users virtual memory consumption

- limiting high io activity such as backups
to slack periods of time

- adding more memory to the system.

- use of top and ps to get more of a picture
of the application use

- use of sar data to get a picture on overall
memory consumption

- monitoring /proc/meminfo will give you some
more granularity on where the allocations
are occurring.

Re: kernel: Out of Memory

Alpha977 — Mon, 26 Feb 2007 17:17:20 GMT

Hello Danesh!

Can you post a "free" result?

Then, i think is an application problem.
I have same problem with a java bug, when for a bad code (mysql connection) the java process generate thousand threads to fill memory.

Check also the number of process with "top".

Regards.

Re: kernel: Out of Memory

Danesh Qureshi — Tue, 27 Feb 2007 06:25:00 GMT

Hello,

I have attached a tar zip file which contains two files. The first file is the messages file where you can see "Out of Memory" errors which occurred around 13:30.

The second file is part of a log file which was generated by a shell script that I ran as a cron job every 10 minutes to keep track of the date, free -m, vmstat and ps commands.

I am currently at the log file to see if I can find anything unusual which my have caused the system to report "out of Memory" errors.

Hope you can spot something in the log file.

Re: kernel: Out of Memory

Rick Beldin — Tue, 27 Feb 2007 14:06:10 GMT

>> I have attached a tar zip file which contains two files.

I don't know what you attached, but what I got was a single file with what looks like the output of the cron job...

A quick look:

- you have ~1280 processes on this system with
2gb of memory
- Oracle and Java appear to be used heavily
- I would suspect process creation/deletion is
somewhat rapid, since I think that pids have
wrapped around

I would:

- validate settings for Oracle (with Oracle)
- some Linux kernel settings can affect Oracle behavior (hugetlb_pool) and reduce vm
use
- see if shutting down Oracle and friends during times of oom killing relieves the problem. See if it comes back right away after restarting.

After that, you'll need far more detailed analysis than can be done in this forum. I would suggest opening a support call at that time.