memory 100%

[John_1017]

Hello:

Can someone tell when this would happen?

dev 262144 67386 194758 25.71% 0 -
reserve - 194758 -194758
memory 385984 385984 0 100.00%

I am curious as to why dev is 25.71% used, memory is 100% used and still the system crashed. This information is from syslog

[A. Clay Stephenson]

This by itself would not crash the system although it would almost certainly perform very poorly. In this case, "memory" is actually pseudoswap is now completely "used up" and real swap has come into play. The system shouldn't crash when it exhausts all virtual memory but it should refuse to fork new processes and begin to deactivate processes.

[Kent Ostby]

Kiran --

There are cases where this could crash the system.

A lot depends on how you define "crash".

If you mean it hung and nothing would run well that is because you are out of memory.

Your choices at that point are to "TOC" the box so HP can read the dump OR if you can still get in, you might run something that is cheap on memory to see who is the hog.

I like:

ps -ef | cut -c42-80 | head

which will show you the top CPU hogs. This are also a lot of times the memory hogs and you can see who to kill.

If by crash, you mean that it rebooted itself, it is possible especially if you are running under Serviceguard and the lack of memory caused a SG TOC .

Other critical programs not running could cause in rare cases, a hang as well , i.e. Spinlock Deadlock problems.

[John_1017]

Hello both, Would vm eventually become 100% when memory leak exists? what would a leak effect first vm or swap?


Top's virtual memory details:
Memory: 1204948K (98112K) real, 147292K (111512K) virtual, 121124K free

A process runs with 1gb memory
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 1297 weblogic 152 20 1264M 1101M run 36:19 0.80 0.80 java

Is this system safe with just 121mb free? The process though using 1gb is not effecting the system ... do you then think the memory leak caused the vm 100%?

[A. Clay Stephenson]

A process with a memory leak would eventually fail to allocate memory. It's behavior from that point is up to the programmer. It typically would log an error message and exit and then the machine would recover the lost memory. Of course, there is no way to know the behavior of someone else's code.

[Bill Hassell]

A memory leak is just a name for sloppy programming...it is not soemthing you can fix in the opsystem except to kill the process. It may be perfectly normal for a process to grow in size for a while. The problem comes up when the process expands without limits, eventually using all RAM, then using all swap space, or it hits a kernel limit like maxdsiz.

NOTE: Always use swapinfo -tm. As mentioned, the memory line in swapinfo refers to RAM in pseudo-swap. The summary line with the -tm option will show total virtual memory. The listing in top concerning memory usage is quite difficult to understand and the man page is not much help.

Now as for a system crash, do you mean that the entire computer halted and rebooted with a system panic? Or do you mean that the application stopped running and created core file(s)?

If you want to see which progams are using the most memory, use this command:

UNIX95=1 ps -eo vsz,pid,ppid,ruser,args | sort -rn | head -20

Keep running this command every few minutes after starting your application and see if some processes grow all the time.

[John_1017]

Hello,

A program created core in root file system which stopped the system. That program is not the one with the memory leak. It appears the program dumped core because it couldn't fork processes as the memory was 100%. In other words, did the memory leak exhaust the pseudoswap? And, if there were no memory leaks, is this system safe with this?
swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 6 1018 1% 0 - 1 /dev/vg00/lvol2
reserve - 1018 -1018
memory 1508 680 828 45%
total 2532 1704 828 67% - 0 -

[Bill Hassell]

If you cannot fix the bad programs, you can at least prevent the core files from occurring. First, it sounds like the program has a working directory of / since you said that the core file was formed in the root directory. Most likely the superuser's $HOME directory is / which is a bad situation (yes, it's the default for several flavors of Unix but it is a bad thing nonetheless). You may want to change the way in which the program starts by changing out of / (the most dangerous directory to be in) and also add the following statement to your program startup script(s):

umask -c 0

If this fails (gives an error), you're using an unpatched version of ksh--/usr/bin/sh works just fine. What this does is to prevent cpore files from forming at all. You can even put this into /etc/profile to prevent corefiles from being created although if you have programmers that need corefiles to fix problems, change the command to:

ulimit -Sc0

You certainly have very small memory and if you're trying to run processes that need more than 1.5Gb of RAM, then you need to add a lot more swap, perhaps another 2-4Gb. However, swapping (actually, paging) will have a very, very serious performance hit, so you may be able to run everything but it will be very slow. The only way to improve performance is to add a lot of RAM, perhaps an additional 4Gb of memory.

Your swapinfo details do not show any usage but unless you run swapinfo when the program is consuming a lot of virtual memory, it won't be a useful metric.

And to clarify, a program that is supposed to use a large amount of memory is normal but a memory leak is a mistake (the program isn't supposed to use the memory). You must obtain the memory requirements from the manufacturer or programmer to see if it is normal.