HPUX Memory "leak"

[Tim Killinger]

We have a family of 7 separate rp5450s running HPUX 11.0 and Oracle 8 and an iplanet served web interface. All boxes are configured (nearly) identically.

Of the 7 servers in production, 2 of them seem to experiencing a "memory leak" where, over the course of a week or two, memory consuption grows from %50 at boot time to 100% and we see symptoms of the application timing out for a few users. A reboot puts us back to %50 and the cycle begins again.

The other servers are fine - memory utilization remains more or less static - no rebots are necessary.

My guess its that it's application code consuming memory over time, but I say that only because of UX's track record for stability.

We are at a loss as to how to trouble shoot this problem. Can anyone offer any suggestions as to what tools or methods we might use to find out the cause? Has anyone ever seen behaviour such as this and found a method and or solutions?

Thanks in advance!!

[A. Clay Stephenson]

The first thing you need to do is get a handle on which processes are growing. I would set up a cron job to run every 10 minutes ort so that does a date command and then a
UNIX95= ps -e -o comm,vsz,etime,stime,time

This data should be appended to a logfile so that you now have a clear record of process sizes vs time.

I would also compare kernel tunables between the "good" and "bad" systems and also pay particular attention to the patch levels. Are the Oracles all patches to exactly the same level? All run the same version of libC patches?

Until you zero in on the processes that are growing, it's difficult to be more specific.

[Bryan D. Quinn]

Hello Tim,

I have experienced this situation a couple of times. Here are a couple of things that I would advice you to do or look at doing.

1. Update patches, this resolved my problem in one situation.

2. Run the following command to monitor processes using memory:

UNIX95= ps -e -o ruser,vsz,pid,args | sort -rnk2 | more

(NOTE that there is a space between the = and ps.)
This command will list your processes in order from largest memory useage to least.

If you have Glance, that would be great for monitoring memory and processes in place of the above command.

Using the command above I was able to track one process over about a weeks time that was growing in memory utilization. I talked with our development people and they were able to resolve the issue.

Hope this helps!
-Bryan

[Marvin Strong]

UNIX95= ps -e -o ruser,vsz,pid,args| sort -rnk2 | more


or redirect that to a file or a few files, should help you find the process.

[Elena Leontieva]

Tim,

To check for memory leaks, generate outputs of the following command on a regular basis and see if the first column for any of the processes is growing over the time:

UNIX95= ps -e -o "vsz args" |sort

HTH, Elena.

[Steven E. Protter]

If all the boxes were configured identically, then there should not be a leak right?

Perhpas a process with a problem is only being used on the effected box. Or part of the Oracle, which is out of support btw has become corrupt.

I would follow A. Clay's suggestion and identify the process. If its an Oracle process, relink oracle. If that fails, you may need to copy modules from a good server to a bad server.

I'd also go to itrc.hp.com patch database and perform patch analysis. You may find your machines are not so identically configured as you think. There are several memory leak patches available in the patch database.

You also want Oracle up to 8.1.7.4, which deals with a whole host of issues.

SEP

[Tim D Fulford]

Hi

From your posting, I assume you mean that 2 servers experence a memory leak no-matter what runs on them. If this is the case it points to patching/configuration (or possibly bad network, but that is stretching the point).... If however you always run the same things on these two servers it seems logical the memory leak in in the app.. If so check the apps configuration..

The problem with problems is that it can be a problem defining what the problem is... if you catch my drift??

Tim

[A. Clay Stephenson]

Another area to look at is not a memory leak but rather more and more processes. If, for example, a process that you normally expect to spawn and then exit after doing some task never exits then you eventually run out of memory witout any leaks at all. The cron'ed date and ps -e's to a log will spot this sort of thing as well.

[Bill Hassell]

As mentioned, this is very likely applications that are consuming more memory. As mentioned, sort the processes by RAM usage using the XPG4 option (UNIX95= ) to show vsz:

UNIX95= ps -e -o vsz,uid,args | sort -rn

Run this into a file on ALL of your servers. Then compare the different files to see what is different about the biggest processes. And for good measure, count the logins and processes:

who | wc -l
ps -e | wc -l

on each machine. One additional note is that you may have apps consuming shared memory. This is not counted in ps ecause it is shared by many processes. Start with:

ipcs -bmop

and compare size and owners/processes that use shared memory. Then get a copy of shminfo to detail shared memory usage: ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

(note: IE 5 and 6 have problems with ftp when folder-view is enaabled. Turn off folder-view in Tools->Internet options->Advanced)