Memory not being released?

[James Murtagh]

Hi Erwin,

As you said I wouldn't jump to any conclusions just yet, a bit more analysis is needed.

I don't know what Steve's script was but a fair guess would be the UNIX95 version of the ps output. The one thing everyone forgets about this is that it doesn't report on shared memory segments or shared mmap areas. For the shared memory usage use:

# ipcs -mob

If there are large segments this is where the memory is!

Otherwise you will have to run the script several times to identify what process is continually asking for more memory. If the processes is 32bit and it is not releasing its memory correctly it will eventually die from an ENOSPACE or ENOMEM error.

The buffer cache is only low as most of it has probably being flushed due to the memory pressure. Trim it down to 300-400 MB.

Regards,

James.

[Bill McNAMARA_1]

-- finding mem leaks from cli

You can monitor for memory leaks from the command line. Assuming your process is running in background or as a daemon, identify the PID by:

# ps -ef | grep
# # Note the PID
# #
# # in what
# # follows.

# export UNIX95=1 # Turns on XPG4
# # behavior of 'ps'
# # (needed for -o
# # option to work)

Now display physical and virtual size of the process:

# ps -p -o sz,vsz,pid,state

If there is a memory leak, the values of sz (physical pages used) and vsz (virtual pages used) will gradually increase while the code executes.

[Bill McNAMARA_1]

check out also psram from:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x026250011d20d6118ff40090279cd0f9,00.html

Later,
Bill

[Jdamian]

Swapinfo output shows:

memory 1934 1891 43 98%

I think your box has no memory enough due to YOUR SWAP SPACE IS NOT ENOUGH.
Your swap space is 1 GB on /dev/vg00/lvol2 but your box has 2.5 GB physical memory (see your report above).

When system fills up the swap space, it starts to use physical memory as swap (PSEUDO-SWAP).

I should run:

1. kmtune -q maxswapchunks -q swchunk.

read the values and multiply them... this outcome is the kernel limit (in KB) of swap space.

2. if this limit is bigger than 1 GB (the current swap space on dev on your box) you can add new swap space.

a) create a new logical volume. It must be contiguous.
b) add it to swap (swapon command)
c) edit /etc/fstab to reflect this config change.

After these steps, swapinfo will report a lower percentage of memory.

Good luck.

[Michael Steele_2]

Most significant data is 'swapinfo -tam':

total 2958 2800 158 95% - 0 -

The swap test for adding more swap is when 'total % utilized exceeds 85%.

lvcreate -L 1024 swap -C y -r n /dev/vg##

swapon -f -p 1 /dev/vg##/swap

If error:

sysdef | grep -i -e maxswapchuncks -e swchunk

maxswapchuncks = total swap / 1024 * swchunk

/etc/fstab
/dev/vg##/swap ... swap pri=1 0 1

NOTE: keep all swap priorities = 1.

[Bill Hassell]

It's pretty rare these days to find any HP-UX memory leaks, so the advice to look at growing apps is good advice. If the biggest applications continue to get bigger every day (every hour?) then you can assume:

1. The app is supposed to do that and the vendor will verify that it is the expected behavior (read: add more RAM, perhaps forever?)

2. The app is not supposed to do that and the behavior is either a coding error in the app, or the libraries used by the app need patching.

2001 is really old for an L1000 (check with: swlist -l bundle). If there have been no other patches added since that time, then things are not in a very good state. If the Ideas and/or NetExpert software uses Xwindows, the underlying libraries for Xwindows have a long history of memory leaks. This system is probably in great need of a complete patch bundle (both HWE and QPK patch sets, Dec 2002 or later)

Pageouts (the only useful metric for low memory problems) may be caused by memory mapped files. Adding more swap space will not improve anything since you are not using all of it even now. And using swapspace is a bad thing anyways! Ideally, swapspace will be used only occasionally (and by memory mapped files for now) and there is plenty of room for the processes you have listed.

Having 100% memory usage is not a bad thing although if Glance shows only 128 megs for the buffere cache implies that memory pressure has indeed occurred and the cache has been automatically reduced from the dbc_max_pct value. The small cache size may limit the benefits of a larger cache (perhaps 250-400 megs).

[Erwin Mascardo]

Thanks so far. No points yet for the folks suggesting more swap space, since at the moment I'm not in a position to add any. (No disks available, and no good time to take the system down to rearrange things.) As a stopgap-to-the-extreme measure, I could add some filesystem swap and see if it relieves the pressure at all.

The script I had used was actually psram, though it wasn't named such when I found it posted. Thanks anyhow, Bill.

So from what I'm seeing so far, the paging is actually the cause of the high memory usage (due to reserving), rather than a symptom of it?

[Erwin Mascardo]

Another clarification point: some of the apps are indeed growing, but this is in line with expected performance. It's to the tune of 10MB (or less) per day in total, and due to regular reboot-required software deployments, the machines almost never stay up for more than 8-10 weeks at a time. So generally we wouldn't notice.

Patch bundle level is June 2001. Ancient. It would take some string-pulling to get the system patched ahead of our next deployment, though, which once again is in mid-June. (We had one scheduled for April, which circumstances pushed to May, and now it's all happening in June...) Not worth explaining why, it's just how this shop works.

[Michael Steele_2]

so what's the question?

Memory leak?

Use glance advisor and note the total VSS metric.

# cd /opt/perf/examples
# glance -adviser_only -syntax ./proc_mem_leak -j30

Free up Virtual Memory?

ipcs -mob
ipcrm -mid

[Bill Hassell · Accepted solution]

Paging occurs because memory has been exhausted (well, technically, pressure from additional apps starting or asking for more RAM) and the system starts two mechanisms to accomodate this pressure:

1. reduce the buffer cache

2. start deactivating idle and/or low priority processes, which are then candidates for paging out to swap

You can add gigabytes of swap and nothing will happen since you aren't using all your swap space even now. Swap does not make the system run better except to allow more processes to run than will actually fit in RAM.

NOTE: #1 (reducing the buffer cache) is painfully slow in early versions of 11.0 (read: old patch bundles) so #2 will start at the same time. A well-patched 11.00 system will have better performance in this area, and 11.11 has a major rewrite of the buffer cache handler that really improves the overall behavior.

However, as mentioned before, page outs may be due to memory mapped files which is (currently) by design.

[Mike Stroyan]

The really troublesome part of the situation is that 1934MB of the
system RAM is tied up as memory swap. It seems that the system had a
high VM usage period when all of the device swap was used and then
almost all of the memory swap was used. Some 425MB of device swap has
freed up, but the memory swap remains. That leaves only 626MB of RAM to
use for all the active procesess.

It is unclear what is holding that memory swap in place. If the process
or shared memory VM totals showed that much VM, then I would advise restarting
some bigger memory user to give it a chance to change from committed swap
locations to either reserve or device swap. Since you can't find any use
for that big VM space, it seems that the memory swap has leaked.

One experiment you could try would be to create a high VM usage process
to see how much swap it can get. There is an outside chance that a new
process with big VM needs would cause the memory swap to be reassociated
with the new process. Then stopping that process could free the memory
swap. You could use a simple program like the attached hog.c. Compile
it with "cc -o hog +DA2.0W hog.c" to make it a really big hog.

[Erwin Mascardo]

Well, the machine went and crashed on us a couple of days ago. The problem here was that it's part of a ServiceGuard cluster, and its companion was in a similar state, so we couldn't simply do a package switch and let everything reset itself. Right now the main machine is running, and the other cluster box is mostly idle, so when the memory gets tight again, we'll see if a package switch helps. We'll get it patched properly during the next maintenance window and we'll see if that fixes things.

Thanks to everyone.