Memory leak in HP-UX 11.00 NFS

[Michael Lampi]

There appears to be a memory leak in HP-UX 11.00 NFS v3. Even with the latest quality pack patch bundle (December 2000) installed, it appears that a simple readdir(3C) program accessing a remote NFS file server can cause the loss of available memory on the local client, until the local system effectively hangs itself by thrashing.

Does anyone know of a patch available that fixes this problem? A search of the patch database with the keyword "nfs" did not appear to come up with anything I have not already installed.

Attached is a copy of my program.

[Steven Sim Kok Leong]

Hi,

PHNE_20344 is a recalled patch that fixed the memory leak in NFS v3 in HP-UX 11.00 for readdirplus ().

======================================
libnfs.a
SR:8606110199
DTS:JAGab82902
Memory leak in NFS version 3 at rfs3_readdirplus().
======================================

I suppose you have already given its superceding patch PHNE_22125 a try since you have already installed the Dec 2000 patch bundle?

Perhaps you would want to swlist to confirm that this patch is in.

Hope this helps. Regards.

Steven Sim Kok Leong
Brainbench MVP for Unix Admin
http://www.brainbench.com

[Russell Davy]

Hi there
I agree with Steven above, but had an issue with having to reload cumulative patches for results. Perhaps you could set NFS server and client to 0 in the /etc/rc.config.d/nfsconf and reboot, then try reload abovementioned patch.
BTW: nice command to find memory leaks:
UNIX95= ps -e -o ruser,vsz,pid,args | sort -rnk2 | more

Note the space after UNIX95=
This lists all processes on the system and how much memory they take.

Cheers
MND

[Michael Lampi]

Thanks a lot for the suggestions above. While I thought I had installed all the latest patches apparently I had not. The ones suggested above plus the "ps" per task memory sort helped to track down the major memory leak problem, which turned out to be the mib2agt leak, which is fixed in PHSS_21046.

At this time there is still a memory leak, but I can not attribute it to any process. It is substantially smaller than before the mib2agt patch was applied, but it is still there.

Any ideas?

[Marc Dijkstra]

Michael. Same guy as above, just finally changed my profile from the one I borrowed!! (got sick of people calling me Russell!)

I did a search for "memory leak" under the patch database and came up with a few.. particularly the lan cumulative patch. Did the command I mentioned above pick up any specific leaks of any major consequence?

[Michael Lampi]

No, the command listed above did jnot uncover _any_ memory leaks.

After growing the number of files in the NFS server directory, I have now managed to cause my C3600 to grind to an absolute halt. vmstat shows all memory is allocated, and top shows vhand to be the top process - no surprise there.

Unfortunately, I have been utterly unable to track down exactly what is causing the memory to be allocated.

[Marc Dijkstra]

Hmm.. Ok obviously with the vhand up so high the swap is killing itsself, does a swapinfo -t indicate this too? Are there ANY errors in /var/adm/syslog/syslog.log? also could you attach on your reply the following:
1)swap size
2) amount of memory in the C
3) the parameters maxdsisz, maxtsisz and maxssisz as well as maxswapchunks from your /stand/system file (if they aint there check SAM, but this indicates that they haven't been pumped up)
4) also if you could a cut-n-paste of the top output.

Have you got a demo copy of glance running on the machine, this may also help drill down to the problem and would be prudent to load.

MND

[Michael Lampi]

# swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1048576 0 1048576 0% 0 - 1 /dev/vg00/lvol2
reserve - 114528 -114528
memory 381448 106964 274484 28%
#
# The following vmstat occurred after some of the RAM
# appears to have been leaked. It starts with some
# 87000 free.
#
# vmstat
procs memory page faults
cpu
r b w avm free re at pi po fr de sr in sy
cs us sy id
0 0 0 4297 67877 65 17 0 0 17 0 0 142 1318
47 1 2 97
#

The C3600 has 512MB RAM.

maxdsiz 0XC0000000
maxdsiz_64bit 0X400000000
maxfiles 200
maxfiles_lim 2048
maxssiz 0X04FB3000
maxssiz_64bit 0X10000000
maxswapchunks 4096
maxtsiz 0X40000000
maxtsiz_64bit 0X100000000


------Note:
After downloading and installing over 100MB of additional patches, it appears that things are working better, at least with the "ls -f1" command. It still takes forever, but the memory leak in that case seems to be gone. Why "ls -f1" should require over 1GB of virtual space to list (unsorted) 8 million short pathnames is rather ridiculous, but that's a different problem.

Using my "ls -f1" replacement program (supplied above), I detected a memory leak, but after reapplying an ARPA related patch that leak seems to have gone away.

Weird!

It is also rather strange to find that swlist does not list all of the patches installed. I had to "what /stand/vmunix" to find many of the kernel and some of the network related patches.

[Mladen Despic]

Michael,

If you have a recent (trial) version of Glance, you can
try the following. Assuming

mysyntax=/opt/perf/examples/proc_mem_leak

verify that the file $mysyntax exists. If it does,
try running

glance -adviser_only -syntax $mysyntax

You can redirect the output to a file, leave the process
running for some time, then review the results.

You can also modify a copy of $mysyntax to define your
own thresholds, add other Glance metrics, etc.

[Mladen Despic]

Correction:

myfile=/opt/perf/examples/adviser/proc_mem_leak

[Michael Lampi]

Current vmstat shows:
# vmstat
procs memory page faults
cpu
r b w avm free re at pi po fr de sr in sy
cs us sy id
0 4 0 1954 671 72 5 60 17 121 0 154 200 196
84 2 9 89
#

Not a whole lot of virtual memory available!

[Marc Dijkstra]

The swapinfo indicates that there is no swap being used in a mail about 3 up, was this during an inactive period? If so what does a swapinfo report when the machine is running flat out with your app?

I cannot for the life of me see the problem. As for the Glance, do you have the application CD's that came with the C-class, if you do pwrhaps it would be prudent to load Glance PAK and see what we can find....

MND

[Michael Lampi]

Since I posted the note above about the ARPA patch clearing up the memory leak for my "ls -f1" replacement program the file volume has grown to about double its previous size.

Running the replacement program (attached to the first message above) seems to cause the memory to leak from *somewhere*, but nothing so far seems to reveal exactly where.

I have been unable to locate my Applications CD so far, so Glance is not (yet) an option, but I'll look for it again.

[Michael Lampi]

Another data point has been found. Mounting the remote file volume using NFS v2 does not create the problem on HP-UX 11.

So, it would appear that there is a definite problem with NFS v3, even with the application of all the available patches for HP-UX 11.00 NFS.