Memory

[Roger Baptiste]

I am trying to debug the sudden spike in memory usage in some of
my systems (all 11.00 , V, K amd N class). Memory usage jumped
up by almost 25% on the system(s) at around the same time; the only
common factor noticed at the time of memory usage rise on the systems was a sudden
spike in the Network Input packets coming into the system. This was
for a short period of 5 minutes.

Apparently that memory which was grabbed due to the network input packets
has not been released. WHen i run kmeminfo, i see a huge value
corresponding to dynamic data column. In this particular system case it is around 2.7g.

Physmem = 2097152 8.0g 100% Physical memory
Freemem = 638363 2.4g 30% Free physical memory
Used = 1458789 5.6g 70% Used physical memory
System = 1018870 3.9g 49% By kernel:
text = 1880 7.3m 0% text
data = 223 892.0k 0% data
bss = 683 2.7m 0% bss
Static = 98851 386.1m 5% for text/static data
Dynamic = 706161 2.7g 34% for dynamic data
Bufcache = 209715 819.2m 10% for buffer cache
Eqmem = 47 188.0k 0% for equiv. mapped memory
SCmem = 4096 16.0m 0% for critical memory
User = 444136 1.7g 21% By user processes:
Uarea = 5080 19.8m 0% for thread uareas
Disowned = 8 32.0k 0% Disowned pages




Apparently this memory leak is not corresponding to any running
process on the system. I confirmed it by running the UNIX95=ps command;
I also checked glance memory, ipcs output and do not see any
running process holding the memory.

My question is , has anyone else seen this problem??
Next, is there a way to free the memory which has leaked/grabbed ??
without a reboot.


thanks
roger

[Paul Sperry]

Almost sounds like a virus.
The only way I know of the fix memory leaks is a reboot:(

[Caesar_3]

Hello!

If you have memory leaks only reboot will
save you, because you don't have any tool
that can free allocated memory by proccess.

Caesar

[Roger Baptiste]

<>

Paul, are you serious??
I would like to see what the networking gurus have to say about it.

[Roger Baptiste]

<>

Paul, are you serious??
I would like to see what the networking gurus have to say about it.

[Dave La Mar]

Roger -
I have this in my .kshrc list of aliases -
alias HOG=`UNIX95= ps -eo "pid ruser pcpu vsz=Kbytes" -o args | sort -rn -k 3 | head -10`

This give me the PID USER %CPU MEMORY in K for the top ten problems processes.

I have used this time and time again to kill a memory or cpu hog process.
Naturally, care must be taken if dealing with Oracle, etc., prior to killing any process.

Best of luck.

Regards,
dl

[Roger Baptiste]

Dave,

I had tried the Unix95 PS. It doesn't show any process consuming more than their normal usage. The memory which has been grabbed doesn't seem to be associated with any process. Even ipcs (shared mem) looks good. Only the kmeminfo output shows the memory surge is associated with dynamic data (not sure what that means). And please note the memory got gobbled in the first place due to a huge network input packets. Not sure about the cause.
This reminds me of windoze enviroment which was/is captive to a network flood.

thanks

[Steven E. Protter]

There are OS patches for certain OS related memory leaks. All require a boot.

There are patches from the various application vendors that deal with the leaks they cause. You will have to contact your app vendors for those.

I'm attaching a set of background performance data collection scripts that might be of assistance in gathering more data.

itrc's search engine isn't producing results but you should try it later.

Here is a search on 11.00 based on memory leak network which produced 76 patche results.

http://www1.itrc.hp.com/service/patch/search.do

SEP

[Steven E. Protter]

Sorry, here is the promised attachment.

[Dave La Mar]

Roger -
Sorry I do not have the quick answer for you, but on the search there are a number of hits for dynamic data memory.
Here is the link -
http://us-support.external.hp.com/emse/bin/doc.pl/sid=e56a6dac1905d6cefe?todo=search&searchtext=dynamic+data+memory&x=27&y=2&searchcriteria=allwords&searchtype=SEARCH_FORUMS&searchcategory=ALL&rn=25&presort=rank

There are many entries for different environments. Hopefully, one will fit your environment and suggest what needs to be tuned.
In our shop, we have a very large Oracle database nad experience memory leak on a regular basis. We have not identified all the processes causing this, therefore, we have had an automated reboot once a week for about the last nine months. During the day we run the Glance report utility and send out a warning when memory gets below 1gb. It has been our experience that once below 1gb we have a hog process. Knock on wood, have not had a warning in months and may have a handle of the culprit processes as well as have the kernel properly tuned for our environment.

I can certainly relate to what you are experiencing.

Best of luck and best regards,

dl

[Roger Baptiste]

This memory hole problem is apparently not due to database/application leaks, since it is on a bunch of systems with/wo database. Attached is the output of kmeminfo These three entries seems unusually high:

Dynamic = 706019 2.7g 34% for dynamic data

bucket[9] = 308651 1.2g 15% size 512 bytes
bucket[11] = 280542 1.1g 13% size 2048 bytes

Any further clues on how to bring it to normal?
Roger

[rick jones]

IIRC, the memory isn't so much allocated as no longer in the completely free and clear pool. If you see it in the pool listings, it ain't leaked :) It has been freed back to the memory subsystem.

The usage in the 512 and 2048 byte buckets is consistent (but not a complete smoking gun) with networking traffic - given the size of farious mblk headers and such the smallest allocation for a packet will be 512 bytes, and then for the inbound buffers, 2048 is not surprising.

If a large enough spike in network traffic happened, it could have caused stuff to be queued in various places in the system and thus a larger allocation of memory.

I'm _far_ from a kernel memory subsystem expert, but over time, the system will try to "reassemble" those smaller memory buffers into larger ones, but I'm not sure the memory will ever appear as completely free and clear again. This tends to make leak detection somewhat more difficult...