Strange memory issues on HP-UX 11.11 Machines

[kj_2]

Hi Everyone,
I have encountered on two of my Superdome partitions over the last few months a strange memory problem which i am sure is application related, but as these machines are running quite a number of apps, i am having trouble tracking down the culprit. My machine configuration is as follows:

2 x HP9000 Superdome Partitions.
Host1 = 8CPU/8Gb RAM + 16Gb Swap
Host2 = 16CPU/16Gb RAM + 32Gb Swap
(Both are patched with a set of HP MC patches which are sent to our site for our production servers to keep our patch levels close, if not the same)

Once every month or so i get reports that the system has run out of memory from my users, if i run a swapinfo with no arguements i get the following information.

dev = 1% used
memory = 100% used.

There is definately memory related issues occuring as i am unable to run commands until i get a bunch of users to log out to free up just enough memory for me to atleast use the shell and related commands (grep, kill etc)

It seems that there is some kind of memory usage spike which isnt swapped out. My guess is that its a shared memory related problem, as i recall, shared memory segments are never swapped out to disk, which could explain why the memory here goes to 100% and nothing is being swapped.

I configured my glance parm file to sort processes into per user ID groups so i could look at memory utilisation at a per user basis.
Upon doing this i have noticed that there are a few users that have a large amount of 'Virtual Memory' being used by them, some as high as 62Gb at times!. I am sure this is not a normal occurance.

This problem used to only occur on Host2 and now also occurs on Host1 and i've been unable to track down the causes of this issue, so i guess my question(s) to you is this:

In the case of strange memory uses, what are the best commands to use to break down the memory usage on a per user or per process basis ?
(So far i have glance, and the UNIX95 command which i have found througout the forums)


Is there a way of displaying how much shared memory is being used on a HP-UX system and by what applications ?
(I know applications like oracle use alot of shared memory, can i track what instances etc are using how much ?)

As this happens infrequently and unpredictibly, i want to be armed with the right tools to attack it when it next occurs.


I look forward to your feedback,

Thanks

./kj

[G. Vrijhoeven]

Hi,

You might want to take a look at the prmavail MEM command. to see wich part of the memory is consumed by the OS ( if you have prm installed(MC installations have). If you want to see the share memory you can take a look at ipcs ( man ipcs for more info). An other issue might be the maxdsiz and maxdsiz_64bit. This can be checked with glance ( memory regions).

HTH,

Gideon

[Jdamian]

For seeing the shared memory use:

ipcs -ma

please, can you show the output of
swapinfo -tam

[Jdamian]

sometime kmeminfo is a useful tool. It is located at /usr/contrib/Q4/bin

[Jdamian]

What is the value of dbc_max_pct ?
This value limits the amount of memory occupied by file-system buffer cache ?

In HP-UX 11.00 its default value is 50 % (I don't know the default value in HP-UX 11.11). If this is your case your system might be allocating too much memory (4 GB in host1, 8 GB in host2) for the buffer-cache.

You can read the amount of physical memory used as buffer-cache in the output of kmeminfo command. See the example below:

Physical memory usage summary (in pages):

Physmem = 262144 Available physical memory:
Freemem = 9245 Free physical memory
Used = 252899 Used physical memory:
System = 183689 by kernel:
Static = 14502 for text and static data
Dynamic = 90013 for dynamic data
Bufcache = 78643 for file-system buffer cache
Eqmem = 19 for equiv. mapped page pool
SCmem = 512 for system critical page pool
User = 65795 by user processes
Uarea = 2736 for thread uareas
Disowned = 6522 disowned pages

[Emil Velez]

First

What OE do you have.. If you have the mission critical OE you may have OVPA formerly called measureware collecting data on many things like buffer cache sizes and memory useage across time every 5 minutes and for selected processes every one minute. I would look at these metrics of memory used and buffer cache used across time and see when and how and when it spiked. Then I would see if any programs are growing in their memory useage.

Using OVPA you can collect data on arbitrary groups of processes by creating entries in the /var/opt/perf/parm I cannot teach you everything about this but this is where I would start.

Good luck