ipcs -m in rx8640 11.23 server

[Jojo Castro]

Hi All...

I have a clustered server (separate servers) running in 11.23 as OS SFRAC 4.1 oracle10g. My database administrators and developers just reported me the high memory utilization 80% on node1 and 75% on node2. After they have killed applications and shutted down oracle, node2 memory utilization dropped to 20%. But in node1, it still as high as 67%. I have run kmeminfo and the output in both nodes are almost the same. But the ipcs -m command shows that the shared memory segments that root has been eating up in node1 is high than that of node2. Please see the outputs below:

NODE1:
# cat ipcs.txt
IPC status from /dev/kmem as of Thu Aug 30 14:24:06 2007
T ID KEY MODE OWNER GROUP
Shared Memory:
m 0 0x411c07db --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x41208954 --rw-rw-rw- root root
m 3 0x00a5c581 --rw------- sfmdb users
m 65540 0x06347849 --rw-rw-rw- root root
m 131077 0x0c6629c9 --rw-r----- root root
m 7 0x49180468 --rw-r--r-- root root
m 4718603 0x4d4e5251 --rw-r--r-- root sys
m 458764 0x55315352 --rw-rw-rw- root sys
m 458765 0x44525354 --rw-r--r-- root sys
m 458766 0x53494152 --rw-r--r-- root sys
m 262159 0x00005643 --rw-rw-rw- root sys
m 262160 0x00005654 --rw-rw-rw- root sys
m 65553 0x0000cace --rw-rw-rw- root sys


NODE2:
$ cat ipcs.txt
IPC status from /dev/kmem as of Thu Aug 30 14:24:34 2007
T ID KEY MODE OWNER GROUP
Shared Memory:
m 0 0x411c07db --rw-rw-rw- root root
m 1 0x4e0c0002 --rw-rw-rw- root root
m 2 0x412089cc --rw-rw-rw- root root
m 3 0x00a5c581 --rw------- sfmdb users
m 196612 0x0c6629c9 --rw-r----- root root
m 5 0x06347849 --rw-rw-rw- root root
m 32774 0x49180468 --rw-r--r-- root root
m 458763 0x4d4e5251 --rw-r--r-- root sys
m 360460 0x55315352 --rw-rw-rw- root sys
m 360461 0x44525354 --rw-r--r-- root sys
m 360462 0x53494152 --rw-r--r-- root sys
m 196623 0x00005643 --rw-rw-rw- root sys
m 196624 0x00005654 --rw-rw-rw- root sys
m 98321 0x0000cace --rw-rw-rw- root sys

For your expertise, can you please help me how to explain these? Is there a possibilities that i can point what particular process consume share memory segments?

Since I don't know what to kill, I just decided to reboot the systems since we are still in POC. Memory utilization is now normal.


Thanks in advance.

[Wim Rombauts]

Since all shared memory segments are owned by root, it's hard to tell what to do. Certainly because both servers show the same. That makes me think you don't have a shared-memory issue, but something els.

If you ever encounter the same issue, run ipcs -ma. The "a"-option will show more details about memory segment sizes and connected processes.

Further, there is a tool called "kmeminfo". It's not standard on your system, but you can get it through HP support (or maybe a download on the internet somewhere). With this tool, you can better see where the exessive memory usage is going to (buffer cache, private segments, kernel memory, shared segments, ...)

[Jojo Castro]

Wim Rombauts Aug 30, 2007 08:18:49 GMT 5 pts

--------------------------------------------------------------------------------
Since all shared memory segments are owned by root, it's hard to tell what to do. Certainly because both servers show the same. That makes me think you don't have a shared-memory issue, but something els.

If you ever encounter the same issue, run ipcs -ma. The "a"-option will show more details about memory segment sizes and connected processes.

Further, there is a tool called "kmeminfo". It's not standard on your system, but you can get it through HP support (or maybe a download on the internet somewhere). With this tool, you can better see where the exessive memory usage is going to (buffer cache, private segments, kernel memory, shared segments, ...)

Thanks for the answer Wim.
Actually, as stated in my mail, I used kmeminfo and the output on both nodes is just the same in terms of memory, rss utilization. Both servers dont show the same.
Please see the ipcs output of node1 and this is what i suspected.

m 4718603 0x4d4e5251 --rw-r--r-- root sys

Thanks!

[Dennis Handly]

As Wim said, we don't want to see any ipcs -m output unless you use -ma. Besides giving the important size, has other useful info.

And while you are at it, provide "swapinfo -tam".

You say the kmeminfo output is almost the same, what percent of the output is the same?

[Jojo Castro]

Please see attached kmeminfo output.

[Dennis Handly]

>Please see attached kmeminfo output.

Your total physical size is close.

But would about kernel tunables for the disk cache? dbc_max_pct(5) dbc_min_pct(5)

[Duncan Edmonstone]

Hi,

Please note that in the row you suspect:

m 4718603 0x4d4e5251 --rw-r--r-- root sys

The value 4718603 is NOT a size, it's a shared memory ID.

As already noted, you need to run ipcs -ma to see the size of these areas.

HTH

Duncan

[Jojo Castro]

Thanks to all.
Nextime i encountered the problem, i will issue the ipcs -ma.

Thanks!