topic Re: High memory utilisation in Operating System - HP-UX

High memory utilisation

Vee_1 — Tue, 10 Jun 2008 16:01:08 GMT

Hello,
I can able to see very high utilization of memory in the below output.Though its having 16GB of physical memory installed ,only 253mb free mem is available .can anybody suggest what could be the issue.

------------------------------------------------------------------------------------------------------------------------------
CPU Util SU | 2% 2% 5%
Disk Util F F | 4% 2% 4%
Mem Util S SU UB B | 98% 98% 98%
Swap Util U UR R | 51% 51% 51%
------------------------------------------------------------------------------------------------------------------------------
MEMORY REPORT Users= 1
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 16 75 2.6 10.1 42.1
Page In 13 34 2.1 4.5 15.0
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 0kb 0.0 0.0 0.0
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 7 7 1.1 0.9 5.1
Deactivations 24 24 4.0 3.2 4.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 0 0.0 0.0 0.0
VM Writes 0 0 0.0 0.0 0.0

Total VM : 17.3gb Sys Mem : 2.8gb User Mem: 11.7gb Phys Mem: 16.0gb
Active VM: 9.0gb Buf Cache: 1.3gb Free Mem: 253mb

================================
uptime
1:00pm up 30 days, 4:26, 1 user, load average: 0.04, 0.04, 0.04
================================

Re: High memory utilisation

Kenan Erdey — Tue, 10 Jun 2008 16:14:16 GMT

Hi,

sys mem seems to be verh high.it can be from memory leak caused by running programs. if it's a database machine you can decrease buffer cache by changing dbc_max_pct kernel parameter. (1.3 gb in your configuration)

Kenan.

Re: High memory utilisation

Steven E. Protter — Tue, 10 Jun 2008 16:53:43 GMT

Shalom,

253 MB is a fairly large chunk of memory to be free.

Perhaps you have an application with a memory leak. Try this script.

http://www.hpux.ws/?p=8

SEP

Re: High memory utilisation

TTr — Tue, 10 Jun 2008 17:12:09 GMT

In your main glance screen, (hit g), look at the RSS column, it provides a good metric of how much memory an application might use. Note I said might because the RSS is the memory that an application has reserved but it may not end up using all of it. The swap utilization looks good at 51%. Hit the w key in glance to see if there is any disk swapping.

Another place to look for memory usage is in shared memory segments, use the command "ipcs -a", it will show you which processes/users are using shared memory.

Controll your kernel buffer cache, search in this forum for dbc_min_pct and dbc_max_pct.

Re: High memory utilisation

Tim Nelson — Tue, 10 Jun 2008 17:21:19 GMT

post the output of the following.

ipcs -ma

and

kmtune |grep dbc
or ( 11.11 or 11.23+ )
kctune|grep dbc

Re: High memory utilisation

Vee_1 — Tue, 10 Jun 2008 18:09:38 GMT

Thanks Kenan,SEP,TTr& tim.
here is the output and the attachment.

# kmtune |grep dbc
dbc_max_pct 8 - 8
dbc_min_pct 5 - 5
#

Re: High memory utilisation

Dennis Handly — Tue, 10 Jun 2008 19:29:14 GMT

>I can able to see very high utilization of memory

This is good, what's the problem?
You aren't doing any Page Outs.
Can you provide "swapinfo -tam".

>here is the output and the attachment.

You ipcs -ma output shows at least two Gb sized segments:
10758 0xac9d4f6c --rw-rw---- oraq44 dba oraq44 dba 53 1201618944
8721 0x00000000 D-rw-rw-rw- q44adm sapsys q44adm sapsys 39 4294967296

Adding these up gives:
Total: 6,818,685,272 count: 48

Re: High memory utilisation

Tim Nelson — Tue, 10 Jun 2008 19:52:45 GMT

you still have about 5-6GB unaccounted for. now you will have to look though glance and RSS sizes..

My next guess is that you have a pile of java applications with huge memory allocs. ( just a guess ).

If you have kmeminfo ( can be found by searching this forum ) you can use that with the -user switch to find these mem hogs.

Re: High memory utilisation

Vee_1 — Thu, 12 Jun 2008 11:34:30 GMT

Here are the outputs
----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 4194304 16.0g 100%
Free memory = 43867 171.4m 1%
User processes = 3069096 11.7g 73% details with -user
System = 1070885 4.1g 26%
Kernel = 735341 2.8g 18% kernel text and data
Dynamic Arenas = 445165 1.7g 11% details with -arena
M_TEMP = 355970 1.4g 8%
M_SPINLOCK = 25524 99.7m 1%
M_SWAP = 13360 52.2m 0%
VFD_BT_NODE = 11310 44.2m 0%
KMEM_ALLOC = 4687 18.3m 0%
Other arenas = 34314 134.0m 1% details with -arena
Super page pool = 43985 171.8m 1% details with -kas
Static Tables = 202609 791.4m 5% details with -static
pfdat = 95988 375.0m 2%
htbl2_0 = 32768 128.0m 1%
nbuf = 31728 123.9m 1% bufcache headers
pfn_to_virt = 15998 62.5m 0%
bufhash = 10240 40.0m 0% bufcache hash headers
Other tables = 15886 62.1m 0% details with -static
Buffer cache = 335544 1.3g 8% details with -bufcache
# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 26272 363 25909 1% 0 - 1 /dev/vg00/pswap
reserve - 16940 -16940
memory 12658 2625 10033 21%
total 38930 19928 19002 51% - 0 -
#
Now the free memory is even more reduced.

------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 1
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 600% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
kernel 12052 12050 154 sdb 4.4/10.5 289398 0.0/ 2.6 6.92gb 225
ps 26621 2016 149 root 0.4/ 0.4 0.0 1.1/ 1.1 56kb 1
midaemon 2322 1 -16 root 0.2/ 0.3 8380.0 0.0/ 0.0 22.1mb 3
java 2016 1 168 root 0.0/ 0.1 2169.5 0.0/ 0.3 35.1mb 18

Re: High memory utilisation

Don Morris_1 — Thu, 12 Jun 2008 15:12:47 GMT

Of course it is... kmeminfo eats more than a few Mb to run.

I don't see a problem in any of the above -- the system is not paging and likely doesn't consider itself under memory pressure so is willing to let the kernel cache objects/pages for performance. (Just under 1% is the default for vhand to start trying to reduce caches like the Arenas (if free and returnable), the Super Page pool (again if returnable) and the DBC. Note the DBC is still at max... pretty clearly the system is happy with where it is at... only if you push free memory lower will it start to reclaim.)

In short -- the kernel believes it is using your memory effectively for performance and you've shown no indications that it is wrong yet. Idle RAM does you no good, so what is the concern here?

Re: High memory utilisation

Vee_1 — Thu, 12 Jun 2008 16:38:00 GMT

But am getting the following error message while taking ignite image.
Pid 10895 received a SIGSEGV for stack growth failure.
Possible causes: insufficient memory or swap space,
or stack size exceeded maxssiz.
sh: 10895 Memory fault(coredump)
(with latest ignite version c.7.6)

Re: High memory utilisation

Don Morris_1 — Thu, 12 Jun 2008 17:21:30 GMT

Stack growth is more than willing to sleep for memory in the general case, only if you needed mlock'd memory would you fail the growth for RAM -- and you show no signs of exhausting lockable memory (since lockable memory has to consume memory swap and you aren't low).

You show swapinfo -atm output that shows sufficient swap, so that isn't it either. (Barring, of course that what you aren't telling us is that when you take the ignite image it eats all your swap reservation space or something...)

That leaves maxssiz -- what's your value for it? What does the virtual size of the process in question get to before it dies? Most likely this has nothing to do with your memory load -- you just need a higher maxssiz for whatever the ignite image is doing. Or the program just formed a bad address beyond the stack in the first place -- can't really know.

Re: High memory utilisation

TTr — Thu, 12 Jun 2008 17:54:01 GMT

> here is the output and the attachment

I looked at the attachment and noticed that there ia a 4GB segment with ID 8721. It was created by process ID 8309 by the q44adm user. The problem with the segment is that it is marked for deletion (indicated by the "D" in its D-rw-rw-rw- mode column) but process ID 8309 still keeps it open. This may be a transient condition and was cleared up soon after you ran the ipcs -a listing but if it is still there it can indicate a problem with PID 8309. Run teh "ipcs -a" command again and if the same segment/PID are still there, check it out.

Another concern is that you have a little bit of disk swapping.

So other than these two issues I agree with other people that your server's memory utilization is quite good.

You probably need to get a little better control of your apps and be able to detect and clear any issues right away.

On the ignite tape error, this could be caused by the system resources being on the margin or by a limiting kernel parameter that was mentioned in the error.

Re: High memory utilisation

Don Morris_1 — Thu, 12 Jun 2008 18:44:12 GMT

Actually, that's good programming practice for SysV shared memory clients who use segments for temporary shared memory space. You create the segment using shmget(), attach via shmat() and fork off any children you want who also attach. Once everyone is attached (if anyone other than yourself), you use shmctl() with IPC_RMID which marks the segment for deletion once you detach. Then either explicit detach or (much more likely) when you exit, the segment cleans itself up. Considering you can exit unexpectedly due to signals, etc... it is very polite to set up temporary segments to clean themselves up this way.

Re: High memory utilisation

Dennis Handly — Fri, 13 Jun 2008 02:46:58 GMT

>But am getting the following error message while taking ignite image.
Pid 10895 received a SIGSEGV for stack growth failure.

The most probable cause is a programming error of recursive stack overflow. (Or Don's maxssiz.)

You did get a coredump so you can use gdb to get a stack trace.

Re: High memory utilisation

Vee_1 — Mon, 16 Jun 2008 12:50:53 GMT

hi denis,
can u tell me how to anaylze coredump using gdb.As of now i didn't get any error in the server.I seems to be like an intermittent issue.

Re: High memory utilisation

Dennis Handly — Mon, 16 Jun 2008 13:05:09 GMT

>can you tell me how to analyze coredump using gdb. As of now i didn't get any error in the server. It seems to be like an intermittent issue.

You could have been temporarily out of swap space?

To analyze:
file core # gives only basename
gdb path-to-executable core
(gdb) bt
(gdb) q

Another quick check would be if your stack size matches maxssiz. This would be in the first 20 to 40 lines of gdb's "info file".

Re: High memory utilisation

Dennis Handly — Tue, 17 Jun 2008 03:03:25 GMT

>This would be in the first 20 to 40 lines of gdb's "info file".
(gdb) info file
Symbols from "a.out".
Local core dump file:
`core', file type hpux-core.
...
0x7f7f0000 - 0x7f7f8000 is .data
0xc0010000 - 0xc00119d6 is $SHLIB_INFO$ in /usr/lib/dld.sl

It's the last .data before something else.

(gdb) p /x $sp
$1 = 0x7f7f0ee0

$sp is in that range.

(gdb) p -(0x7f7f0000 - 0x7f7f8000)
$3 = 32768

The size in my case is small but I didn't get that stack overflow error.

Re: High memory utilisation

Vee_1 — Tue, 17 Jun 2008 14:11:47 GMT

hello Dennis,
i dont find any core files in my system.But my system is now normal.

Re: High memory utilisation

Gaël LEPETIT — Wed, 25 Jun 2008 13:20:07 GMT

Hi,

Same problem on our HP-UX 11i v2, Dec 07 servers. M_TEMP arena eats up to 6 gb of memory.
After a diagnostic with HP support, they found a bug and a fix (PHKL_38436) will be released by the end of the month.