topic Re: Memory Usage by process - Always the BIG ? in Operating System - HP-UX

Memory Usage by process - Always the BIG ?

shab1207 — Tue, 04 May 2010 14:25:09 GMT

Hi,

I have been looking through innumerous threads regarding ddetermining the correct memory utilization of the system and the proces occuping max memory for HPUX systems
It is depressing to see multiple replies but none really conclusive.

Here upon, I request all the genius to put in their expertise and make this a nobel thread.
I think using commands like
UNIX95= ps -ef -o vsz,sz.. | sort -nrk1/2 etc or swapinfo -tam are all useless since they never provide a consolidated information..

Is there any genuine way where we can extract the % of memory occupied and process occuping max memory (in %) and ofcouse validate the results by some simple calculation.

A generic script for HPUX platforms should be FINE

Re: Memory Usage by process - Always the BIG ?

R.O. — Tue, 04 May 2010 14:42:06 GMT

Hi,

Yo can use "glance" or the script "kmeminfo".

Regards,

Re: Memory Usage by process - Always the BIG ?

shab1207 — Tue, 04 May 2010 14:52:54 GMT

I think there would be infinite such threads and i am sure i have explored most of them.

looking for some script which may really provide a consolidate memory usage status and process taking max memory

Re: Memory Usage by process - Always the BIG ?

Steven E. Protter — Tue, 04 May 2010 15:19:11 GMT

Shalom,

http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=51050

This link or the two sysadmin scripts that came first which are linked internally is a wealth of scripts that do what you need.

What you are asking for has done, or can be done with one of the scripts in the thread, with minimal modification.

SEP

Re: Memory Usage by process - Always the BIG ?

Pulse001 — Tue, 04 May 2010 15:21:17 GMT

Hi shab1207,

have you tried kmeminfo ?

i am sure kmeminfo will serve your purpose. It will give you full listing of all individual processes alongwith pid and owner.I found it more than sufficient.

Re: Memory Usage by process - Always the BIG ?

shab1207 — Tue, 04 May 2010 15:26:40 GMT

Thanks...

Can you please share your kmeminfo script.
I am always confused when it comes to expressing in terms of %.
so if you could help me with the calculation as well
Thanks in advance

Re: Memory Usage by process - Always the BIG ?

Pulse001 — Wed, 05 May 2010 03:43:51 GMT

hi shab1207,

kmeminfo will show your total memory, free memory , memory consumed by user processes and memory consumed by system / kernel, in absolute value as well as percentage.

A sample kmeminfo output will look like as below:-

tool: kmeminfo 7.10 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.23 64bit IA64 on host
core: /dev/kmem live
link: Mon Mar 15 08:34:12 GMT 2010
boot: Fri Mar 19 05:07:26 2010
time: Wed Mar 31 07:26:56 2010
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 3144280 12.0g 100%
Free memory = 95835 374.4m 3%
User processes = 2200622 8.4g 70% details with -user
System = 753301 2.9g 24%
Kernel = 441570 1.7g 14% kernel text and data
Dynamic Arenas = 158320 618.4m 5% details with -arena
vx_global_pool = 34657 135.4m 1%
vx_inode_cache = 23437 91.6m 1%
spinlock = 22553 88.1m 1%
VFD_BT_NODE = 14870 58.1m 0%
vm_pfn2v_arena = 12340 48.2m 0%
Other arenas = 50463 197.1m 2% details with -arena
Super page pool = 22983 89.8m 1% details with -kas
DMA32 free pool = 65464 255.7m 2%
Static Tables = 148791 581.2m 5% details with -static
pfdat = 73694 287.9m 2%
nbuf = 34544 134.9m 1% bufcache headers
vhpt = 16384 64.0m 1%
bufhash = 8192 32.0m 0% bufcache hash headers
text = 7414 29.0m 0% vmunix text section
Other tables = 8562 33.4m 0% details with -static
Buffer cache = 311731 1.2g 10% details with -bufcache
UFC file mrg = 0 0.0 0%
UFC meta mrg = 0 0.0 0%

If you do a kmeminfo -user, it will show you memory consumption of each process (in mb / gb) alongwith pid and ppid.

----------------------------------------------------------------------
Summary of processes memory usage:

List sorted by physical size, in pages/bytes:

virtual physical swap
pid ppid pages / bytes pages / bytes pages / bytes command
12681 12665 884774 3.4g 336345 1.3g 343144 1.3g java_q4p
29064 29048 383931 1.5g 296866 1.1g 299612 1.1g java
13127 13111 274445 1.0g 209538 818.5m 212907 831.7m java
18271 1 349614 1.3g 171990 671.8m 174315 680.9m java
27912 1 96174 375.7m 70163 274.1m 71268 278.4m cleard
16852 1 335227 1.3g 67946 265.4m 70956 277.2m java
18266 1 334350 1.3g 67019 261.8m 70076 273.7m java
17470 1 333179 1.3g 66026 257.9m 68897 269.1m java
28104 1 333179 1.3g 65962 257.7m 68897 269.1m java
28062 1 336251 1.3g 60793 237.5m 63488 248.0m java
27982 1 89194 348.4m 58767 229.6m 64253 251.0m cleard
27935 27883 113894 444.9m 48737 190.4m 106043 414.2m WSH
27888 27883 83462 326.0m 47885 187.1m 75458 294.8m WSH
23972 1 92746 362.3m 43663 170.6m 67822 264.9m cleard
27980 1 63370 247.5m 37067 144.8m 38299 149.6m cleard
27913 1 60874 237.8m 34283 133.9m 35790 139.8m cleard
27981 1 64555 252.2m 34171 133.5m 39489 154.3m cleard
27911 1 59114 230.9m 32659 127.6m 34021 132.9m cleard
27932 27883 60774 237.4m 24281 94.8m 52656 205.7m WSH
27933 27883 57030 222.8m 21513 84.0m 48893 191.0m WSH
27931 27883 61254 239.3m 20909 81.7m 53138 207.6m WSH
27901 27883 56038 218.9m 20457 79.9m 47896 187.1m WSH
27934 27883 46246 180.6m 20021 78.2m 38055 148.7m WSH
1060 1 28781 112.4m 13654 53.3m 15334 59.9m vxsvc
27963 1 35434 138.4m 8923 34.9m 10223 39.9m cleard
27955 1 33898 132.4m 7571 29.6m 8679 33.9m cleard
27961 1 32266 126.0m 6107 23.9m 7039 27.5m cleard
27962 1 32491 126.9m 6027 23.5m 7264 28.4m cleard
27964 1 32490 126.9m 5935 23.2m 7264 28.4m cleard
27968 1 31786 124.2m 5772 22.5m 6557 25.6m cleard
27970 1 31370 122.5m 5376 21.0m 6139 24.0m cleard
27969 1 31370 122.5m 5372 21.0m 6139 24.0m cleard
27967 1 31786 124.2m 5292 20.7m 6557 25.6m cleard
23909 1 31242 122.0m 5039 19.7m 6010 23.5m cleard
23929 1 31242 122.0m 5039 19.7m 6010 23.5m cleard
27956 1 31114 121.5m 4991 19.5m 5881 23.0m cleard
27984 1 30731 120.0m 4515 17.6m 5495 21.5m cleard
27983 1 30538 119.3m 4319 16.9m 5302 20.7m cleard
27928 1 30282 118.3m 4299 16.8m 5045 19.7m cleard
27899 1 30474 119.0m 4267 16.7m 5238 20.5m cleard
27927 1 30218 118.0m 4227 16.5m 4980 19.5m cleard
27926 1 30186 117.9m 4187 16.4m 4948 19.3m cleard
27941 1 29994 117.2m 4015 15.7m 4755 18.6m cleard
1019 1 30122 117.7m 3907 15.3m 4884 19.1m cleard
27917 1 29866 116.7m 3875 15.1m 4627 18.1m cleard
27918 1 29866 116.7m 3867 15.1m 4627 18.1m cleard
27925 1 29866 116.7m 3863 15.1m 4627 18.1m cleard
27919 1 29834 116.5m 3851 15.0m 4594 17.9m cleard
27898 1 29834 116.5m 3847 15.0m 4594 17.9m cleard
27946 1 29834 116.5m 3831 15.0m 4594 17.9m cleard
27951 1 29802 116.4m 3827 14.9m 4562 17.8m cleard
27945 1 29802 116.4m 3823 14.9m 4562 17.8m cleard

If you have a support contract with HP, you can ask for the kmeminfo script, they will provide you, it is free but not supported by HP.

Re: Memory Usage by process - Always the BIG ?

Pulse001 — Wed, 05 May 2010 03:51:04 GMT

In fact kmeminfo was very helpful for me in doing an investigation for probable memory leak in my server. You can check the memoru consumption of a single process using kmeminfo as below:-

#kmeminfo

We had a process which was crashing and throwing "out of memory" error everytime load was put on it.The application vendor as usual was pointing finger at the OS. When i checked the memory footprint of that process using kmeminfo, i found it started with 1gb and subsequently increased to 4 gb ,after which the process crashed. I presented the memory consumption of that process to my application vendor and they had to agree that there was indeed a memory leak in their process and did the resolution at their end.

Re: Memory Usage by process - Always the BIG ?

shab1207 — Thu, 06 May 2010 14:05:23 GMT

Still dosent answer my query.
none of the scripts still address the query.

Expressing process using highest memory in terms of percent.
seems everyone is running away from the fact since it is truely difficult to get this thing calculated due to swap,core,physical memory etc.. and this makes it truely impossible.

Re: Memory Usage by process - Always the BIG ?

Pulse001 — Thu, 06 May 2010 15:36:51 GMT

hi shab,

i think my response was clear enough and shows memory usage per process basis.

calculating the percent when you have absolute values ( e.g 3.4gb out of 12gb ) is not rocket science.

Re: Memory Usage by process - Always the BIG ?

shab1207 — Thu, 06 May 2010 15:40:57 GMT

Hi pulse,

How would the % be calculated since a process will occupy physical,virtual and swap memory.
Now while calculating the % should i do a

(occupied physical + occupied virtual + occupied swap ) / (total physical + total virtual + total swap ) * 100
I dont these calculations are as simple as you indicate

Re: Memory Usage by process - Always the BIG ?

Don Morris_1 — Thu, 06 May 2010 16:20:44 GMT

Well, your calculation would be less than simple -- but that's also because it is less than obvious that it would have any real meaning. A process consuming a lot of virtual memory may not be consuming any swap or any physical memory at all (mmap() a very large file without touching the mapping and you get just this). Or it could be consuming a lot of virtual+swap but no physical (large swap-backed object) or it could be consuming a lot of physical and virtual memory with no swap (touching that mapping mentioned before), etc.

So I don't get at all what you're asking. There's no real concept of a process using a percentage of "total system resources", each resource type has some relation to the other, but a tenuous one in most cases and should be analyzed separately.

Here's something which might get you going. If you really, really want to add up the totals for some reason, the framework is there (add a field for total_global or something and each proc_global would += (proc_phys + proc_virt + proc_swap + proc_mlock), keep a top_global -- and then you can do the comparison to (sbuf.physical_memory + dbuf.psd_vm + vbuf.psv_swapspc_max + vbuf.psv_swapmem_max + vbuf.psv_swapmem_max) [fudging a little in just calling Memory Swap maximum the maximum for Lockable memory -- it is actually a little lower, depending on OS version and tunables, etc.] I don't see the point -- but there you go.

Sample output (and I don't write UIs, so yes, this is probably ugly):

Memory Stat total used avail %used
physical 11591.2 2502.7 9088.6 22%
active virtual 712.6 527.4 185.2 74%
active real 385.9 274.1 111.8 71%
memory swap 11024.8 1528.6 9496.2 14%
device swap 8192.0 405.1 7786.9 5%
Activations: 0 total, 0 rate. Deactivations: 0 total, 0 rate.
Reclaims from Swap: 0 total (Up 0), 0 rate.
Top Physical PID: [cimprovagt] 1534 Phys: 100 Mb (0.8687%) of RAM.
Top Virtual PID: [cimprovagt] 1534 Virt: 436 Mb (61.2214%) of Total Virt.
Top Swap PID: [cimprovagt] 1534 Swap: 79 Mb (0.4159%) of Total Swap.
Top Mlock PID: [midaemon] 1913 Mlock: 32 Mb (0.2967%) of Memory Swap.

The numbers don't line up 100% with kmeminfo -- but pstat tries to account for shared object references more than kmeminfo does, so that isn't surprising.

Also, the Total Virtual resource is really unknown. Since the virtual space reported includes files, we'd have to know the maximum space of any and all filesystems ever to be attached, etc. The kernel doesn't bother predicting that and as such, doesn't report it (we'd need max file space + max swap + memory swap to equal the absolute total Virtual space). So that "Total Virt" is really the current total virtual load of User space.

Re: Memory Usage by process - Always the BIG ?

shab1207 — Thu, 06 May 2010 16:30:07 GMT

Hi Don,

I think I am getting a great sign of hope now.

Even I think i am confused but I will explain -
A simple summarization of the % of the HPUX memory being consumed and the process occuping the highest % of memory

Need a perl or shell script :(

Re: Memory Usage by process - Always the BIG ?

Bill Hassell — Thu, 06 May 2010 17:45:34 GMT

> looking for some script which may really provide a consolidate memory usage status and process taking max memory...
> Expressing process using highest memory in terms of percent.

This really won't provide any useful information. A memory leak is nothing but a programming artifact...it may be intentional (the program needs more memory) or unintentional and therefore a leak. But knowing what percentage of RAM a single process is using is like trying to fix a filesystem by looking for big files. All the big files (or processes) are running as designed, and the real problem is with dozens of smaller files (or processes) that are not supposed to be growing.

So let's assume that the program using the largest amount of local or heap memory is Java and it has a resident set size of 2000MB. Do you know how to rewrite the Java code to not use so much memory? If not, then all you can do is apply the latest patches and buy more RAM if necessary.

Or perhaps it is shared memory (which is not part of local memory for any process, but Oracle requested 5000MB. So it is using a lot of memory but you must talk to the DBA about this usage. You may be informed that to reduce it to 1000MB means that all Oracle transactions will take 500% longer to complete.

The amount of memory used by each process is a function of how it was written. While you can impose a memory limit for each process, the majority of programs will simply crash with an errno 12 message (not enough core). So you will have solved the problem for programs that take more than the allowed space but no one gets any work done.