1829692 Members
7516 Online
109992 Solutions
New Discussion

memory usage in HP-UX v3

 
SOLVED
Go to solution
Trojan36
Frequent Advisor

memory usage in HP-UX v3

On a BL870c server, is it possible to tune 11iv3 kernel to use certain amount of memory (say 10GB on a system with 128GB of memory)?

Thx in advance
12 REPLIES 12
Don Morris_1
Honored Contributor

Re: memory usage in HP-UX v3

Only if you extend the meaning of 'tune' to include kernel settings and the workload (including network packets) you impose upon the kernel.

This assumes you mean that you want the OS as a whole to see the whole 128Gb but the kernel consumption of physical memory to be limited to only 10Gb (if you meant the whole OS seeing only 10Gb even though there are 128Gb visible at the machine level, that would be `hpux vmunix -M 10240` at boot).

A few comments on this in case you feel strongly about it:

1) As a raw idea, the idea is not impossible (though not with the number you provide, see next comment). However -- the impact of getting this wrong is pretty severe. Kernel memory exhaustion results in one of two scenarios in almost all cases: An explicit panic (someone asked for memory with a "No Wait" state, didn't get it even though they think they should -- and panic'd the kernel... or clustering/monitoring software such as ServiceGuard can bring down the box due to low response times [packets can be dropped a _lot_, etc.]) or what is effectively a system hang (usually un-pingable). Neither is desirable in most contexts.

I should clearly state that by "not impossible" I mean that mechanisms exist that could be leveraged to do this sort of thing -- but there are absolutely no supported interfaces or means to perform such a modification/restriction without at the least a new 11.31 VM patch. So this would require a formal Enhancement Request through your support channel (and given the aforementioned impact, I really wouldn't have high expectations in this space).

2) From a raw numbers status -- v3 tends to need/use around 8 to 9% right off the top [5% is mandatory if you don't change the base_pagesize tunable... you could probably squeeze down a good bit if you lower it, but as I understand it, there have been application issues with larger page sizes -- so caveat emptor]. 10Gb out of 128Gb is 7.8125%, so I strongly suspect that just booting the kernel up would approach this limit on a 128Gb box. Running any sort of workload implies kernel metadata (to track processes, threads, address space, inflight I/Os, etc.) so you are going to push above that very quickly.
Trojan36
Frequent Advisor

Re: memory usage in HP-UX v3

Thanks for your detailed answer. So there are no easy ways to limit kernel memory usage to a set number without risking system panic. How about trying to reduce kernel memory percentage? We can also tune File Cache to be lower. Do you see anything else we can do to lower system (kernel) memory usage?

Thanks again.
Don Morris_1
Honored Contributor

Re: memory usage in HP-UX v3

My stock answer for v3 is to always look at vx_ninode since it directly drives several layers of caching due to file cache activity (the inodes are cached, VM metadata related to the inodes are cached, the large pages of kernel dynamic memory both of these are built out of get cached, etc.).

The secondary answer (as alluded to in (2) above) would be if you can raise base_pagesize, consider doing so as it reduces the per-physical frame cost of kernel management. This should be done in a testing environment to ensure that your applications work properly before rolling this into a production environment.

If the problem is more in HPVM guests, you can consider lowering the dma32_pool_size tunable (as this memory is pre-reserved as System consumed). Non-HPVM guest environments on the BL870c should not see any difference as the tunable should be effectively ignored.

With the tunable work that went into v3, I don't recall anything else that pre-allocates aggressively (unlike some of the SysV tunables in prior releases, I don't believe the v3 ones chew a bunch of memory if you set them too high) -- so that leaves mainly reactive allocations which stem from the workload itself. That requires tuning the workload, not the kernel.

This would be more effective if you could describe the problem you're seeing, by the way -- is it really that the kernel load is too high and not releasing memory to user space when the user load grows and free memory drops (to around 1%)? Or are you really just trying to keep some X Gb of RAM around "just in case" (in which case, it should as always be noted that HP-UX caches aggressively for performance and does not favor leaving RAM around as "free" if it can find other ways to use it)? What percentage are you at now?
Steven E. Protter
Exalted Contributor

Re: memory usage in HP-UX v3

Shalom,

You could use HPVM and create a vitural machine with 10 GB of RAM in it. That would use precisely the amount of memory you wish.

The answer is, so long as you are not too exact, you can do this. A 10 GB kernel is huge and wasteful. If your goal is to use less than 10 GB of RAM int he kernel, this should be very much possible.

Take a look at glance Memory report and you can get an idea how much kernel is currently using.

I'd start with kctune output to identify buffer pool.

I assume the goal is to use as little memory in kernel as possible without impeding performance.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Trojan36
Frequent Advisor

Re: memory usage in HP-UX v3

Hi Don,

Here is the output from the kmeminfo. Any way we can lower system (kernal) memory usage will be great. Thx

root@secstn01:/software/hpux/11iv3/kmeminfo/kmeminfo
tool: kmeminfo 10.00 - libp4 9.383 - libhpux 1.288 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "secstn01"
core: /dev/kmem live
link: Thu Jan 21 13:48:50 EST 2010
boot: Thu Feb 4 19:02:15 2010
time: Tue Feb 23 13:10:37 2010
nbpg: 4096 bytes


----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 33546679 128.0g 100%
Free memory = 4412685 16.8g 13%
User processes = 21588954 82.4g 64% details with -user
Detached SHMEM = 2 8.0k 0% details with -shmem
System = 7156089 27.3g 21%
Kernel = 4153835 15.8g 12% kernel text and data
Dynamic Arenas = 1853171 7.1g 6% details with -arena
BTREE_NODE_OLA_ = 776888 3.0g 2%
btree_chunk_are = 396323 1.5g 1%
vm_pfn2v_arena = 131476 513.6m 0%
vx_global_kmcac = 70104 273.8m 0%
FCACHE_ARENA = 55774 217.9m 0%
Other arenas = 422606 1.6g 1% details with -arena
Super page pool = 1531 6.0m 0% details with -kas
Emergency pool = 66106 258.2m 0% system critical reserve
UAREA's = 11216 43.8m 0%
Overflow pte's = 262378 1.0g 1%
Static Tables = 1919337 7.3g 6% details with -static
pfdat = 1638021 6.2g 5%
vhpt = 262144 1.0g 1%
text = 9006 35.2m 0% vmunix text section
bss = 5540 21.6m 0% vmunix bss section
inode = 2086 8.1m 0%
Other tables = 2537 9.9m 0% details with -static
File Cache = 3002254 11.5g 9% details with -ufc
Don Morris_1
Honored Contributor

Re: memory usage in HP-UX v3

Frankly, that looks pretty good to me already.

Filecache is at 9%, yes -- but you already mentioned that you tuned it and there's obviously a performance tradeoff to be made with it.

The file related stuff looks pretty small, and the Super page pool does not show signs of fragmentation.

Your biggest dynamic clients are both btree related -- which implies you are touching a lot of virtual address space (i.e. mapping lots of little files [don't think so, since your fcache/file system stuff would be higher] or have lots of little processes [Uareas would probably be higher] or have some really large virtual objects (say very large file mappings, anonymous mmaps or SysV objects) which have sparse references.

What's the results of the following:

`kmeminfo -arena BTREE_NODE_OLA_`

`kmeminfo -arena btree_chunk_are`

`kmeminfo -user`

?

Keep in mind, though -- even getting btree usage down, you've got 9% in the file cache, a fixed 6% for per-page (PFDAT and overflow ptes) metadata, 1% for the translation table (vhpt) -- of which only the file cache can be adjusted. That's 16% of your 21% right there... and I don't see much usage below the btree arenas so you're likely stuck with 3 to 4% no matter what -- which means what we're really discussing is if you can get the kernel down 1% and _maybe_ 2% if possible.

This is where base_pagesize would make a much larger difference. Go to 8 instead of 4 and you get at least 2.5% back up front (quite likely more since the btrees would be covering twice as much with each entry, etc.). Go to 16 (usually the preferred setting) and you should get back at least 3.75% (more likely at least 5 to 6%, but I prefer to low-ball estimates).
Michael Steele_2
Honored Contributor

Re: memory usage in HP-UX v3

HI

If you're looking for a min max constant to put on memory or cpu or other 11.31 resources then refer to PRM, an add on utility for just what you're talking about.

http://h20338.www2.hp.com/enterprise/w1/en/os/hpux11i-prm-overview.html

http://h20341.www2.hp.com/integrity/downloads/Integrity_BL870_SPECjApp_092208.pdf
Support Fatherhood - Stop Family Law
Don Morris_1
Honored Contributor

Re: memory usage in HP-UX v3

PRM on memory affects User space, not kernel space.
Steven E. Protter
Exalted Contributor

Re: memory usage in HP-UX v3

Shalom again,

There is no easy way to do this.

kmeminfo is a good place to identify kernel pigs.

Then you can reduce those parameters and thereby reduce kernel memory usage.

Note though, if for example the system runs Oracle, there are kernel parameters that must meet minimum levels or Oracle will not run or will run poorly.

So you need to check the documentation on major applications and make sure when all is complete, the kernel meets the needs of applications.

Low kernel memory usage is a good goal, but must be seen in the context of having a smooth running system that performs well for the applications. Some applications work better with liberal kernel settings. Some are not impacted.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Trojan36
Frequent Advisor

Re: memory usage in HP-UX v3

Don,

Thanks again for the detailed answer. Our application uses 16k IO size so we can possibly increase base_pagesize to match it. Currently our filecache is set as follows.

filecache_max 13069631078 10% Imm (auto disabled)
filecache_min 6534815539 5% Imm (auto disabled)

Perhaps we can continue to adjust them.

Please see below output.

root@secstn01:/software/hpux/11iv3/kmeminfo/kmeminfo -arena BTREE_NODE_OLA_
tool: kmeminfo 10.00 - libp4 9.383 - libhpux 1.288 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "secstn01"
core: /dev/kmem live
link: Thu Jan 21 13:48:50 EST 2010
boot: Thu Feb 4 19:02:15 2010
time: Tue Feb 23 17:44:26 2010
nbpg: 4096 bytes


Fixed arena "BTREE_NODE_OLA_" owns 776888 pages (3.0gb):
kmem_arena_t 0xe00000013f993a00
Attributes: KMT_DEFAULT|KMT_NO_LGPG|KMT_KVADDR_CACHE|KMT_OLA_SAFE KT_DEFAULT|KT_NUMA_ILV_DEFAULT|KT_NUMA_NO_DEFAULT|KT_NUMA_TRANSITORY_OK KAS_ALIVE
Regular (Interleaved First) free lists account for 0 pages (0.0b):
Regular (Closest First) free lists account for 776888 pages (3.0gb):
Free objects represent 2.5gb (86%) of all memory in this arena.
Per index summary (all cpu's):
idx objsz pages bytes % nobjs used free %
0 4096 776888 3.0g 100 776888 110853 666035 86
Per cpu free list details:
idx cpu kmem_flist_hdr_t pages bytes % nobjs used free %
0 0 0xe00000013f996e00 109821 429.0m 14 109821 1096 108725 99
0 2 0xe0000001bd92a280 110826 432.9m 14 110826 2237 108589 98
0 4 0xe0000001bd9a7700 112183 438.2m 14 112183 12170 100013 89
0 6 0xe0000001be46ea80 66496 259.8m 9 66496 6602 59894 90
0 8 0xe0000001be52d700 110489 431.6m 14 110489 69532 40957 37
0 10 0xe0000001be4a8080 107430 419.6m 14 107430 2334 105096 98
0 12 0xe0000001bea4d500 112880 440.9m 15 112880 14924 97956 87
0 14 0xe0000001bea24480 46763 182.7m 6 46763 1958 44805 96
root@secstn01:
root@secstn01:/software/hpux/11iv3/kmeminfo/kmeminfo -arena btree_chunk_are
tool: kmeminfo 10.00 - libp4 9.383 - libhpux 1.288 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "secstn01"
core: /dev/kmem live
link: Thu Jan 21 13:48:50 EST 2010
boot: Thu Feb 4 19:02:15 2010
time: Tue Feb 23 17:45:47 2010
nbpg: 4096 bytes


Fixed arena "btree_chunk_are" owns 396323 pages (1.5gb):
kmem_arena_t 0xe00000013f993b80
Attributes: KMT_DEFAULT KT_DEFAULT|KT_NUMA_ILV_DEFAULT|KT_NUMA_NO_DEFAULT|KT_NUMA_TRANSITORY_OK KAS_ALIVE
Regular (Interleaved First) free lists account for 0 pages (0.0b):
Regular (Closest First) free lists account for 396323 pages (1.5gb):
Free objects represent 1.5gb (97%) of all memory in this arena.
Per index summary (all cpu's):
idx objsz pages bytes % nobjs used free %
0 512 396323 1.5g 100 2774261 86291 2687970 97
Per cpu free list details:
idx cpu kmem_flist_hdr_t pages bytes % nobjs used free %
0 0 0xe000000150184080 48204 188.3m 12 337428 12865 324563 96
0 2 0xe0000001bd914a00 46238 180.6m 12 323666 7546 316120 98
0 4 0xe0000001bd942f00 47358 185.0m 12 331506 9853 321653 97
0 6 0xe0000001be46e080 52069 203.4m 13 364483 14290 350193 96
0 8 0xe0000001be51ec80 54334 212.2m 14 380338 11683 368655 97
0 10 0xe0000001be5f0500 55949 218.6m 14 391643 8797 382846 98
0 12 0xe0000001bea5f380 44805 175.0m 11 313635 10011 303624 97
0 14 0xe000000150800680 47366 185.0m 12 331562 11246 320316 97
root@secstn01:
Don Morris_1
Honored Contributor
Solution

Re: memory usage in HP-UX v3

Interesting -- that indicates a spiky workload cycling around your system, creating and destroying a large set of btrees (both file and non-file). [The per-cpu freelists in Arena are designed for rapid alloc/free time, so they try not to interfere with each other. As such, if a cpu freelist is empty, even if another in the same Arena has memory - the empty list goes to the next layer down. The garbage collector eventually balances this stuff out, but you have to have some memory pressure for that to run -- and you don't have pressure].

So -- from your output: You could get 4% back by setting filecache_max to the value of filecache_min instead of letting it float between 5 to 10% (currently at 9). Since you will get that memory back anyway if you're under pressure and you have plenty, I probably wouldn't do it... but it is your system and you're welcome to try. (It is dynamic, and you can always raise it back if you want). You can get more by setting both lower than 5% (down to 1%) -- but of course you're going to incur more I/Os by losing the caching effects.

The btree arenas will have one of two effects -- either this itinerant workload still occurs, in which case since 7 (of 8?) freelists have plenty of memory you shouldn't see much increase as the system will just recycle the objects already there. Or if it doesn't, these objects will (eventually) be returned to the system -- but with no pressure, we're talking a timescale of days.

On the positive side, the BTREE_NODE_OLA arena has a couple important characteristics here -- first, it allocates objects equal in size to a system page (which simplifies garbage collection) and second, it needs to be able to swap these pages out to disk under pressure -- so the pages do not come from the Super Page Pool layer [and this arena is therefore not dependent on that layer being unfragmented to return memory to the system]. If memory pressure occurs -- this arena is going to contribute significantly to a garbage collection pass, and the memory can go right back to the system (each free object is returnable by itself).

As far as forcing that out -- I'd want to check something before I discuss that, so I'll follow up tomorrow morning. But realistically, this memory should come back for user space allocations readily, so I wouldn't worry much about it.
Don Morris_1
Honored Contributor

Re: memory usage in HP-UX v3

Ok -- on that followup, if you happen to have PHKL_38980 and its dependencies (especially PHCO_39121) you have the Locality-Optimized Resource Alignment enhancement. [If not, then all you can really do is wait for the system to harvest the memory for you].

One of the things the loratune command provided in those patches does is to endeavour to reclaim memory from caches to the system (with the presumed reason that you want to then use it in a locality-aware fashion, but the reclamation is locality agnostic as the 'wrong' locality might have been used because the 'right' locality for the caching layer was already allocated elsewhere -- hence the cache flushing isn't really targeting the user desired locality). If you invoke it with no arguments, you'll take a cpu spike in the short term (since it will go do all this GC behavior) but you can flush that memory back to the system on the Arena (possibly the SPP) side.