- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- memory usage in HP-UX v3
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 06:08 AM
02-23-2010 06:08 AM
Thx in advance
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 08:06 AM
02-23-2010 08:06 AM
Re: memory usage in HP-UX v3
This assumes you mean that you want the OS as a whole to see the whole 128Gb but the kernel consumption of physical memory to be limited to only 10Gb (if you meant the whole OS seeing only 10Gb even though there are 128Gb visible at the machine level, that would be `hpux vmunix -M 10240` at boot).
A few comments on this in case you feel strongly about it:
1) As a raw idea, the idea is not impossible (though not with the number you provide, see next comment). However -- the impact of getting this wrong is pretty severe. Kernel memory exhaustion results in one of two scenarios in almost all cases: An explicit panic (someone asked for memory with a "No Wait" state, didn't get it even though they think they should -- and panic'd the kernel... or clustering/monitoring software such as ServiceGuard can bring down the box due to low response times [packets can be dropped a _lot_, etc.]) or what is effectively a system hang (usually un-pingable). Neither is desirable in most contexts.
I should clearly state that by "not impossible" I mean that mechanisms exist that could be leveraged to do this sort of thing -- but there are absolutely no supported interfaces or means to perform such a modification/restriction without at the least a new 11.31 VM patch. So this would require a formal Enhancement Request through your support channel (and given the aforementioned impact, I really wouldn't have high expectations in this space).
2) From a raw numbers status -- v3 tends to need/use around 8 to 9% right off the top [5% is mandatory if you don't change the base_pagesize tunable... you could probably squeeze down a good bit if you lower it, but as I understand it, there have been application issues with larger page sizes -- so caveat emptor]. 10Gb out of 128Gb is 7.8125%, so I strongly suspect that just booting the kernel up would approach this limit on a 128Gb box. Running any sort of workload implies kernel metadata (to track processes, threads, address space, inflight I/Os, etc.) so you are going to push above that very quickly.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 10:32 AM
02-23-2010 10:32 AM
Re: memory usage in HP-UX v3
Thanks again.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 12:36 PM
02-23-2010 12:36 PM
Re: memory usage in HP-UX v3
The secondary answer (as alluded to in (2) above) would be if you can raise base_pagesize, consider doing so as it reduces the per-physical frame cost of kernel management. This should be done in a testing environment to ensure that your applications work properly before rolling this into a production environment.
If the problem is more in HPVM guests, you can consider lowering the dma32_pool_size tunable (as this memory is pre-reserved as System consumed). Non-HPVM guest environments on the BL870c should not see any difference as the tunable should be effectively ignored.
With the tunable work that went into v3, I don't recall anything else that pre-allocates aggressively (unlike some of the SysV tunables in prior releases, I don't believe the v3 ones chew a bunch of memory if you set them too high) -- so that leaves mainly reactive allocations which stem from the workload itself. That requires tuning the workload, not the kernel.
This would be more effective if you could describe the problem you're seeing, by the way -- is it really that the kernel load is too high and not releasing memory to user space when the user load grows and free memory drops (to around 1%)? Or are you really just trying to keep some X Gb of RAM around "just in case" (in which case, it should as always be noted that HP-UX caches aggressively for performance and does not favor leaving RAM around as "free" if it can find other ways to use it)? What percentage are you at now?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 01:09 PM
02-23-2010 01:09 PM
Re: memory usage in HP-UX v3
You could use HPVM and create a vitural machine with 10 GB of RAM in it. That would use precisely the amount of memory you wish.
The answer is, so long as you are not too exact, you can do this. A 10 GB kernel is huge and wasteful. If your goal is to use less than 10 GB of RAM int he kernel, this should be very much possible.
Take a look at glance Memory report and you can get an idea how much kernel is currently using.
I'd start with kctune output to identify buffer pool.
I assume the goal is to use as little memory in kernel as possible without impeding performance.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 01:13 PM
02-23-2010 01:13 PM
Re: memory usage in HP-UX v3
Here is the output from the kmeminfo. Any way we can lower system (kernal) memory usage will be great. Thx
root@secstn01:/software/hpux/11iv3/kmeminfo/kmeminfo
tool: kmeminfo 10.00 - libp4 9.383 - libhpux 1.288 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "secstn01"
core: /dev/kmem live
link: Thu Jan 21 13:48:50 EST 2010
boot: Thu Feb 4 19:02:15 2010
time: Tue Feb 23 13:10:37 2010
nbpg: 4096 bytes
----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):
Physical memory = 33546679 128.0g 100%
Free memory = 4412685 16.8g 13%
User processes = 21588954 82.4g 64% details with -user
Detached SHMEM = 2 8.0k 0% details with -shmem
System = 7156089 27.3g 21%
Kernel = 4153835 15.8g 12% kernel text and data
Dynamic Arenas = 1853171 7.1g 6% details with -arena
BTREE_NODE_OLA_ = 776888 3.0g 2%
btree_chunk_are = 396323 1.5g 1%
vm_pfn2v_arena = 131476 513.6m 0%
vx_global_kmcac = 70104 273.8m 0%
FCACHE_ARENA = 55774 217.9m 0%
Other arenas = 422606 1.6g 1% details with -arena
Super page pool = 1531 6.0m 0% details with -kas
Emergency pool = 66106 258.2m 0% system critical reserve
UAREA's = 11216 43.8m 0%
Overflow pte's = 262378 1.0g 1%
Static Tables = 1919337 7.3g 6% details with -static
pfdat = 1638021 6.2g 5%
vhpt = 262144 1.0g 1%
text = 9006 35.2m 0% vmunix text section
bss = 5540 21.6m 0% vmunix bss section
inode = 2086 8.1m 0%
Other tables = 2537 9.9m 0% details with -static
File Cache = 3002254 11.5g 9% details with -ufc
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 01:37 PM
02-23-2010 01:37 PM
Re: memory usage in HP-UX v3
Filecache is at 9%, yes -- but you already mentioned that you tuned it and there's obviously a performance tradeoff to be made with it.
The file related stuff looks pretty small, and the Super page pool does not show signs of fragmentation.
Your biggest dynamic clients are both btree related -- which implies you are touching a lot of virtual address space (i.e. mapping lots of little files [don't think so, since your fcache/file system stuff would be higher] or have lots of little processes [Uareas would probably be higher] or have some really large virtual objects (say very large file mappings, anonymous mmaps or SysV objects) which have sparse references.
What's the results of the following:
`kmeminfo -arena BTREE_NODE_OLA_`
`kmeminfo -arena btree_chunk_are`
`kmeminfo -user`
?
Keep in mind, though -- even getting btree usage down, you've got 9% in the file cache, a fixed 6% for per-page (PFDAT and overflow ptes) metadata, 1% for the translation table (vhpt) -- of which only the file cache can be adjusted. That's 16% of your 21% right there... and I don't see much usage below the btree arenas so you're likely stuck with 3 to 4% no matter what -- which means what we're really discussing is if you can get the kernel down 1% and _maybe_ 2% if possible.
This is where base_pagesize would make a much larger difference. Go to 8 instead of 4 and you get at least 2.5% back up front (quite likely more since the btrees would be covering twice as much with each entry, etc.). Go to 16 (usually the preferred setting) and you should get back at least 3.75% (more likely at least 5 to 6%, but I prefer to low-ball estimates).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 01:53 PM
02-23-2010 01:53 PM
Re: memory usage in HP-UX v3
If you're looking for a min max constant to put on memory or cpu or other 11.31 resources then refer to PRM, an add on utility for just what you're talking about.
http://h20338.www2.hp.com/enterprise/w1/en/os/hpux11i-prm-overview.html
http://h20341.www2.hp.com/integrity/downloads/Integrity_BL870_SPECjApp_092208.pdf
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 02:07 PM
02-23-2010 02:07 PM
Re: memory usage in HP-UX v3
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 03:02 PM
02-23-2010 03:02 PM
Re: memory usage in HP-UX v3
There is no easy way to do this.
kmeminfo is a good place to identify kernel pigs.
Then you can reduce those parameters and thereby reduce kernel memory usage.
Note though, if for example the system runs Oracle, there are kernel parameters that must meet minimum levels or Oracle will not run or will run poorly.
So you need to check the documentation on major applications and make sure when all is complete, the kernel meets the needs of applications.
Low kernel memory usage is a good goal, but must be seen in the context of having a smooth running system that performs well for the applications. Some applications work better with liberal kernel settings. Some are not impacted.
SEP
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 03:22 PM
02-23-2010 03:22 PM
Re: memory usage in HP-UX v3
Thanks again for the detailed answer. Our application uses 16k IO size so we can possibly increase base_pagesize to match it. Currently our filecache is set as follows.
filecache_max 13069631078 10% Imm (auto disabled)
filecache_min 6534815539 5% Imm (auto disabled)
Perhaps we can continue to adjust them.
Please see below output.
root@secstn01:/software/hpux/11iv3/kmeminfo/kmeminfo -arena BTREE_NODE_OLA_
tool: kmeminfo 10.00 - libp4 9.383 - libhpux 1.288 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "secstn01"
core: /dev/kmem live
link: Thu Jan 21 13:48:50 EST 2010
boot: Thu Feb 4 19:02:15 2010
time: Tue Feb 23 17:44:26 2010
nbpg: 4096 bytes
Fixed arena "BTREE_NODE_OLA_" owns 776888 pages (3.0gb):
kmem_arena_t 0xe00000013f993a00
Attributes: KMT_DEFAULT|KMT_NO_LGPG|KMT_KVADDR_CACHE|KMT_OLA_SAFE KT_DEFAULT|KT_NUMA_ILV_DEFAULT|KT_NUMA_NO_DEFAULT|KT_NUMA_TRANSITORY_OK KAS_ALIVE
Regular (Interleaved First) free lists account for 0 pages (0.0b):
Regular (Closest First) free lists account for 776888 pages (3.0gb):
Free objects represent 2.5gb (86%) of all memory in this arena.
Per index summary (all cpu's):
idx objsz pages bytes % nobjs used free %
0 4096 776888 3.0g 100 776888 110853 666035 86
Per cpu free list details:
idx cpu kmem_flist_hdr_t pages bytes % nobjs used free %
0 0 0xe00000013f996e00 109821 429.0m 14 109821 1096 108725 99
0 2 0xe0000001bd92a280 110826 432.9m 14 110826 2237 108589 98
0 4 0xe0000001bd9a7700 112183 438.2m 14 112183 12170 100013 89
0 6 0xe0000001be46ea80 66496 259.8m 9 66496 6602 59894 90
0 8 0xe0000001be52d700 110489 431.6m 14 110489 69532 40957 37
0 10 0xe0000001be4a8080 107430 419.6m 14 107430 2334 105096 98
0 12 0xe0000001bea4d500 112880 440.9m 15 112880 14924 97956 87
0 14 0xe0000001bea24480 46763 182.7m 6 46763 1958 44805 96
root@secstn01:
root@secstn01:/software/hpux/11iv3/kmeminfo/kmeminfo -arena btree_chunk_are
tool: kmeminfo 10.00 - libp4 9.383 - libhpux 1.288 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.31 64bit IA64 on host "secstn01"
core: /dev/kmem live
link: Thu Jan 21 13:48:50 EST 2010
boot: Thu Feb 4 19:02:15 2010
time: Tue Feb 23 17:45:47 2010
nbpg: 4096 bytes
Fixed arena "btree_chunk_are" owns 396323 pages (1.5gb):
kmem_arena_t 0xe00000013f993b80
Attributes: KMT_DEFAULT KT_DEFAULT|KT_NUMA_ILV_DEFAULT|KT_NUMA_NO_DEFAULT|KT_NUMA_TRANSITORY_OK KAS_ALIVE
Regular (Interleaved First) free lists account for 0 pages (0.0b):
Regular (Closest First) free lists account for 396323 pages (1.5gb):
Free objects represent 1.5gb (97%) of all memory in this arena.
Per index summary (all cpu's):
idx objsz pages bytes % nobjs used free %
0 512 396323 1.5g 100 2774261 86291 2687970 97
Per cpu free list details:
idx cpu kmem_flist_hdr_t pages bytes % nobjs used free %
0 0 0xe000000150184080 48204 188.3m 12 337428 12865 324563 96
0 2 0xe0000001bd914a00 46238 180.6m 12 323666 7546 316120 98
0 4 0xe0000001bd942f00 47358 185.0m 12 331506 9853 321653 97
0 6 0xe0000001be46e080 52069 203.4m 13 364483 14290 350193 96
0 8 0xe0000001be51ec80 54334 212.2m 14 380338 11683 368655 97
0 10 0xe0000001be5f0500 55949 218.6m 14 391643 8797 382846 98
0 12 0xe0000001bea5f380 44805 175.0m 11 313635 10011 303624 97
0 14 0xe000000150800680 47366 185.0m 12 331562 11246 320316 97
root@secstn01:
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-23-2010 06:32 PM
02-23-2010 06:32 PM
SolutionSo -- from your output: You could get 4% back by setting filecache_max to the value of filecache_min instead of letting it float between 5 to 10% (currently at 9). Since you will get that memory back anyway if you're under pressure and you have plenty, I probably wouldn't do it... but it is your system and you're welcome to try. (It is dynamic, and you can always raise it back if you want). You can get more by setting both lower than 5% (down to 1%) -- but of course you're going to incur more I/Os by losing the caching effects.
The btree arenas will have one of two effects -- either this itinerant workload still occurs, in which case since 7 (of 8?) freelists have plenty of memory you shouldn't see much increase as the system will just recycle the objects already there. Or if it doesn't, these objects will (eventually) be returned to the system -- but with no pressure, we're talking a timescale of days.
On the positive side, the BTREE_NODE_OLA arena has a couple important characteristics here -- first, it allocates objects equal in size to a system page (which simplifies garbage collection) and second, it needs to be able to swap these pages out to disk under pressure -- so the pages do not come from the Super Page Pool layer [and this arena is therefore not dependent on that layer being unfragmented to return memory to the system]. If memory pressure occurs -- this arena is going to contribute significantly to a garbage collection pass, and the memory can go right back to the system (each free object is returnable by itself).
As far as forcing that out -- I'd want to check something before I discuss that, so I'll follow up tomorrow morning. But realistically, this memory should come back for user space allocations readily, so I wouldn't worry much about it.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-24-2010 06:47 AM
02-24-2010 06:47 AM
Re: memory usage in HP-UX v3
One of the things the loratune command provided in those patches does is to endeavour to reclaim memory from caches to the system (with the presumed reason that you want to then use it in a locality-aware fashion, but the reclamation is locality agnostic as the 'wrong' locality might have been used because the 'right' locality for the caching layer was already allocated elsewhere -- hence the cache flushing isn't really targeting the user desired locality). If you invoke it with no arguments, you'll take a cpu spike in the short term (since it will go do all this GC behavior) but you can flush that memory back to the system on the Arena (possibly the SPP) side.