- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - HP-UX
- >
- Re: Sys Mem consuming > 1/3 of Physical RAM
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2008 12:55 PM
06-12-2008 12:55 PM
Sys Mem consuming > 1/3 of Physical RAM
It has the following config:
8 Intel(R) Itanium 2 9000 series processors (1.6 GHz, 24 MB)
Firmware info:
Firmware revision: 7.048
FP SWA driver revision: 1.18
IPMI is supported on this system.
BMC firmware revision: 2.05
Memory: 32568 MB (31.8 GB)
This runs 2 Multi Terra-Byte Oracle 10.2.0.3 databases whose combined SGA takes 16GB and combined PGA takes 2.5 GB.
There is excessive paging observed on this box. One another observation is that GLANCE shows us Sys Mem at 9.5GB or round-abouts consistently.
Isn't this a very excessive value for the kernel memory?
Is this essential and can some of that memory be freed up?
FYI : We have enabled the DB to use ASYNCIO by granting the MLOCK priv to the DBA group, and use FILESYSTEM_IO_OPTIONS=setall, and ASYNC_DISK_IO=TRUE.
Thanks.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2008 01:56 PM
06-12-2008 01:56 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
kmeminfo is a great tool to help isolate. ( found on this forum)
As far as swapping/pageouts are concerned, there is still about 4GB left of free ( or at least unaccounted for ).
What does glance say about free mem ?
What does swapinfo say about swap usage ?
What does glance say about deactivations ?
If free mem is truely zero then you need to go searching for who is using up the rest, how to reduce the system usage, and / or make sure you have enough swap enabled so all your RAM can be used.
Start posting some stats and maybe we can guide you.
T.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2008 06:41 PM
06-12-2008 06:41 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
Bill Hassell, sysadmin
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-12-2008 08:17 PM
06-12-2008 08:17 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
Total VM : 13.4gb Sys Mem : 9.7gb User Mem: 18.5gb Phys Mem : 31.8gb
Active VM: 7.3gb Buf Cache: 1mb Free Mem: 3.0gb FileCache: 616mb
Results from swapinfo -t:
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4191900 0% 0 - 1 /dev/vg00/lvol2
dev 70352896 199064 70146536 0% 0 - 0 /dev/vg00/lvol8
dev 70352896 198848 70146752 0% 0 - 0 /dev/vg00/lvol9
reserve - 20958284 -20958284
memory 31721764 25726780 5994984 81%
total 176621860 47082976 129521888 27% - 0 -
Using a tool from the forum (memdetail):
Memory Stat total used avail %used
physical 32568.9 29589.3 2979.6 91%
active virtual 13905.5 7944.4 5961.0 57%
active real 10770.8 5431.1 5339.7 50%
memory swap 30978.3 25126.2 5852.1 81%
device swap 141487.4 20472.7 121014.7 14%
As for the filecache settings:
filecache_max 649661726 2% Imm (auto disabled)
filecache_min 324830863 1% Imm (auto disabled)
Since this is a DB server and we're using ASYNCIO we'd set the FILECACHE to the minimum.
The database files are on vxfs filesystems something like below from the fstab:
/u01/dat10 vxfs largefiles,delaylog,nodatainlog 0 2
Let me know if I can provide additional info.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-14-2008 02:47 PM
06-14-2008 02:47 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
Memory Stat total used avail %used
physical 32568.9 29164.6 3404.4 90%
active virtual 12221.1 3410.9 8810.3 28%
active real 10152.6 1998.2 8154.4 20%
memory swap 30978.3 25221.1 5757.2 81%
device swap 141487.4 19456.1 122031.3 14%
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 10:59 AM
06-19-2008 10:59 AM
Re: Sys Mem consuming > 1/3 of Physical RAM
Any help is appreciated.....
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 11:57 AM
06-19-2008 11:57 AM
Re: Sys Mem consuming > 1/3 of Physical RAM
please.
Also, on 11.31 glance should report buff cache at 0 (zero) as it is obsoleted. What version of glance are u running ?
kmeminfo ouputs would also help.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 01:08 PM
06-19-2008 01:08 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
As a first pass you can run the attached, but if it just basically boils down to a bunch of kernel dynamic, there's really no way to break things down without using a Support tool like kmeminfo. (Have you tried talking to support? Yes, I really hate recommending kmeminfo be run willy-nilly... but probably about the only way to break this down.)
On a side note, are you really paging (with 3Gb of RAM free, you shouldn't be) -- or is vhand having to sync dirty pages out of the file cache and that's what you're seeing as po in vmstat? With a 1 to 2% file cache that's showing as full size -- it wouldn't surprise me if you can easily get into cache thrashing. If you've got 3Gb of RAM laying around, it is worth considering giving the file cache some breathing room there.
Last but not least -- what's your VM patch level so we have context? (I'd recommend the 11.31.0803 level of PHKL_37452 if you aren't there, lots of vhand/filecache interaction adjustments through that patch stream).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 01:09 PM
06-19-2008 01:09 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
As a first pass you can run the attached, but if it just basically boils down to a bunch of kernel dynamic, there's really no way to break things down without using a Support tool like kmeminfo. (Have you tried talking to support? Yes, I really hate recommending kmeminfo be run willy-nilly... but probably about the only way to break this down.)
On a side note, are you really paging (with 3Gb of RAM free, you shouldn't be) -- or is vhand having to sync dirty pages out of the file cache and that's what you're seeing as po in vmstat? With a 1 to 2% file cache that's showing as full size -- it wouldn't surprise me if you can easily get into cache thrashing. If you've got 3Gb of RAM laying around, it is worth considering giving the file cache some breathing room there.
Last but not least -- what's your VM patch level so we have context? (I'd recommend the 11.31.0803 level of PHKL_37452 if you aren't there, lots of vhand/filecache interaction adjustments through that patch stream).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 01:11 PM
06-19-2008 01:11 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 02:14 PM
06-19-2008 02:14 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
KCTUNE -S :
Tunable Value Expression Changes
filecache_max 649661726 2% Imm (auto disabled)
filecache_min 324830863 1% Imm (auto disabled)
ksi_alloc_max 65536 nproc*8 Immed
max_thread_proc 1200 1200 Immed
maxfiles 20480 20480
maxfiles_lim 20480 20480 Immed
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
maxtsiz 134217728 134217728 Immed
maxtsiz_64bit 1073741824 1073741824 Immed
maxuprc 7372 nproc*9/10 Immed
msgmni 8192 nproc Immed
msgtql 8192 nproc Immed
nflocks 20480 20480 Imm (auto disabled)
ninode 67584 8*nproc+2048
nkthread 14352 nproc*7/4+16 Immed
nproc 8192 8192 Immed
semmni 8192 nproc
semmns 16384 16384
semmnu 8188 nproc-4
shmmax 27487790694 27487790694 Immed
shmmni 1024 1024 Immed
shmseg 360 360 Immed
swchunk 8900 8900
vps_ceiling 64 64 Immed
Glance Version: C.04.55.00
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 02:19 PM
06-19-2008 02:19 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
phys mem: 8337641 31.8g
Initial free: 7597774 29.0g
Max User Memory: 7597774 29.0g
free mem: 751040 2.9g
cfree mem: 751720 2.9g
total kernel: 4873564 18.6g
kern dyn mem: 2225206 8.5g
kern misc: 2241158 8.5g
metadata: 407200 1.6g
Daemons: 18607 72.7m
vm buf cache: 98 392.0k
file cache: 156218 610.2m
RSS mem: 2538114 9.7g
active RSS mem: 1303087 5.0g
user mem calc: 2556721 9.8g
phys mem calc: 8319034 31.7g
missing memory: 0 0.0 bytes
Kernel Classification Breakdowns:
kernel total: 2804933 10.7g
kern static: 17199 67.2m
kern dynamic: 2497466 9.5g
kern Lg Pg: 290198 1.1g
fs metadata: 70 280.0k
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-19-2008 06:05 PM
06-19-2008 06:05 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
print(" bufcache pages:", (uint64_t) pstci.psc_pgcount[PC_BCACHE],
stat.page_size);
around line 172 and re-run.
It looks to me on this limited data (and keep in mind, first this isn't exactly a well cooked pstat program and second, I can't get a breakdown within kernel dynamic) that there's been a lot of inode or direct buffer writes to the disk. There isn't currently much in the buffer cache, but I strongly suspect that that's where a good chunk of the "kernel misc" is -- free pages cached in the arena layer that don't get reported as general dynamic because they're only for buffer cache. That type of activity can also cause a bunch of filesystem metadata, which I suspect is what drove up the kern dynamic. (1.1Gb of it is just large page translation cache stuff -- it will get freed up as things garbage collect or used for new allocations... just a mid-level cache. You aren't under memory pressure, so that reclamation isn't in any hurry).
If PC_BCACHE doesn't show several Gb even with the buffer cache pstat showing only Mb (i.e. the cache has burned through several Gb of buffers but is now only a few Mb of active cache... the free buffers are being held in a cache behind _that_ (object level caching)) then I'll have to punt to getting kmeminfo output, sorry. (Well, I could try to walk you through parsing this stuff in kwdb.. but kmeminfo would be a lot easier).
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-20-2008 12:11 PM
06-20-2008 12:11 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
phys mem: 8337641 31.8g
Initial free: 7597774 29.0g
Max User Memory: 7597774 29.0g
free mem: 1046815 4.0g
cfree mem: 1046390 4.0g
total kernel: 4582793 17.5g
kern dyn mem: 1934979 7.4g
kern misc: 2240614 8.5g
metadata: 407200 1.6g
Daemons: 18607 72.7m
vm buf cache: 98 392.0k
file cache: 156631 611.8m
RSS mem: 2532697 9.7g
active RSS mem: 1204120 4.6g
user mem calc: 2551304 9.7g
phys mem calc: 8319034 31.7g
missing memory: 0 0.0 bytes
Kernel Classification Breakdowns:
bufcache pages: 1025611 3.9g
kernel total: 2514354 9.6g
kern static: 17199 67.2m
kern dynamic: 2237658 8.5g
kern Lg Pg: 259427 1013.4m
fs metadata: 70 280.0k
Don, as for the kmeminfo... I do not have the root access to this box.
I administer the databases, and am encountering some serious performance bottlenecks which I can drill down to the OS layer, and in addition when kernel mem is this high, I'm unable to add more memory to the SGA & PGA regions.
My SA is already overloaded, and doesn't have cycles for this, and I want to address this ASAP.
So, a few more questions:
Our Oracle datafile filesystems are all vxfs, and we had enabled the DB to use any means to write to the disks using filesystemio_option=setall.
Since ASYNC doesn't work with vxfs & directio is not enabled, are we being bottlenecked by buffered i/o?
Could our low setting of filecachemax @ 2%of RAM cause my problems?
BTW, databases running on this box are extreme I/O customers, single block and multiblock I/Os.
Let me know if you'd still need the kmeminfo output.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-24-2008 03:25 PM
06-24-2008 03:25 PM
Re: Sys Mem consuming > 1/3 of Physical RAM
Thanks for your time on this one.
Do you have any update based on the last set of info furnished?
Thanks.