cancel
Showing results for 
Search instead for 
Did you mean: 

memory utilization is increasing

T Jose
Advisor

memory utilization is increasing

In one of my vpar, memory utilization is gradually increasing after last reboot.
In kmeminfo I can see system is taking more memory. What could be the reason?



Physical memory usage summary (in page/byte/percent):

Physical memory = 7458552 28.5g 100%
Free memory = 379637 1.4g 5%
User processes = 2953752 11.3g 40% details with -user
System = 3493139 13.3g 47%
Kernel = 3492441 13.3g 47% kernel text and data
Dynamic Arenas = 2973589 11.3g 40% details with -arena
M_TEMP = 2393803 9.1g 32%
misc region are = 47968 187.4m 1%
vx_inode_kmcach = 47664 186.2m 1%
FCACHE_ARENA = 46159 180.3m 1%
reg_fixed_arena = 44821 175.1m 1%
Other arenas = 393174 1.5g 5% details with -arena
Super page pool = 19119 74.7m 0% details with -kas


Regards,
T Jose
9 REPLIES
Shibin_2
Honored Contributor

Re: memory utilization is increasing

Have you run any other utilities such as top, glance ( if you have ) to find out, what is eating your memory ?
Regards
Shibin
SoorajCleris
HPE Pro

Re: memory utilization is increasing


Hi ,

Dynamic Arenas = 2973589 11.3g 40% details with -arena
M_TEMP = 2393803 9.1g 32% ====> Here is the leak.

which is your OS version?
you may need to search for the VM patches which addressed memory leak at M_TEMP arena!!

Thats it!

Regards,
Sooraj
"UNIX is basically a simple operating system, but you have to be a genius to understand the simplicity" - Dennis Ritchie
Hein van den Heuvel
Honored Contributor

Re: memory utilization is increasing

T Jose,

You showed us a single snapshot with no time indication.
Please gives us an indication what specific numbers suggested to you that there might be a problem.
And why not toss in the platform, OS version, Filesystem (version)used, and essential Patch information such that we are not temped to wildly suggest to add more patches.

What is 'gradually'. Hours? Days? Weeks?
What specific component in the kmeminfo is increasing, by how much?

That M_TEMP is high, but is that the component which was growing?
By giving us more numbers the folks who lack the clairvoyance skills that Sooraj apparently has may be able to try to help better.

Can you perhaps show output this week and last week... at a similar time of day? Comparing with data right after the reboot is probably not useful, as applications still have to get up and get settled.

The apparent leak might just be goodness, as caches becoming more and more useful.
For example for 11.11 and JFS 3.3 M_TEMP is used for inode cache according to the elusive document ftp://eh:spear9@hprc.external.hp.com/memory.htm

And other topic to poke at:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1009741


Regards,
Hein
mvpel
Trusted Contributor

Re: memory utilization is increasing

Hello,

We've recently gotten to the bottom of a very similar problem, having to do with the way in which the kernel's superpage pool - which is used to supply pages of memory to various kernel memory arenas by the arena alocator - is responding to certain patterns of allocation and freeing.

When the kernel needs more memory to satisfy a request from a kernel subsystem, and it does not have any blocks already available of sufficient size, it pulls in a new fixed-size region of memory, called a "superpage," from the system-wide free memory pool. The size of the region depends on the amount of physical memory.

On our 32GB systems, a new superpage is 256MB, so that may be about the size your 28.5GB system is using. Our 64GB systems use a 1GB superpage.

Then the superpage is broken in half enough times to supply a memory region of sufficient size for the arena's request. So if a kernel driver or the like requests 64MB of kernel memory, your 265MB superpage will be broken in half once, then once again to a 64MB block, and two blocks - a 128MB and a 64MB - will be placed on the superpage free list.

Things break down, though, if the requests are not a power of two. If the request is 129MB, for example, you'll have a 128.5MB superpage with one 129MB region, plus a free list consisting of 128MB, 64MB, 32MB, 16MB, 8MB, 4MB, 2MB, 1MB, and a 512k regions. The kernel pulled 256MB from free memory to satisfy a 128.5MB allocation. If there's another 128.5MB allocation, it can't fit in any of the free spaces left in the previous superpage, so the kernel has to fetch another 256MB out of the system free pool and divvy it up in the same way.

Then, if some other kernel arena needs even one page of memory (4k) it may take it out of the 49.9% free space in one of these superpages. And until every last allocation in a superpage is freed, that 256MB can't be returned to the system free pool.

And as more and more allocations come along, the chances of any one superpage being freed dwindle to nearly zero, leaving many gigabytes stuck in the "system" category. One of our superpages had ONE page allocated out of a full gigabyte.

You can also see this behavior during backups if the inode cache table is set too large - the VxFS system during a backup winds up needing a huge amount of kernel memory for the vx_inode_cache arena to cache inodes which will only be touched once during a backup, so more and more superpages are pulled in.

But while the inodes are cached sequentially by the backup and push the superpage pool to a high water mark, they're unlikely to be expired sequentially, so you end up with a lot of one-page vx_inode_cache allocations scattered across many nearly-empty superpages.

Judging by the superpage pool free number you show of only 74.7 megabytes, that's evidently not your problem - most of the allocations from the pool are still reserved by the arenas that are using them.

You can get a better sense of this by looking at the kmeminfo -arena output - you'll probably see one of the dynamic arenas such as M_IOSYS overwhelmingly responsible for the memory utilization in that list. The M_TEMP arena would also merit some scrutiny.

You may also be able to do "kmeminfo -kas analyze" - better to do it on a dump image than a live system, though - to see how each of the superpages is broken out to the arenas. Warning: it takes a very, very long time to run.

Here's another thread with some good references:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1290356069741+28353475&threadId=1009741

It's possible that if you push your freemem number below lotsfree, you'll spur some memory reclamation activity and free up some unused space from arenas back into the superpage pool, but that won't be guaranteed to free up an entire superpage for return to the system free pool.

Good luck!
T Jose
Advisor

Re: memory utilization is increasing

Hi All,

Our OS is B.11.31 U ia64
I am attaching two output which was collected in 1 month gap.
M_TEMP is increasing.
I am also including -arena.

Regards,
T Jose
Laurent Menase
Honored Contributor

Re: memory utilization is increasing

2 things:
1) kmeminfo is a private support tool which should be used on support request only as indicated in the output headers you cut.

2) you probably have a memory leak on M_TEMP index 0, now find what is the root cause you have 2 ways:
a) guess work, and blind patching - looking at patch texts which indicate MEMORY LEAK-
b) have it analyzed by hp support, with proper tools, and tools interpretation.

Don Morris_1
Honored Contributor

Re: memory utilization is increasing

Given the size and M_TEMP, do you have PHKL_37874 already installed (or PHKL_39842 which supersedes it?) That seems the closest match to what you're describing.
Laurent Menase
Honored Contributor

Re: memory utilization is increasing

If you choose guess work I bet on PHKL_38928
Laurent Menase
Honored Contributor

Re: memory utilization is increasing

oops wrong copy past I mean PHKL_37874