Hello,
We've recently gotten to the bottom of a very similar problem, having to do with the way in which the kernel's superpage pool - which is used to supply pages of memory to various kernel memory arenas by the arena alocator - is responding to certain patterns of allocation and freeing.
When the kernel needs more memory to satisfy a request from a kernel subsystem, and it does not have any blocks already available of sufficient size, it pulls in a new fixed-size region of memory, called a "superpage," from the system-wide free memory pool. The size of the region depends on the amount of physical memory.
On our 32GB systems, a new superpage is 256MB, so that may be about the size your 28.5GB system is using. Our 64GB systems use a 1GB superpage.
Then the superpage is broken in half enough times to supply a memory region of sufficient size for the arena's request. So if a kernel driver or the like requests 64MB of kernel memory, your 265MB superpage will be broken in half once, then once again to a 64MB block, and two blocks - a 128MB and a 64MB - will be placed on the superpage free list.
Things break down, though, if the requests are not a power of two. If the request is 129MB, for example, you'll have a 128.5MB superpage with one 129MB region, plus a free list consisting of 128MB, 64MB, 32MB, 16MB, 8MB, 4MB, 2MB, 1MB, and a 512k regions. The kernel pulled 256MB from free memory to satisfy a 128.5MB allocation. If there's another 128.5MB allocation, it can't fit in any of the free spaces left in the previous superpage, so the kernel has to fetch another 256MB out of the system free pool and divvy it up in the same way.
Then, if some other kernel arena needs even one page of memory (4k) it may take it out of the 49.9% free space in one of these superpages. And until every last allocation in a superpage is freed, that 256MB can't be returned to the system free pool.
And as more and more allocations come along, the chances of any one superpage being freed dwindle to nearly zero, leaving many gigabytes stuck in the "system" category. One of our superpages had ONE page allocated out of a full gigabyte.
You can also see this behavior during backups if the inode cache table is set too large - the VxFS system during a backup winds up needing a huge amount of kernel memory for the vx_inode_cache arena to cache inodes which will only be touched once during a backup, so more and more superpages are pulled in.
But while the inodes are cached sequentially by the backup and push the superpage pool to a high water mark, they're unlikely to be expired sequentially, so you end up with a lot of one-page vx_inode_cache allocations scattered across many nearly-empty superpages.
Judging by the superpage pool free number you show of only 74.7 megabytes, that's evidently not your problem - most of the allocations from the pool are still reserved by the arenas that are using them.
You can get a better sense of this by looking at the kmeminfo -arena output - you'll probably see one of the dynamic arenas such as M_IOSYS overwhelmingly responsible for the memory utilization in that list. The M_TEMP arena would also merit some scrutiny.
You may also be able to do "kmeminfo -kas analyze" - better to do it on a dump image than a live system, though - to see how each of the superpages is broken out to the arenas. Warning: it takes a very, very long time to run.
Here's another thread with some good references:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?admit=109447626+1290356069741+28353475&threadId=1009741It's possible that if you push your freemem number below lotsfree, you'll spur some memory reclamation activity and free up some unused space from arenas back into the superpage pool, but that won't be guaranteed to free up an entire superpage for return to the system free pool.
Good luck!
