Math/kwdb and looking at the data provided.
sizeof(pfd_t) on v3 - 200 bytes. Base pagesize reported 4096 bytes. (200 * 100) / 4096 = 4.8828125. Hence pfdat mandatory cost [being a per-page structure] is 4.88%. In practice, you pick up a little more fluff for the other levels of the table, so 5% is a good rule of thumb.
The VHPT tries to be about 1% of memory [sizeof(pte_t) / 4096 = 0.78%, but there is a little extra+ rounding].
The Overflow PTEs are another 1% or so [sizeof(ovfl_pte_t) / 4096] and this tries to be one per page plus extra for Alias translations and Memory Mapped I/O.
The system critical pool area works out to about 0.2% (I don't think that heuristic is documented, nor should it be).
The PFN2V area is around 0.4% (sizeof(pfn2v_entry_t) / 4096 = .39%).
So that's 5% + 1% + 1% + .4% + .2% or 7.6%.
There's a metadata cache to help the filecache that's about 1%, so that's 8.6%.
Of all that -- the PFDAT, PFN2V, VHPT and Overflow PTE sizes are all per-page, so raise the base pagesize, you have fewer pages -- the cost is less.
And filecache_min is a flat reservation as stated in `man 5 filecache_min`: "The amount of physical memory that is specified by the filecache_min tunable is reserved and guaranteed to be available for file caching." Hence kmeminfo will report it as used (since from the System perspective, that amount is -- even if the particular physical pages aren't chose yet).
(3) is a matter of the dump provided showing almost 1Gb in the Super page pool. Barring this being a ccNUMA based system with 17 localities and the submitter leaving that little fact out -- that's a lot of memory hanging around on a 16Gb system (kmeminfo gives the total SPP layer memory, but there may be multiple distinct caches in play based on the ccNUMA layout). Since memory only hangs around in the SPP layer when some of it is in use and Free memory is close to 0% (hence Garbage Collection should be in play) that strongly implies the arenas using memory are also preventing SPP coalescing [GC is pushing what it can to the SPP layer, but it isn't going further]. Since the VM white paper both predates the whole Super Page Pool implementation and I don't believe goes into any real detail on Kernel Dynamic Memory anyway -- there's nothing external to cite here.
(2) is granted a matter of experience and knowing that all the top arenas other than Async are File System/File Caching related and are all cached above the Arena layer when inodes are cached. Hence if you reduce inode caching, you reduce the caching of these other objects. Reduce the caching of the other objects, the memory gets freed to Arena -- the GC can find it... and hence (3) can clear up as well. The reference to it being aggressive by default is in the man page as I mentioned.
Alternately, you could start from
http://docs.hp.com/en/7779/commonMisconfig.pdf and then realize that in v3 there are several additional structures in VM (VAS, pregion, region plus UFC-specifc stuff) -- making the section on the VxFS inode tuning more important.
You can ask me how I know, I suppose -- but since the answer is "I read the UFC design and implementation and have had to triage v3 for years now" I don't see how it does much good or is anything beyond an argument from authority at heart.
(4) is purely a matter of that's the arena for the async driver and hence checking the configuration of said driver would be the only way that memory load could be reduced. That load may be appropriate and required for Oracle performance, of course -- hence the recommendation to check said documentation on what Oracle seeks to do with this driver.
