High system memory usage, max_dbc_pct is not high.

[ricardor_1]

Dynamic Arenas:


Variable arena "M_IOSYS" owns 1445287 pages (5.5gb):
Free objects represent 140.0kb (0%) of all memory in this arena.
idx objsz pages bytes % nobjs used free %
0 24 1443373 5.5g 100 181864998 181864802 196 0
1 56 910 3.6m 0 57330 57248 82 0
2 120 13 52.0k 0 403 367 36 9
3 248 101 404.0k 0 1515 1497 18 1
4 504 45 180.0k 0 315 130 185 59
5 1016 174 696.0k 0 522 503 19 4
6 2040 243 972.0k 0 243 243 0 0
15 4096 4 16.0k 0 4 4 0 0
16 8192 2 8.0k 0 1 1 0 0
23 36864 422 1.6m 0 46 46 0 0

[ricardor_1]

The attached files contains the requested commands.

We do have ipfilter installed:


IPF-HP A.11.23.15.01 HP IPFilter 3.5alpha5
PFIL-HP A.11.23.15.01 HP IPFilter PFIL Interface


You mean PHKL_37233? We have PHKL_37305 installed.

[Don Morris_1]

PHKL_37305 was one of the ones I wanted to see, yes.

IPFilter has some fixes in A.11.23.17 that I thought might play into this - but those should show a 512 byte allocation pattern, not a 24 byte object pattern as you show.

That small of an allocation does make it harder to track down by just pawing through the existing allocations. Rather, monitoring the actual allocations to get the code path causing the leak should be the next step. Unfortunately, that involves more intrusive tools than kmeminfo by itself.

Contact support and have them work with you to get a leak analysis done on one or the other of the machines exhibiting the issue. (The tool involved to do this can most certainly negatively impact the system in some cases [the way it should be used for yours is fairly unobtrusive, but I suspect you'll need a reboot to be able to run the tool and see the leak in progress], so I don't want to just hand it to you and hope for the best).

[ricardor_1]

Don,

Thank you very much for your attention. We have an open support case and they asked if there were any removed LUNs from the system in the past, before the last reboot, which is the case.

I´ll submit your points and update this thread as soon as we find sometime. We may need to reboot this server anyway, since the existing paging rate is degrading performance considerably.

[mvpel]

My M_IOSYS arena has 14.5GB in it, nearly all of which are extra-large pages.

Don, what's the command needed to paw through the allocations?