Very high system memory

[manoj_pu]

Hi Guys ,
I am getting very high utilisation for system memory of my Superdome Server. It's consuming 4GB out of 16 GB. Please help me to trace this problem.
OS version is HP-UX 11.11
Please help in analysing the output of "crl -v" to take the remedy to reduce memory usage.

=======================
= Kernel Memory Usage =
=======================
Buffer cache pages = 293314 (1.12 GB)
Total kernel dynamic memory pages = 878823(3.35 GB)
Kernel memory in super page pool = 40159 (156.87 MB)

Top 10 kernel memory arenas
===========================
Arena Pages
----- -----
M_TEMP 619189 (2.36 GB)
M_SPINLOCK 43319 (169.21 MB)
VFD_BT_NODE 26632 (104.03 MB)
M_NFS_GEN 25137 (98.19 MB)
ALLOCB_MBLK_LM 21931 (85.67 MB)
M_LVM 7012 (27.39 MB)
KMEM_ALLOC 6636 (25.92 MB)
M_SWAP 4320 (16.88 MB)
ALLOCB_MBLK_SM 3567 (13.93 MB)
M_REG 3313 (12.94 MB)

Total all arenas 785810 (3.00 GB)
=====================
= User Memory Usage =
=====================


Thanks in advance
Manoj

[Geoff Wild]

Looks like DBC_MAX_PCT is set to high...

kmtune -q dbc_max_pct

Should be around 7 for your system...

Rgds...Geoff

[RAC_1]

glance -m
swpainfo -mat

[Don Morris_1]

4Gb out of 16Gb doesn't mean much without context. If you aren't low on memory, the kernel is very relaxed about releasing allocations to allow for bursty workloads. Specifically, if vhand isn't running yet and you aren't low on kernel virtual address space (which you shouldn't be with only 3Gb out of 16Gb used for kernel dynamic), the arenas are not going to be garbage collected. So if you generated a large network load (or some other client of M_TEMP) that memory will tend to sit there until you drive available memory down to lotsfree.

Other possibilities are that vhand is getting preempted too much, a leak in a client of M_TEMP (so the memory isn't freed back to the Arena), etc.

What's the system free memory? Is it above lotsfree? If it isn't -- what problem are you experiencing regarding the system usage? (If the memory is just sitting there anyway -- what's the issue with the kernel holding onto it until users request more?)

[manoj_pu]

Hi Don,

Presently there is no problem in the system.
Normally the kernel memory usage vary between 1.5GB to 1.8GB in the system.

Few days back there was a problem with application test (Bad coding). Application process occupied rest of the physical memory aswell as swap.

After that system started showing kernel memory uaseg as 4GB.

I am not sure that I am asking the right question.
Is it possible to reduce M_TEMP kernel memory usage?


Thanks & Regards
Manoj

[Mohanasundaram_1]

Hi Unni,

If you know that the application has caused this, then it is a good Idea to start digging the issue from there.

What has changed on the system since that "bad coding"?

Did the number of users increase? Are there batch programs running which takes memory?

Is the memory utilisation always 3 GB or does it go up/ come down frequently.

Often performance issues are difficult to pinpoint without studying the system a lot. Is there any user complaint about the performance.

If you are just concerned by the amount of memory usage, you need to check the sizing done initially and compare it.

With regards,
Mohan.

P.S. I am just assuming you are Manoj.p. Unni at Mumbai.

[manoj_pu]

Hi Mohan,

There is no change in the system before we faced the application problem.

When we do application test, it runs couple of batch jobs simultaneously and those batch jobs require lots of physical memory and also it reserve huge virtual memory.

As I mentioned in my previous reply system started showing 4.xxGB as system memory usage in the glance.
This system memory usage does not vary much.

Before we experienced the application problem the system memory usage was 1.5GB to 1.8GB.

As of now there is no performance degradation in the system and the users are happy.

My concern is that Is it possible to reduce the system memory usage? What is M_TEMP in the kernel memory?

If I reboot system does it help to reduce M_TEMP kernel memory usage?


Thanks & Regards

Manoj

P.S. You are right Mohan I am the same Manoj who got a chance to work with you at Mumbai.

[Sameer_Nirmal]

If I am not wrong, ( other people can correct me ) M_TEMP is the arena who request the memory from free memory pool on behalf ok kernel to do kerenl jobs.
Reducing the memory for M_TEMP will depend upto kernel parameters configured and application usage of system resources.

Did you change any kernel parameters before or after running that test application? What was the system statistics when test application was ran? As you said it was using complete swap space , just curious to know what was other stats.

Can you put the current kernel parameters , swapinfo -tam , ps -efl, top,vmstat, sar

[Raj D.]

Hi Manoj ,

check this command ,

# ps -el | sort -r -k10 | head -20

It will list the top 20 processes using high memory page in the SZ filed , ie column 10. And hope you can find out something.

Many Thanks ,

Raj.D.
-----

[Don Morris_1 · Accepted solution]

Manoj --

It would be tough to tell you if you can reduce M_TEMP without knowing if those pages are still in use or free. (And even if they're free -- there isn't a user-space interface to force garbage collection of kernel memory... you'd pretty much have to force memory pressure by allocating a bunch in user space and touching it all).

M_TEMP is, frankly, the "Misc" pool for kernel memory allocations. People who don't want to create their own Arenas tend to use M_TEMP (at least on older releases... things have gotten better as time has gone on), so I can't say exactly what's taking the space within the kernel.... could be packets for a driver, could be Filesystem code, could be SysV semaphores, etc.

If you're really curious as to how much the kernel is really using (as opposed to memory waiting to be garbage collected), I'd recommend the unsupported tool kmeminfo at:

ftp://eh:spear9@hprc.external.hp.com/

Take a look at the summary first (just run kmeminfo /stand/vmunix /dev/kmem), and if you want to zoom in on Arena usage, kmeminfo -V -a M_TEMP (or whatever arena you care to peek at). If you see a lot free, then that's a sign you had a kernel memory spike that'll get freed back once there's enough pressure for garbage collection to run. If you still see a lot used after the problem application -- you may well have a kernel memory leak and I would think you'd want to have support take a look. They'll likely want to look at the system as it replicates, but that's their call.

[manoj_pu]

Thanks everyone especially Don for the help.