kalloc: out of kernel virtual space

[Rick Garland]

Got a K370 with 10.20 loaded. The system did a reboot after panic during the night. The message in the /etc/shutdownlog says the following: "Reboot after panic: kalloc: out of kernel virtual space". I am not seeing much in the syslog.log file or through dmesg.

Help is greatly appreciated!

[Steffi Jones_1 · Accepted solution]

Hi Rick,

is your system patched ok?
One patch coming to my mind is PHKL_16751 and the dependencies. Check too if the patches are all properly configured with
swlist -l fileset -a state

The message can indicate that the kernel virtual space has been exhausted
or sysmap is empty. It could be because of fragmentation, a lot of "sysmap: rmap ovflo" messages is a good indication that the cause is
fragmentation, although some fragmentation is considered normal.

This can be fixed by increasing nproc.Sysmap is sized by NPROC (NPROC * 2), so it should be increased to greater than 800, and monitored and increased further if sysmap rmap ovflo messages still appear. Sysmap is a resource map used by the kernel to manage free areas of
virtual space available to the kernel.

Hope that helps a bit,

Steffi Jones

[James R. Ferguson]

Hi Rick:

The KB contains several documents on this one. Take a look at #KBRC00003488 (: Panic string : kalloc: out of kernel virtual space):

/QUOTE/

A "kalloc: out of kernel virtual space" panic is typically caused due to sysmap fragmentation. This is fixed by increasing the size of the kernel parameter nproc.

Sysmap is sized by NPROC (NPROC * 2), so it should be increased to greater than 800, and monitored and increased further if sysmap rmap ovflo messages still appear.

Sysmap is a resource map used by the kernel to manage free areas of virtual space available to the kernel. The map is a fixed size (in
pages) and address pairs. As kernel virtual space is allocated, sysmap is updated appropriately.

As areas are freed and so returned to sysmap,
the kernel will coalesce the returned entry with an existing entry, if possible. If the area being returned is not adjacent to an already free area, and so the entry can't be coalesced with an existing entry, the kernel needs to add a new entry to sysmap. As this happens more often, which it will as a normal occurance, the number of entries in sysmap grows. This is "fragmentation".

If fragmentation continues to occur to the extent that the sysmap table fills up, then the next time the kernel frees an area and tries to add that entry to sysmap the "sysmap: rmap ovflo" message is displayed on the console with details of the entry.

A "kalloc: out of kernel virtual space" panic may also occur. What this means is that kernel virtual space has been exhausted or sysmap is empty. It could be because of fragmentation, a lot of "sysmap: rmap ovflo" messages is a good indication that the cause is fragmentation, although some fragmentation is considered normal.

Some systems that use super pages will require patch PHKL_16751 + dependancies (or its superceding patch). To find out if your system uses super pages, use adb command:

Command: echo cpu_has_var_size_pages/D | adb /stand/vmunix /dev/kmem

Output: cpu_has_var_size_pages:
cpu_has_var_size_pages: 0 or 1 <---

If the above variable is ZERO (0), then you are experiencing fragmentation from other causes. 1 means you are using super pages and you will need to install PHKL_16751 + dependancies (or its superceding patches).

/END_QUOTE/

...JRF...

[Andy Monks]

Rich, although the patches mentioned might fix the problem and changing nproc, might move the problem further away, the best solution is to get the dump looked at by HP. They will probably want to run a tool called vmtrace to monitor memory usage. With this tool, you can get it 'right' first time, rather than guesses.