panic kalloc: out of kernel virtual space

[Omar Al-Saiari]

hi,

we have hp-ux 10.20 s800 today on system painc give this error

$ more INDEX

comment savecore core dump INDEX file

hostname hp01

modelname 9000/800/K380

panic kalloc: out of kernel virtual space

dumptime Mon Apr 5 13:45:09 WAT 2004

savetime Mon Apr 5 13:55:39 WAT 2004

release @(#)9245XB HP-UX (B.10.20) #1: Sun Jun 9 06:31:19 PDT 1996

memsize 1073729536

chunksize 134217728

module /stand/vmunix vmunix 0x0000000000000000 0x0000000000000000 9939200 226

3989379

chunk 0x0000000000000000 0x0000000008000000 core.1.1 1852214058

chunk 0x0000000008000000 0x0000000008000000 core.1.2 726922542

chunk 0x0000000010000000 0x0000000008000000 core.1.3 1470908281

chunk 0x0000000018000000 0x0000000008000000 core.1.4 812427326

chunk 0x0000000020000000 0x0000000008000000 core.1.5 1711175015

chunk 0x0000000028000000 0x0000000008000000 core.1.6 4071357713

chunk 0x0000000030000000 0x0000000008000000 core.1.7 2146265378

warning savecore: running in the background

chunk 0x0000000038000000 0x0000000007ffd000 core.1.8 3495230809

INDEX: END



Best regards,

Omar

[Mark Grant]

Seems one of those "get more RAM/swap/both" situations.

[Don Morris_1]

You get this panic due to kernel virtual address fragmentation. The kernel keeps a data structure called the "sysmap" to keep track of what virtual addresses are free. The map is only so large, so if you get a lot of small allocations which are freed back non-contiguously (in other words, if your virtual address space is Used-Free-Used-Free ...) you can overflow the sysmap. The kernel deals with this by just dropping virtual addresses it can't fit back into the map... which means that after this goes on for a while, the kernel doesn't have much virtual address space left to play with at all.

You should see "rmap overflow" messages in the syslog prior to this - showing you what virtual addresses are being dropped from the kernel virtual address space.

For 11.0 and later, there is a kernel tunable to try to handle this (basically it is a hint to the system - if you hit this scenario, you increase the tunable [double/triple is good] to decrease the likelihood of having to drop addresses). I don't believe it is there on 10.20 since I can't find it documented as such. Of course, the documentation didn't have an explicit tunable list, either -- so it would be worth it to check on your system.

The usual cause of this type of address space fragmentation is a bursty allocation workload (heavy network traffic with small packets).