Reboot after panic: kalloc: out of kernel virtual space

[yulianto piyut]

hi,

we have a system running hp-ux 11.11. we plan to install data migration software called Topio. During installation, system panic when "loadable module".
in shutdownlog, OS panic coz of out of kernel virtual space. i have checked in kernel configuration, the nsysmap is 8192.

03:51 Fri Jul 27 2007. Reboot after panic: kalloc: out of kernel virtual space

thx,

-yut-

[Sandman!]

How much memory (real and virtual) do you have on that box? And what is your swap space consumption? Post the output of:

# swapinfo -tam

[skt_skt]

looks like swap space is lesser.

In order to confirm

how much is physical mem
swapinfo -tam
crashconf
what is the mem/swap usage when the system crashed(make use of PerfView)
cat /etc/shutdownlog
anything specific in syslog
how many databses running?is that Oracle
if it is an application , do they consume much swap space.?

#kmtune|grep -i chunk

[yulianto piyut]

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
dev 7168 0 7168 0% 0 - 1 /dev/vg00/lvol9
reserve - 6906 -6906
memory 34025 14320 19705 42%
total 42217 21226 20991 50% - 0 -

[yulianto piyut]

system running one DB oracle. belows, grep of /etc/shutdownlog:

17:29 Mon Jul 09 2007. Reboot after panic: kalloc: out of kernel virtual space
20:30 Thu Jul 26 2007. Reboot after panic: kalloc: out of kernel virtual space
01:29 Fri Jul 27, 2007. Reboot: (by Athpux10!yulianto)
15:19 Thu Jul 26, 2007. Reboot:
03:51 Fri Jul 27 2007. Reboot after panic: kalloc: out of kernel virtual space

# kmtune |grep -i chunk
maxswapchunks 16384 - 16384
swchunk 65536 - 65536
voliomem_chunk_size 65536 - (64*1024)

[Dennis Handly]

I don't think this message has anything to do with the anything that swapinfo -tam will show. It is the memory that the kernel needs for itself. And you also found it deals with nsysmap.

google finds: nsysmap tunable
http://docs.hp.com/en/B2355-60105/nsysmap.5.html

Searching ITRC for that string shows some hits too.

You should contact the Response Center because there have been bugs with that message.

Do you have patch PHNE_36168?
You could compare your stack trace with that of CR JAGaf47960 there.

[yulianto piyut]

bugs ?

we don't have PHNE_36168

# swlist |grep PHNE_36168

root@Athpux10
/
#

[Don Morris_1]

First check your syslogs to see if there are messages of the form:

sysmap{32|64}: rmap ovflo, lost [X,Y)

If you're running the 32-bit kernel 11.11, you may literally just be out of kernel virtual address space (it only gets 1Gb).

If you're running the 64-bit kernel, nsysmap64 is the tunable that matters, not nsysmap. If you've got the sysmap overflow message(s) cited above, then you likely have fragmentation which leaked the virtual address space. Increase nsysmap64 (doubling is the usual practice). You may wish to contact support to see if there's anything odd going on aggravating the fragmentation as well, but this can happen and is why the tunable is there.

If you don't have the overflow messages, and are running 64-bit 11.11 -- definitely give one of these dumps to support so they can determine where the kernel virtual address space went.

[skt_skt]

As suggested check if value for nsysmap/nsysmap64 is too low.

Also the value for swchunk is set too high on your side whihc could casue high memory utilization(by kernel too).

Check the kernel parameter details for swchunk.

[yulianto piyut]

hi,

system running on hpux 11.11 using superdome server. i think is 64 bit. OS is panic and no other message in syslog.log. the message only at /var/adm/shutdown.log "03:51 Fri Jul 27 2007. Reboot after panic: kalloc: out of kernel virtual space".

only these error i found in my system

[Don Morris_1]

Yeah, the fact that your swapinfo indicated over 32Gb of RAM should have clicked in my mind that you're 64-bit.

With no fragmentation messages, and a 64-bit kernel out of virtual address space (and you should have had something like 60-70Gb of kernel virtual address space), you'll need to give the dump to support to determine what happened here. [I wouldn't worry about swchunk -- first, that doesn't eat all that much memory compared to what you have, and second -- the kernel ran out of virtual space, not physical... which is very rare (since normally the kernel consumes virtual to map physical and since it starts with around 3x more virtual in the first place).] This is almost certainly a leak - and since you don't have signs of the best reason for a leak (fragmentation), dump analysis to try to determine what did leak and why is needed.

[skt_skt]

try "kmeminfo" whihc can tell if there is a memory leak;

[yulianto piyut]

hi,

below the informations from kmeminfo:

# ./kmeminfo
kmeminfo (5.06)
unix: /stand/vmunix 11.11 64bit PA2.0
core: /dev/kmem live
link: Thu Dec 8 10:36:54 TST 2005
boot: Fri Jul 27 03:34:07 2007
dump: Wed Aug 1 10:33:02 2007
nbpg: 4096 bytes

----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):

Physical memory = 10969088 41.8g 100%
Free memory = 4636303 17.7g 42%
User processes = 4490933 17.1g 41% details with -user
System = 1816627 6.9g 17%
Kernel = 939099 3.6g 9% kernel text and data
Dynamic Arenas = 273105 1.0g 2% details with -arena
M_TEMP = 180292 704.3m 2%
M_SPINLOCK = 21610 84.4m 0%
M_LVM = 19304 75.4m 0%
VFD_BT_NODE = 12275 47.9m 0%
M_IOSYS = 8095 31.6m 0%
Other arenas = 31529 123.2m 0% details with -arena
Super page pool = 3029 11.8m 0% details with -kas
Static Tables = 555928 2.1g 5% details with -static
pfdat = 250585 978.8m 2%
htbl2_0 = 131072 512.0m 1%
nbuf = 70992 277.3m 1% bufcache headers
pfn_to_virt = 41764 163.1m 0%
bufhash = 40960 160.0m 0% bufcache hash headers
Other tables = 20554 80.3m 0% details with -static
Buffer cache = 877528 3.3g 8% details with -bufcache


I don't know the meaning of those.

[skt_skt]

this output looks very much normal.

[skt_skt]

check the following parameters too. using higher value on dbc_max_pct(default is 50) can also create the memory crunch.

kctune|grep dbc
dbc_max_pct 10 10 Immed
dbc_min_pct 5 Default Immed

[skt_skt]

check the following parameters too. using higher value on dbc_max_pct(default is 50) can also create the memory crunch.

#kctune|grep dbc
dbc_max_pct 10 10 Immed
dbc_min_pct 5 Default Immed

[Don Morris_1 · Accepted solution]

This isn't going to be a DBC tunable problem (and please stop suggesting random things -- this isn't a kernel physical memory consumption problem, its a kernel virtual address space issue... they're not the same at all).

Anyway -- I think I know what happened here.

This is 11.11, so it is obviously PA-RISC. One gotcha on PA is that kernel executable code has to be within a 32-bit offset of the text start (there's a issue with the jump instruction arguments used here...). As such, all PA Dynamic Loadable Kernel Modules (DLKMs) must use a virtual address within a 32-bit compliant range [there's a special set-aside virtual address space for this].

If the allocation for the executable part of the DLKM is made such that it blocks until it succeeds, failure to allocate from *that* small virtual address pool will result in this panic. (A non-blocking allocation that can tolerate failure would just get failure).

The size of this 32-bit virtual address pool is only 1Gb... and any low address physical memory consumed during boot (which gets an "equivalent" map or the same virtual address) can also drain it, so if your kernel is fairly large or there's a lot of early boot (equivalently mapped) memory, you may simply not have much left to load the module in question.

kmeminfo -sysmap

will give some information on this -- run it on the dump, and look at the 32-bit sysmap, also run it on your live system (preferably shortly after boot) to see what you have to work with. Reducing kernel tunables which cause early boot table (or compiled in tables) may help (the SysV semaphores, shmmni, max_mem_windows all come to mind -- others may have other suggestions).

Also check the size of the Topio module that was trying to be loaded -- if it is very large, you may be fighting a losing battle.
You may need to talk with the provider if this is the case to see if they can trim the module down somehow.

Again, if you really want to pursue this -- you should engage support to get their expertise and tool set. (I can tell you that the 32-bit PA limitation is not something that can be changed, as much as we'd like to since this also affects vPar PA layouts, etc.)

[Dennis Handly]

>Don: Also check the size of the Topio module that was trying to be loaded -- if it is very large

Isn't one solution is to statically link it as part of the kernel? It isn't as flexible but it may get it to work?

[Don Morris_1]

Good point -- that's certainly worth a shot. (If its a true static linking and not a DLKM-load-on-boot... then it may push the static data further up, but that can use the virtual addresses above 1Gb safely enough).

[yulianto piyut]

thx to all,

for testing installation, i have tried to install this replication software to other partition (4 CPU and 8 GB mem) in same server (Superdome with 2 partitions). OS version is same and installation running well,, doesn't cause system panic.

memory and cpu in system going to panic during installation consists of 14 CPU and 42 GB memory. there is no patches PHKL_32668 (s700_800 11.11 vPar enablement,DLKM load panic).

PHKL_32668:
( SR:8606385530 CR:JAGaf45684 )
PHKL_29890, fix for JAGae89817, breaks the assumption made
by a locking mechanism for freeing memory. This lead to
the locking mechanism to hold up memory and cause a
leak. This inappropriate assumption made by the locking
mechanism was rectified by PHKL_24554 (JAGad68768). As the
hard dependency of PHKL_29890 on PHKL_24554 was not enforced
this problem was exposed.

Resolution:
This patch which supersedes PHKL_29890 enforces the hard
dependency on PHKL_24554.

PHKL_29890:
( SR:8606327556 CR:JAGae89817 )
This happens on large memory systems due to
unavailability of 32-bit kernel virtual addresses.
DLKM needs 32-bit kernel virtual addresses to load
the module, but these addresses are already
taken by kernel data structures.

Resolution:
Now 32-bit kernel virtual addresses are left
unallocated for later use by other modules such
as DLKMs.

[skt_skt]

PHKL_29890 has some warnings.

PHKL_32668 is the supersed one and this is the one we use.

[yulianto piyut]

Case closed!

install patches PHKL_32688

thx for all!