rp8420 -11.11 - Reboot after panic: Arena pointer not in list

[moonchild]

Hi

server rebooted after panic with the following message in shutdownlog:

Reboot after panic: Arena pointer not in list

ts99 has no valid timestamp
and the MP log shows:

9420 OS 0,3,5 *3 78800c6235e001f8 a0e1d8c01100b000 PAT_ENCODED_FIELD_WARNING

9420 01/01/2008 01:28:51
9421 OS 0,3,5 *3 74800c6835e001fa 00000000000005e9 PAT_DATA_FIELD_WARNING

9421 01/01/2008 01:28:51
9422 OS 0,3,5 *3 78800c6235e001fc a0e1d8c01100b000 PAT_ENCODED_FIELD_WARNING

9422 01/01/2008 01:28:51
9423 OS 0,3,5 *3 74800c6835e001fe 00000000000005e9 PAT_DATA_FIELD_WARNING

9423 01/01/2008 01:28:51
9424 SFW 0,2,0 2 57800f7320e00200 2000000000000000 ERR_CPU_CHECK_SUMMARY

9424 01/01/2008 01:28:51
9425 SFW 0,3,0 2 57800f7330e00202 2000000000000000 ERR_CPU_CHECK_SUMMARY

9425 01/01/2008 01:28:52
9426 SFW 0,3,2 2 57800f7332e00204 2000000000000000 ERR_CPU_CHECK_SUMMARY

9426 01/01/2008 01:28:52
9427 SFW 0,3,4 2 57800f7334e00206 2000000000000000 ERR_CPU_CHECK_SUMMARY
9427 01/01/2008 01:28:52

9428 SFW 0,3,2 2 578003dd32e00208 0000000000000000 ERR_TOC_TASK
9428 01/01/2008 01:28:52

9429 SFW 0,3,4 2 578003dd34e0020a 0000000000000000 ERR_TOC_TASK

thank you in advance and happy new year.

[Don Morris_1 · Accepted solution]

Panic stack should be (at the bottom):

remove_from_arena_list()
arena_rele()
kmem_arena_destroy()

The function which called kmem_arena_destroy() and the arena handle in question are the important things to track down here. What's happened is that someone passed either garbage as a kernel memory arena handle or a previously destroyed handle to be destroyed again (the equivalent of a double free, but on allocator meta-data instead of on a pointer itself).

Since most of the core kernel arenas don't get destroyed (and since you mentioned this was on shutdown), my gut is that this is from a driver unloading itself -- could be the driver tries to make multiple arenas (one per instance for example) and got confused, etc. Can't really say without knowing where this was coming from.

If you want to try things yourself, load up the dump using q4 and do "trace event 0" and see who called the arena destroy function and if it is recognizable as a driver, then check that driver for updated patches.

An easier route would be to use any support contract/contact you have and get the dump to them for analysis so they can track down the problem and patches (if any exist yet, this may be a new problem).

[moonchild]

Don,

Thank you for the reply. I will try to run q4 on the crash and look into it. Meanwhile this happened while system is up and running it was not on shutdown.

[moonchild]

Thank you Don.

The Panic encountered is a known issue and it's fixed by installing the following patch:

Patch Name: PHKL_33387

Patch Description: s700_800 11.11 VM kmem arena fix

Creation Date: 05/06/03

Post Date: 05/07/11

Hardware Platforms - OS Releases:
s700: 11.11
s800: 11.11

Products: N/A

Filesets:
OS-Core.CORE2-KRN,fr=B.11.11,fa=HP-UX_B.11.11_32,v=HP
OS-Core.CORE2-KRN,fr=B.11.11,fa=HP-UX_B.11.11_64,v=HP

Automatic Reboot?: Yes

Status: General Release

Critical:
Yes
PHKL_33387: PANIC

Category Tags:
defect_repair general_release critical panic

Path Name: /hp-ux_patches/s700_800/11.X/PHKL_33387

Symptoms:
PHKL_33387:
( SR:8606394315 CR:JAGaf54338 )
System panics with this trace:
panic+0x6c
remove_from_arena_list+0xac
arena_rele+0x3c
foreach_arena+0x8c
kmem_garbage_collect+0xe8
kmem_arena_stealpages+0x54
vhand+0x2c0
im_vhand+0xd4
DoCalllist+0x3c
main+0x28
$vstart+0x48
$locore+0x94

[moonchild]

The Panic encountered is a known issue and it's fixed by installing the following patch:

Patch Name: PHKL_33387