vhand hung ? crash dump available

[Tanuj Vohra]

I have an rx1620 with 4GB RAM running 11.31 which hangs regularly when under sustained memory consuming load. I can provide a crash dump if needed or grunge around in it with kwdb if needed.

Are there any patches that solve such a problem ?

$ /usr/sbin/swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 5128192 0 5128192 0% 0 - 1 /dev/vg00/lvol2
reserve - 450760 -450760

Most of the processes are stuck like this :

Stack trace for process "engine.pure" at 0xe0000001452c5e00 (pid 7870)
Thread at 0xe000000145629480 (tid 522528)
#0 0xe000000000d34f60:0 in swtch_to_thread+0x2e0 ()
at /ux/core/kern/common/pm/sched/pm_swtch.c:465
#1 0xe000000000d3eb80:0 in slpq_swtch_core+0x310 ()
at /ux/core/kern/common/pm/sched/sched_sleep.c:3262
#2 0xe000000000c60690:0 in inline real_sleep+0x360 ()
at /ux/core/kern/common/pm/sched/sched_sleep.c:701
#3 0xe000000000c60330:0 in sleep+0x90 ()
at /ux/core/kern/common/pm/sched/sched_sleep.c:523
#4 0xe00000000227c290:0 in memory_sleep_conditional+0x130 ()
at /ux/core/kern/common/vm/resm/vm_memreserve.c:840
#5 0xe00000000088b810:0 in $cold_virtual_fault+0x6b0 ()
#6 0xe000000000d8d140:0 in inline hdl_vfault+0x2a0 ()
at /ux/core/kern/em/vm/vasm/pregion_hdl.c:2030
#7 0xe000000000d8cea0:0 in vfault+0x530 ()
at /ux/core/kern/common/vm/vasm/vas_proc.c:420
#8 0xe000000000d87d30:0 in vm_hndlr+0x560 ()
at /ux/core/kern/em/svc/misc/hl_ivt.c:862
#9 0xe000000001def780:0 in bubbleup+0x880 ()
#10 0x1ffffffffea64b06 in ?? () <--- Trap in User mode
#11 0x1ffffffffea647c0 in ?? ()

And vhand is spinning on the CPU without freeing up physical memory:

Processor 1 was running process "vhand" at 0xe0000001327f0000 (pid 2)
Thread at 0xe000000131ffe700 (tid 82)
Event selected is 1. It was a INIT
#0 0xe000000000df55d0:0 in inline kmem_gc_small+0x1af ()
at /ux/core/kern/common/vm/kmem/vm_arena_gc.c:1361
#1 0xe000000000df5420:1 in inline kmem_gc_freelist+0xb1 ()
at /ux/core/kern/common/vm/kmem/vm_arena_gc.c:1121
#2 0xe000000000df5370:0 in kmem_gc_arena+0x1f0 ()
at /ux/core/kern/common/vm/kmem/vm_arena_gc.c:1021
#3 0xe000000000f36d60:0 in inline kmem_do_gc+0x80 ()
at /ux/core/kern/common/vm/kmem/vm_arena_gc.c:0
#4 0xe000000000f36ce0:0 in inline foreach_arena_ingroup+0x80 ()
at /ux/core/kern/common/vm/kmem/vm_arena_mgmt.c:6594
#5 0xe000000000f36c60:0 in kmem_garbage_collect_group+0x1e0 ()
at /ux/core/kern/common/vm/kmem/vm_arena_gc.c:457
#6 0xe0000000006d4520:0 in $cold_kmem_garbage_collect+0x70 ()
#7 0xe0000000006d3fc0:0 in $cold_kmem_arena_stealpages+0x40 ()
#8 0xe0000000021edb00:0 in vhand_global_pager_steal_mrg+0x4e0 ()
at /ux/core/kern/common/vm/vmmon/vm_vhand.c:1856
#9 0xe0000000021ee080:0 in vhand_global_pager_steal+0x2a0 ()
at /ux/core/kern/common/vm/vmmon/vm_vhand.c:1724
#10 0xe0000000008bd620:0 in $cold_vhand_service_mem_class+0x5e0 ()
#11 0xe0000000021eec20:0 in vhand+0x1d0 ()

[Venkatesh BL · Accepted solution]

You could try installing PHKL_37452 (if its not installed already). If the problem persists, raise a case HP support.

[melvyn burnard]

I suggest you either load the latest patch bundle, or open a case with your local HP Response Centre

[Don Morris_1]

Agreed -- PHKL_37452 is a likely fix (if, in fact vhand is spending all the time in GC paths which can happen under high User+UFC load on small systems -- that was one of the primary fixes in that patch).

If the problem still occurs with PHKL_37452 installed, contact support and have them contact the lab. If this is a new issue, we'll need/want to work it (and if not, we can see where things fit in schedules for other alternatives).

[Tanuj Vohra]

PHKL_37452 has fixed my problem. Thanks a lot to both Venkatesh and Don.

-Kapil