rx4640 panic crashes: mblk is bad

[Kang Sun]

We have been running an rx4640 Integrity Server as an HPVM host (HPUX B11.31) for two years with no problem. For the past week it has crashes several times. The crash dump pointing to NIC card, or vNICs shared by two clients.

I have five crash dumps, crash1 and crash5 give panic messages and stack-trace as follows:

hpvmdvr: AVIO VSD driver registered successfully, VSD version = 2:2:1
pmanLanFrameDataAvailable: given mblk is bad. mblk=e00000071a37a180, ptr=e00401000f21e936.
panic: pmanLanFrameDataAvailable: given mblk is bad.
Stack Trace:
IP Function Name
0xe00000014e97b600 pmanLanFrameDataAvailable+0x180
0xe00000014e977600 PmanLanRx+0x840
0xe000000157918c40 switch_packet+0x5d0
0xe000000157911880 hpvmntdvrrput+0x1e0
0xe000000000bda230 putnext+0x140
0xe000000001233480 hp_dlpi_intr_rawdata_in+0x220
0xe0000000008644a0 $cold_hp_dlpi_intr_put+0xad0
0xe000000000bc2030 kstr_put+0x380
0xe000000001231340 hp_dlpi_route_promisc+0xc00
0xe0000000006407d0 $cold_hp_dlpi_ether_ib_intr_slowpath+0x420
0xe000000000bc2ab0 hp_dlpi_ether_intr+0x900
0xe00000014cbbbfa0 igelan_ilan_recv_frames+0x4c0
0xe00000014cbb9ce0 igelan_ilan_recv_and_replenish+0xe0
0xe00000014cbb90d0 igelan_recv+0x460
0xe00000014cbb9980 igelan_isr+0x180
0xe000000000c2a6e0 sapic_interrupt+0x60
0xe000000000c293c0 external_interrupt+0x4b0
0xe000000001c88780 bubbledown+0x0
0xe000000000bde370 intr_strobe_clear_idle+0x50
0xe000000000bdbf20 idle+0x7a0
End of Stack Trace

The other crash dumps simply says "bad news":

hpvmdvr: AVIO VSD driver registered successfully, VSD version = 2:2:1
Bad News: pr == 0x144000c0b10c003d
Bad News: Cannot use the Kernel Stack when interrupted on the ICS.
Bad News: Predicate set: 0x144000c0b10c003d.

I have one external NIC card, shared by two VMs

0/1/2/0 lan HP A6825-60101 PCI 1000Base-T Adapter

Since the trace has a line with hpvm,
hpvmntdvrrput+0x1e0,

It makes me think it might be software related to the HPVM, not really the physical NIC.

Any advice is appreciated.

Thanks

-- Kang Sun
ps. Attached is the crash analysis from adb.

[Laurent Menase]

Hi ,
mblocks are used by all the network stack.
a bad mblock just mean that there may have been an error before- double free for instance.

It never work with guess work,
you should contact hp support.

[Prashanth.D.S]

Hi Kang,

This is a known issue with NFS version B.11.31.04 and is fixed in version B.11.31.06.

Suggest you to install the latest version from www.software.hp.com and search for ONCplus.

https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=ONCplus

Best Regards,
Prashanth

[Laurent Menase]

As a fist guess you can indeed bet on ONC 11.31.04, and indeed it gives that type of result.
But it is not the only one.
Statistically if you have ONC 11.31.03 OR 11.31.04 it is the most frequent problem which lead to that type of panic but any double free of a mblock will give the same result.
but here it is only guess work.