c-Class BL460cGen8 automatically got shutdown

[ajai4evr]

Hi Team,

   After checking the /var/log/message file, I was able to find the below logs which points to shutdown.

 Mar 17 20:20:00 mp kernel: bnx2x: [bnx2x_stats_update:1230(eth02)]storm stats were not updated for 3 times
Mar 17 20:20:00 mp kernel: bnx2x: [bnx2x_stats_update:1231(eth02)]driver assert
Mar 17 20:20:00 mp kernel: bnx2x: [bnx2x_panic_dump:916(eth02)]begin crash dump -----------------
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:926(eth02)]def_idx(0x1c48) def_att_idx(0x81ba) attn_state(0x0) spq_prod_idx(0x53) next_stats_cnt(0x1c3d)
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:931(eth02)]DSB: attn bits(0x0) ack(0x10) id(0x0) idx(0x81ba)
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:932(eth02)] def (0x0 0x0 0x0 0x0 0x0 0x0 0x0 0xb4fc 0x0 0x0 0x0 0x0 0x0 0x0 0x0 0x0) igu_sb_id(0x0) igu_seg_id(0x1) pf_id(0x0) vnic_id(0x0) vf_id(0xff) vf_valid (0x0) state(0x1)
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:983(eth02)]fp0: rx_bd_prod(0x5da2) rx_bd_cons(0xda4) rx_comp_prod(0xbf46) rx_comp_cons(0xaf46) *rx_cons_sb(0xaf46)
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:986(eth02)] rx_sge_prod(0x81c0) last_max_sge(0x7de6) fp_hc_idx(0x4ed7)
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:1003(eth02)]fp0: tx_pkt_prod(0xafbf) tx_pkt_cons(0xafbf) tx_bd_prod(0x5f0) tx_bd_cons(0x5ef) *tx_cons_sb(0xafbf)
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:1014(eth02)] run indexes (0x4ed7 0x0)
Mar 17 20:20:01 mp kernel: bnx2x: [bnx2x_panic_dump:1020(eth02)] indexes (0x0 0xaf46 0x0 0x0 0x0 0xafbf 0x0 0x0)pf_id(0x0) vf_id(0xff) vf_valid(0x0) vnic_id(0x0) same_igu_sb_1b(0x1) state(0x1)
Mar 17 20:20:01 mp kernel: SM[0] __flags (0x0) igu_sb_id (0x2) igu_seg_id(0x0) time_to_expire (0x46779c44) timer_value(0xff)
Mar 17 20:20:01 mp kernel: SM[1] __flags (0x0) igu_sb_id (0x2) igu_seg_id(0x0) time_to_expire (0x46a9e09b) timer_value(0xff)

     May I know what is the reason behind this shutdown? Is it something related to BCM57810 adapater?

[ajai4evr]

I saw this thread "https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c04722043" saying that this issue will be observed under firmware of 7.12.xx; but in this particular case, the firmware is 7.13.75.

[Kashyap02]

Due to a bug in the unload sequence of the pre-boot drivers, the firmware of some internet adapters can become unresponsive after the bnx2x driver takes over the device. The "bnx2x" driver detects the problem and returns the message "storm stats were not updated for 3 times" in the kernel log. 

Refer to the redhat article for the same: https://access.redhat.com/solutions/1351793 

Update the NIC driver and firmware to the latest. 

[ajai4evr]

I am not able to access the link you provided. Could you please paste here the important points from that particular link?

Also, may I know which is the latest version of NIC driver and firmware as in this particular case it is 7.13.75

[djhlasdkjfhlkaj]

Paraphrasing: "You are SOL"

Resolution

Red Hat bugzillas which were raised for this issue has been closed with status as

CLOSED CANTFIX

because the issue found to be present with

NIC firmware OR with non Red Hat shipped component.

HP has provided a

firmware bundle

that contains the fix.

One need to Contact Hardware / NIC vendor and get the firmware updated to latest / recommended version.

Root Cause

Due to a bug in the unload sequence of the pre-boot drivers, the firmware of some internet adapters can become unresponsive after the

bnx2x

driver takes over the device. The

bnx2x

driver detects the problem and returns the message

storm stats were not updated for 3 times

in the kernel log. To work around this problem, apply the latest NIC firmware updates

provided by hardware vendor. As a result, unloading of the pre-boot firmware now works as expected and the firmware no longer hangs after

bnx2x

takes over the device.