ongoing DL580G7 Qlogic NIC Firmware Hangs

[pirx4711]

Hi,

we had several "firmware hang detected" problems with our 10 GbE NC523SFP iSCSI NICs in the past and replaced them with Emulex NICs (VMware vSphere 4.1).

Yesterday I installed the latest Qlogic nx_nic driver for vSphere 4.1 and today one onboard NC375i NIC had the same error as the NC523SFP cards. The Management Network was disconnected for a couple of seconds.

Mar 29 08:53:55 sdev2003 vmkernel: 0:00:40:37.460 cpu15:4519)<3>nx_nic[vmnic0]: Firmware hang detected. Severity code=0 Peg number=0 Error code=c9 Return address=0

[...]

Mar 29 08:53:55 sdev2003 vmkernel: 0:00:40:37.598 cpu15:4519)<3>nx_nic[vmnic0]: Load stored FW

~ # ethtool -i vmnic0
driver: nx_nic
version: 4.0.602
firmware-version: 4.0.579
bus-info: 0000:04:00.0

Looking at the search results and the Advisory (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02964542&lang=en&cc=us&taskId=130&prodSeriesId=4118472&prodTypeId=12169) it seems that this is a ongoing problem.

Does anyone know if this actually is a hardware problem that can never be solved  by driveror firmware updates? This is a massive problem. Especially because our DL580's have 4 onboard Qlogic NICs + a Quad Qlogic Card (+ the replaced 10 GbE dual port card).

[Dave73]

Hardware problem on the 375i and T nics.

Ill try get more details for you on Monday as my collugue is on leave who is working with it.

[pirx4711]

Thanks. Seems that all Qlogic NICs have more or less hardware problems...

[pirx4711]

After collecting core dumps with a tool that HP provided, our NC375T cards and the SPI boards with the onboard NC375i will be replaced with cards that have a new revision of the chip/hardware.

[gbaccus]

---

@pirx4711 wrote:

After collecting core dumps with a tool that HP provided, our NC375T cards and the SPI boards with the onboard NC375i will be replaced with cards that have a new revision of the chip/hardware.

---

Hey there pirx,

If you don't mind me asking, can you provide any details about the replacement (firmware version, driver version, etc)? Like you, we are at the latest available versions, and have seen the card reset twice on one DB cluster node, and three times on the other. We put in an additional NIC and teamed across it, so that when it happens now, we at least don't lose the entire cluster!  :-)

Any more info you could provide would be much appreciated. We are putting our second HP ticket in on this issue.

Regards,

   Grant

[pirx4711]

The new NC375T cards are Rev. 0G, the old ones were 0F. The SPI Board is 0B instead of 0A. I've no idee what details have changed.

[gbaccus]

Thanks, Pirx. We are also getting 4 new SPI boards rev 0B for 4 DL580s we have, currently at rev 0A.

   Grant

   MTS

[RHWebster]

Its a firmware problem. You need firmware version 4.0579. The file name is CP015900.scexe (64 bit version). You need to do an off line upgrade of the firmware from the system console because the NIC's will restart. You could use ILO remote console. More instructions here:
http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/Embedded-NC375i-firmware-problems/m-p/5603853/highlight/false#M128063

[pirx4711]

In our case it was a hardware problem. We replaced all NICs with new boards (rev. 0G for the NC375T and rev. 0B for the NC375i SPI board) and haven't had a single hang since then.

[RHWebster]

That's interesting. HP have, to be fair to them, embarked on a process of replacing all our SPI boards (16) with REV B edition and all our NC 375T cards. However, there are a group of DL580G7's that have had the firmware upgrade but not the SPI replaced and have not had a hang since. I wonder if there was a bunch of boards that got bad firmware and a faulty chipset. Also, i note one poster who mentioned that his servers lost network conectivity during high load - I've got two hosts sitting in MM ( ESX servers) that regularly have the firmware hang.

Cheers

R