ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

ongoing DL580G7 Qlogic NIC Firmware Hangs

pirx4711
Frequent Advisor

ongoing DL580G7 Qlogic NIC Firmware Hangs

Hi,

 

we had several "firmware hang detected" problems with our 10 GbE NC523SFP iSCSI NICs in the past and replaced them with Emulex NICs (VMware vSphere 4.1).

 

Yesterday I installed the latest Qlogic nx_nic driver for vSphere 4.1 and today one onboard NC375i NIC had the same error as the NC523SFP cards. The Management Network was disconnected for a couple of seconds.

 

Mar 29 08:53:55 sdev2003 vmkernel: 0:00:40:37.460 cpu15:4519)<3>nx_nic[vmnic0]: Firmware hang detected. Severity code=0 Peg number=0 Error code=c9 Return address=0

[...]

Mar 29 08:53:55 sdev2003 vmkernel: 0:00:40:37.598 cpu15:4519)<3>nx_nic[vmnic0]: Load stored FW

 

~ # ethtool -i vmnic0
driver: nx_nic
version: 4.0.602
firmware-version: 4.0.579
bus-info: 0000:04:00.0

 


Looking at the search results and the Advisory (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02964542&lang=en&cc=us&taskId=130&prodSeriesId=4118472&prodTypeId=12169) it seems that this is a ongoing problem.

 

Does anyone know if this actually is a hardware problem that can never be solved  by driveror firmware updates? This is a massive problem. Especially because our DL580's have 4 onboard Qlogic NICs + a Quad Qlogic Card (+ the replaced 10 GbE dual port card).

 

 

 

14 REPLIES
Dave73
Regular Advisor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Hardware problem on the 375i and T nics.

 

Ill try get more details for you on Monday as my collugue is on leave who is working with it.

pirx4711
Frequent Advisor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Thanks. Seems that all Qlogic NICs have more or less hardware problems...

pirx4711
Frequent Advisor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

After collecting core dumps with a tool that HP provided, our NC375T cards and the SPI boards with the onboard NC375i will be replaced with cards that have a new revision of the chip/hardware.

gbaccus
Visitor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs


pirx4711 wrote:

After collecting core dumps with a tool that HP provided, our NC375T cards and the SPI boards with the onboard NC375i will be replaced with cards that have a new revision of the chip/hardware.


Hey there pirx,

 

If you don't mind me asking, can you provide any details about the replacement (firmware version, driver version, etc)? Like you, we are at the latest available versions, and have seen the card reset twice on one DB cluster node, and three times on the other. We put in an additional NIC and teamed across it, so that when it happens now, we at least don't lose the entire cluster!  :-)

 

Any more info you could provide would be much appreciated. We are putting our second HP ticket in on this issue.

 

Regards,

 

 

   Grant

pirx4711
Frequent Advisor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

The new NC375T cards are Rev. 0G, the old ones were 0F. The SPI Board is 0B instead of 0A. I've no idee what details have changed.

gbaccus
Visitor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Thanks, Pirx. We are also getting 4 new SPI boards rev 0B for 4 DL580s we have, currently at rev 0A.

 

 

   Grant

   MTS

RHWebster
Occasional Visitor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Its a firmware problem. You need firmware version 4.0579. The file name is CP015900.scexe (64 bit version). You need to do an off line upgrade of the firmware from the system console because the NIC's will restart. You could use ILO remote console. More instructions here:
http://h30499.www3.hp.com/t5/ProLiant-Servers-ML-DL-SL/Embedded-NC375i-firmware-problems/m-p/5603853/highlight/false#M128063
pirx4711
Frequent Advisor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

In our case it was a hardware problem. We replaced all NICs with new boards (rev. 0G for the NC375T and rev. 0B for the NC375i SPI board) and haven't had a single hang since then.

RHWebster
Occasional Visitor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

That's interesting. HP have, to be fair to them, embarked on a process of replacing all our SPI boards (16) with REV B edition and all our NC 375T cards. However, there are a group of DL580G7's that have had the firmware upgrade but not the SPI replaced and have not had a hang since. I wonder if there was a bunch of boards that got bad firmware and a faulty chipset. Also, i note one poster who mentioned that his servers lost network conectivity during high load - I've got two hosts sitting in MM ( ESX servers) that regularly have the firmware hang.

 

Cheers

 

R

VSS
Occasional Visitor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Hi pirx4711,

is this tool mentioned public available or how do got it? On one of DL585G7 the replacement of memory boards seems to solve the problem and now I would like to check other hosts potentially affected.

 

Kind regards

seefriedm
Occasional Visitor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Our servers are running as VMware 4.0 and 4.1 Hosts. We have had Hangs and Guests powering off with no information in the Log Bundles as to we.

 

HP has an Advisory to resoleve these on the 375i cards.

Version 4.0.602

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02964542

 

After this patch our issues go away as far as network causing this is concerned.

chr3
Occasional Contributor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Hi,

we've updated the firmware and the vmware driver for th nc375i, vsphere esx 4.1 U3 and still have firmware resets with following nic disconnects. Are there any new advisories/ conclusions from HP?

 

regards

Chr3

pirx4711
Frequent Advisor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

HP replaced our NC375T NICs and the SPI boards. No firmware/driver update will solve this problem.

Dave73
Regular Advisor

Re: ongoing DL580G7 Qlogic NIC Firmware Hangs

Contact HP, i am under the impression they will replace your cards.