HP Proliant DL380 G5 NIC's not found after firmware update

[l_a_r]

One can use the "nvm cfg" command to manually set parameters to match those of the HP NC373i.

The attached file shows the "nvm cfg" settings on one of my "good" NC373i ports.

[Marko_MSI]

Great idea l_a_r. Thank's for help.

Regards,

[KLL-IT]

Another way to get the original settings any namings of the NICs is to dump the NVRAM of a working NIC and program it into a broken/reanimated one. I restored 3 broken NICs in our enviroment last week.

Attention:

I always reanimated the NICs with the correct bootcode and entered the MAC-addresses first, before I restore a dump !

If you have a ESX(i) running on a server with a broken NIC, you'll find the MAC-addresses in /etc/vmware/esx.conf.

e.g. on DL385 G5:

NIC1 is vmnic0 with MAC=00:1f:xx:xx:e8:52

-> NIC1 iSCSI       MAC=00:1f:xx:xx:e8:53 (+1)

-> NIC2             MAC=00:1f:xx:xx:e8:50 (-2 of NIC1) – in my case vmnic5

-> NIC2 iSCSI       MAC=00:1f:xx:xx:e8:51 (+1 of NIC2)

Create a NVRAM-dump:

1. xdiag –b06eng
2. Select working device with device x (nr. of device)
3. device 1
4. nvm dump filename.nvr
5. exit

Restoe a dump, e.g. on NIC2 of the same server:

1. xdiag –b06eng
2. Select reanimated device with device x (nr. of device)
3. nvm prg 0 filename.nvr
4. nvm cfg (check if all MAC-Addressen fit)
5. Press [ANYKEY] to check all parameters
6. enter save [ENTER]
7. exit
8. power down server, wait and reboot

All NICs should show the correct MAC-Addresses in RBU and e.g. vSphere on the vSwitches as  "HP NC373i Multifunction Gigabit Server Adapter" (in my case).

You'll find a complete dokumentation for xDiag.exe under ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/diag/NX2/EngrManual.pdf .

I also have documented a complete "reanimation"-procedure, but only in German. If anyone want it, I will post it here.

BTW -Here's the reason, why the NVRAM was broken during Update-process on our DL385 G5s:

In CP021069.scexe on SPP-2014-02 there's the CCM V7.8.21 (Broadcom Comprehensive Configuration Management) included. If you flash that firmware onto a BCM5708c (NC373i), you will brick it.

I've tested it with CP021849.scexe under vSphere ESXi 5.1. As long, as you only flash the firmwares without the CCM component, it will work.

Maybe a hint for everyone .... and HP.

BR

Florian

[taktosan]

Hi!

I also have this issue on a DL 380 G5 after using SPP 022014. After reanimating the NISs with the procedure described obove they are usable under Windows Server 2008 R2 but only with the standard Microsoft drivers. The naming is wrong ( now Broadcom ) and the original Broadcom driver won't fit.

Could you please send me the german document for complete reanimating these NICs.

Thanks in advance.

[GR89]

Could you please upload a NVRAM dump of a working NC373i ?

As I only own one DL380 G5 (which I was preparing for studying purposes) with both NICs bricked by the SPP 02/2014...

Thanks, would be much appreciated!

[Gerardo_Arceri]

I get "File image (of type 100) not for SERDES." after trying nvm upgrade -bc XXXXXXX.bin

[AndrewSmith1066]

I think you are making the same mistake I was until my boss spotted it. The short answer is try the 

bc08s740.bin

in

ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/onchip_firmware/bootcode/5708/

After following the same procedure I have network connectivity again.

Hope this helps,

A

[Gerardo_Arceri]

I was able to restore the BL460c G1 Nics (NC373i) back to working conditions by using the file indicated by AndrewSmith1066 to get rid of the SERDES error, except that i extracted the file from the official HP windows firmware package included in the NICFW directory of SPP2013.09 (cp018055.exe)

I'm including the bootcode firmware file i used along a Text Dump of the "nvm cfg" values for a working NC373i NIC.

BTW, for some reasons BL680c G5 servers which include this same NIC model are not affected by CP021069 upgrade.

Regards, Gerardo

[LanairAdmin]

Let me prefix this by saying: HOLY CRAP;  Thank you everyone!  In particular, My final solution was reached by following hase3d's guide

I was working from a mac so burning a bootable iso of freedos with the changes prooved challenging as I usually use ultraiso for similar things to this on PC, but my workaround was to simply burn two discs, one was a straight burn of the image of freedos, and another of freedos's contents combined with the xdiag and bc08c740.bin files, I swapped the disks once freedos was fully loaded.

I took a picture of the mac addresses of my cards as they were presented with the xdiag -b06eng command with intent to restore the same addresses when I was finished in hopes it preserved all functionality in vmware esxi which was running on this server (it seemed to work as I haven't had to make any changes in esxi after booting), but I noticed the second nic had the same mac as the first nic... so I changed the second nic by one digit (which I wasn't using in esxi anyhow and didn't need) now I am wondering if that is where the original conflict was coming from all together, I wonder if the second nic's mac was just messed up, and am curious what would happen if I had not loaded a new bin, but instead changed only the mac addresses via the nvm cfg command and rebooted.

All of that aside... IT WORKED!

I can't believe it, but IT WORKED!

I would have NEVER EVER EVER figured this out on my own, so thank you so much for going to the trouble of creating a guide for this, working download links and all!

NOTE: My server was a DL385 G5, so it probably has slightly different hardware, but the .bin link you gave worked just fine for me.  My original symptoms occured almost exactly the same way.  I updated firmware via the old style hp firmware disks, everything went fine.  Tried to upgrade to the 2013 disk, and it wouldn't let me, so Went down to the 2012 SPP disk, and it worked fine (I didn't try loading up esxi at this point, so I don't know if the cards were working here or not, but it booted just fine).  Then I loaded a 2014 disk to apply my last patches and the latest updates (it needed a firmware update from 2012 first to resolve the issue I encountered in the 2013 disk), suddenly the server would not boot!  I ended up resetting the rom by opening the server and flipping switches 1, 5, and 6, as described in another tutorial I found on another site - thankfully the server booted and everything looked fine.  Unfortunately I was unaware that the network cards were not working, and to resolve the issue of them not loading in vmware ESXI, I had to burn cd's with vmware driver updates (I assumed that the cards were now too "new" for esxi 5.1 to reccognize and I needed driver updates... this assumption was wrong) to no avail.  Finally, I gave up and found this forum post and followed the instructions to resolve my issue.  Also, in hopes some ESXI users stumble in here, I was getting " vmfs3 failed to load " errors as well as " no compatible network adapters found " errors, and despite what it may lead you to believe, my network cards were actually being seen as "removed".

THANK YOU!  (sorry this is a wall of text, wife waiting for me to meet her for dinner, I just really felt the need to reply and say thanks!)

[Michael Leu]

FYI here is the official HP advisory c04258304 for this issue:

https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c04258304

Advisory: CUSTOMER ATTENTION REQUIRED: HP Broadcom-Based Network Adapters - Updating Comprehensive Configuration Management Firmware Version 7.8.21 on Certain HP Broadcom-Based Network Adapters May Result in the Network Adapter becoming Nonfunctional
On certain HP ProLiant servers, certain HP Broadcom-Based Network adapters listed in the Scope may become non-functional when they are updated with the Comprehensive Configuration Management (CCM) firmware Version 7.8.21 using firmware smart component, HP Smart Update Manager (HP SUM) or the HP Service Pack for ProLiant (HP SPP) 2014.2.0.

[Moderator edit: Updated the link.]