HP Proliant DL380 G5 NIC's not found after firmware update

[Saturas]

I tried to update firmware server NIC's.

I downloaded the file for Linux x86_64 which was named  CP018942.scexe and ran it.

[root@host ~]# ./CP018942.scexe
Copyright (c) 2013 Hewlett-Packard Development Company, L.P.
HP Broadcom Online Firmware Upgrade Utility for Linux x86_64


Found HP NC373i Multifunction Gigabit Server Adapter MAC: 001A4BEB977C
*** WARNING *** - Installed Boot Code is the same version as selected Boot Code.
Update Boot Code 1.9.6 to 1.9.6 -  y/n/q (n):y
*** WARNING *** - Installed UMP is the same version as selected UMP.
Update UMP 1.1.8 to 1.1.8 -  y/n/q (n):y
Update CCM to 7.8.21 -  y/n/q (y):y

The Firmware Upgrade will take several minutes.  Please be patient.

Selecting HP NC373i Multifunction Gigabit Server Adapter MAC: 001A4BEB977C
Upgrading Boot Code 1.9.6 to 1.9.6


*******************************************************************************
Broadcom NetXtremeII Firmware Upgrade Utility for Linux v2.6.4
*******************************************************************************

C Brd      MAC      Drv                     Name
- ---- ------------ --- ------------------------------------------------------
0 164C 001A4BEB977C Yes NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth0)
1 164C 001A4BEB977A No  NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth1)
Programming from: 0x00000600 to 0x0000299c
BC Version: b8c1 1.9.6 (0x01090605)
update bootstrap: start=600; byte_cnt=9120; crc=0x9f2c35c
Programming from: 0x000029a0 to 0x000062e0
Programming from: 0x000062e4 to 0x000066e8
Quitting program ...
Program Exit Code: (0)
Upgrading UMP 1.1.8 to 1.1.8


*******************************************************************************
Broadcom NetXtremeII Firmware Upgrade Utility for Linux v2.6.4
*******************************************************************************

C Brd      MAC      Drv                     Name
- ---- ------------ --- ------------------------------------------------------
0 164C 001A4BEB977C Yes NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth0)
1 164C 001A4BEB977A No  NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth1)
Programming from: 0x000066ec to 0x0000ab44
Quitting program ...
Program Exit Code: (0)
Upgrading CCM to 7.8.21


*******************************************************************************
Broadcom NetXtremeII Firmware Upgrade Utility for Linux v2.6.4
*******************************************************************************

C Brd      MAC      Drv                     Name
- ---- ------------ --- ------------------------------------------------------
0 164C 001A4BEB977C Yes NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth0)
1 164C 001A4BEB977A No  NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth1)
A NVRAM directory entry was not found.
Quitting program ...
Program Exit Code: (30)
ERROR: Upgrade of HP NC373i Multifunction Gigabit Server Adapter MAC: 001A4BEB977C failed.


*******************************************************************************
Broadcom NetXtremeII Firmware Upgrade Utility for Linux v2.6.4
*******************************************************************************

C Brd      MAC      Drv                     Name
- ---- ------------ --- ------------------------------------------------------
0 164C 001A4BEB977C Yes NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth0)
1 164C 001A4BEB977A No  NC373i Integrated Multifunction Gigabit Server Adapter rev 12 (eth1)
Programming from: 0x0001c268 to 0x0002b210

After that NIC's do not work and do not seen by the server.

How can I repair this?

[Saturas]

I turned on the S6 system maintenance switch and power on the server.
When the system was booted, I turn off the server, turned off the S6 switch and turned on the system power again.
But it did not help.
The only thing that I have found -  is that the Broadcom NetXtreme Ethernet Boot Agent setup menu is available again.

Some screenshots in attachments.

[Saturas · Accepted solution]

I completly solved the problem by use xdiag utility from Broadcom.

I created bootable CD based on Freedos with xdiag and firmwares and repair NVRAM of both server NICs.

[csvarasdy]

would you share please, how to fix it?

i've already tried xdiag, but with no luck. It said:

Reason:400::BootROM cannot load bootcode (0xd)

[Saturas]

That's right. You repeat my quest. At first, I also used the uxdiag, but it is for user level only.

Then I found a xdiag utility (which has the engineering mode xdiag -b06eng).

After that, I made the ISO image based on Freedos and followed the instructionsof the setup.txt file:

Programming Non-volatile Memory for New NIC/LOM
===============================================

Perform the steps below only with a board that has nothing in the flash
memory. Otherwise, your existing configuration will be erased. 

Bootcode installation setup (A user needs to familiarize oneself with
some basic DOS commands):                                                                                                     
                                                                                                                              
0. Once booted to DOS, change directory to where the diagnostic tool                                                          
   resides. (Use "cd" command).                                                                                               
1. Start the diagnostics in engineering mode (Type "xdiag -b06eng").                                                          
2. Make sure the first 1.5 kB of flash content are zeroes. Type                                                               
   "nvm fill 0 0x600 0".                                                                                                      
3. Upgrade the boocode: Type "nvm upgrade -bc <filename>" at the                                                              
   prompt.                                                                                                                    
      - The <filename> can include directory path in which the file is                                                        
        found. Contrary to the DOS convention, make sure to use forward                                                       
        slashes for path separators.
      - For 5716 chip, please use 5709C boot code.
4. Configure device with the command "nvm cfg".
   - At the new prompt, type "default" to restore default configuration.
   - Then, at the next prompt, type "16=10" to select the 32M BAR size.
     However, skip if you have 5716.
   - Then, at the next prompt, type "1=00:10:18:xx:xx:xx" to program 
     the primary MAC address for that particular board.
   - Then, at the next prompt, type "33=00:10:18:yy:yy:yy" to program 
     the iSCSI MAC address for that particular board. This step is not
     necessary for 5716 chip.
   - Then, type "save" to exit "nvm cfg" command.
5. Repeat step 4 on the secondary device if you have 5709/16.
6. Power cycle (not just a reboot) the system and then restart xdiag 
   in engineering mode.
7. Verify the BAR setting.
   - Type "nvm cfg".
   - Check entry 16 to make sure it indicates the size of 32M for
     5706/08/09. The size is 256k for 5716.
   - Check the same entry 16 setting for the secondary device if you
     have 5709/16.
   - Type "cancel" to exit.
8. Terminate the xdiag session by typing "exit".

[Saturas]

NICs of my ProLiant DL380 G5 - BMC5708.

I used following steps:

1.     Boot from CD
2.     nvm fill 0 0x600 0
3.     nvm upgrade -bc c:/nx2/firmware/bootcode/5708/bc08c740.bin
4.     nvm cfg (and setup BAR size, MAC and iSCSI MAC)
5.     device 2
6.     repeat steps from 2 to 4 for device 2
7.     power off the server (unplug cable) and power on again
8.     PROFIT!

[chris33]

Unfortunately I now find myself dealing with this issue as well.

Can you please post a download link where you found the xdiag iso file that contains the xdiag utility with the engineering mode? 

I've been searching the broadcom Netextreme II site and I'm not finding it.  I havent' tried older downloads yet.

That would be greatly appreciated.

[jnz3]

Thanks a lot for sharing your knowleadge!

Works just fine.

[Ernest Franzen]

Bummer, we just bricked 3 DL380 G5's using the SPP dated 2/2014. It doesn't help that they are no longer under warranty.

We were proactivily updating the firmware on 20 servers and the onboard NICs are no longer working on three of them.

We replace the System Board, verified the NICs were ok, then applied the SPP Firmware package again and the NICs are now showing disabled in the System BIOS with no way to enable them.

Booting to Ctrl-B brings up the NetExtreme Setup and the data showing within the fields are scrambled.

We are now using the SPP from 9/2013 and no longer having issues.

So, now I want to fix these System Boards with the messed up NICs? I was going to use these instructions, but where did you source that bootcode file bc08c740.bin from? A quick Google search brings up 3 sites, one is this one and other 2 look scary for downloading from.

Is this bootcode file the correct one for the 373i NICs?

Thanks

[KLL-IT]

Hi we have just the same issue with SPP 2-2014 and 2 of our DL385 G5 ESXi-hosts.

After one full automatic update on the first server we bricked our first two onboard NICs (NC373i). After a manual update on the second server, we bricked another NIC (NC373i).

On the second server we used SPP 2-2014 at interactive mode and excluded the NC373i's. After the system came up again, we updated the NICs with „CP021849.scexe“  under ESXi 5.1. After that they are also gone.

As Ernest wrote, we also had scrambled fields at NetXtreme Setup and at RBSU only iSCSI-MACs are displayed - no "normal" NIC-MACs (disabled).

We opend a support request at HP and our Reseller.

Let's see what happens.

[SamedKS]

Hi,

anyone help me to download xdiag which contains engineering mode (xdiag-b06eng) ?

Regards

[HPusers]

Same Problem with SPP 02.2014 and DL380G5..... 

[scsbradley]

Hi Satus,

Do you have an image you can share with me?  I am having trouble sourcing all the necessary files.

[ServerUP]

Anyone have any luck getting a solution from HP?  It would appear it's a fault of theirs given the firmware file they provided is causing these issues. 

I'm struggling with the proper XDIAG files needed as well.

[Letze01]

Hello,

at least it seems I was able to figure this issue out, based on Saturas's information. Here the steps I made:

1. Download all necessary tools

- download FreeDOS http://www.freedos.org/download/download/fd11src.iso

- download XDIAG.exe ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/diag/NX2/xdiag.exe

- download bc08c740.bin ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/onchip_firmware/bootcode/5708/bc08c740.bin

- read all informations in ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/onchip_firmware/bootcode/5708/setup.txt

2. prepare the FreeDOS.iso

- after downloading open the iso with a tool like UltraISO

- add the XDIAG.exe and the bc08c740.bin to the iso

- save the iso with a new name

- burn it or mount it with ilo

3. boot from FreeDOS

- select 'Install to harddisk' (in my case nothing was written to the harddisk, because FreeDOS was unable to read the existing NTFS volume)

- press '1'

- select your language

- press ESC

- select 'run FreeDOS from CD-ROM'

4. Programming Non-volatile Memory for New NIC/LOM (follow the steps in the setup.txt)

5. Restoring corrupted NIC/LOM (follow the steps in the setup.txt)

6. reset system settings

- turn on the dip switch S6 for system maintenance and power on the server

- boot until the message for resetting the dip switch appears

- power off the system and switch S6 back to normal

Please keep in  mind, that this informations are provided without any warranty. As I said, these steps worked for me with a DL380 G5.

Hope this helps someone solving this issue.

Regards

Andreas

[ServerUP]

Looks like the NIC's are back up and running.  Thanks for all the help from everyone. 

[hase3d]

Hello all, and thanx for the great tips here.. I got the same problem as you all did and solved it with the help of both Saturas and Letze01 posts.

NICs of my ProLiant DL380 G5 - BMC5708c.

I used the following steps 

1. Download all necessary tools

- download FreeDOS http://www.freedos.org/download/download/fd11src.iso

- download XDIAG.exe ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/diag/NX2/xdiag.exe

- download bc08c740.bin ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/onchip_firmware/bootcode/5708/bc08...

- read all informations in ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/onchip_firmware/bootcode/5708/setu...

2. prepare the FreeDOS.iso

- after downloading open the iso with a tool like UltraISO

- add the XDIAG.exe and the bc08c740.bin to the iso

- save the iso with a new name

- burn it or mount it with ilo 

3. Boot from CD   

4. Run xdiag in engineering mode xdiag -b06eng

5. type device 1

6. nvm fill 0 0x600 0

7. nvm upgrade -bc "path to bin file/bc08c740.bin

8. nvm cfg and typed "default", then typed 16=10 wich sets the BAR size to 32 for this NIC (see the guide), then typed SAVE

9. device 2  and repeat steps 6-8, run the command 1=00:00:18:xx:xx:xx <--- change the last digit for different mac on device 2. 

10. SAVE

11. EXIT

12. do a full POWER CYCLE (unplug) of the server, run xdiag in eng mode again and confirm the that the MAC and RAM size are still changed. 

13. DONE

up and running again. 

didnt need to do the steps for restoring corrupt lom or change the S6 dip switch to get it to work. 

tried this on 2 of the servers and it worked on both. 

Br

Hase3d

[Marko_MSI]

Thans to Saturas and Letze01 posts.

I made the same procedure as hase3d did and it WORKS!

The only problem is that these NICs are not detected as HP 373i embedded any more (now the manufacturer is Broadcom).  If some of you want the ISO including all the utilities for succesful upgrade just make a reply to this post.

Regards,

Marko

[stricklandj]

I would appreciate the ISO. I have been on the phone with HP for 1.5 hours so far and they can't find a solution. I have referenced this post but they don't care what ya'll have to offer.

[Marko_MSI]

http://we.tl/v9GQstMkhK

and use hase3d's steps.

regards,

Marko

[l_a_r]

One can use the "nvm cfg" command to manually set parameters to match those of the HP NC373i.

The attached file shows the "nvm cfg" settings on one of my "good" NC373i ports.

[Marko_MSI]

Great idea l_a_r. Thank's for help.

Regards,

[KLL-IT]

Another way to get the original settings any namings of the NICs is to dump the NVRAM of a working NIC and program it into a broken/reanimated one. I restored 3 broken NICs in our enviroment last week.

Attention:

I always reanimated the NICs with the correct bootcode and entered the MAC-addresses first, before I restore a dump !

If you have a ESX(i) running on a server with a broken NIC, you'll find the MAC-addresses in /etc/vmware/esx.conf.

e.g. on DL385 G5:

NIC1 is vmnic0 with MAC=00:1f:xx:xx:e8:52

-> NIC1 iSCSI       MAC=00:1f:xx:xx:e8:53 (+1)

-> NIC2             MAC=00:1f:xx:xx:e8:50 (-2 of NIC1) – in my case vmnic5

-> NIC2 iSCSI       MAC=00:1f:xx:xx:e8:51 (+1 of NIC2)

Create a NVRAM-dump:

1. xdiag –b06eng
2. Select working device with device x (nr. of device)
3. device 1
4. nvm dump filename.nvr
5. exit

Restoe a dump, e.g. on NIC2 of the same server:

1. xdiag –b06eng
2. Select reanimated device with device x (nr. of device)
3. nvm prg 0 filename.nvr
4. nvm cfg (check if all MAC-Addressen fit)
5. Press [ANYKEY] to check all parameters
6. enter save [ENTER]
7. exit
8. power down server, wait and reboot

All NICs should show the correct MAC-Addresses in RBU and e.g. vSphere on the vSwitches as  "HP NC373i Multifunction Gigabit Server Adapter" (in my case).

You'll find a complete dokumentation for xDiag.exe under ftp://www.supermicro.com.tw/driver/Broadcom/B57BCMCD_T7.6.4.1/dos/diag/NX2/EngrManual.pdf .

I also have documented a complete "reanimation"-procedure, but only in German. If anyone want it, I will post it here.

BTW -Here's the reason, why the NVRAM was broken during Update-process on our DL385 G5s:

In CP021069.scexe on SPP-2014-02 there's the CCM V7.8.21 (Broadcom Comprehensive Configuration Management) included. If you flash that firmware onto a BCM5708c (NC373i), you will brick it.

I've tested it with CP021849.scexe under vSphere ESXi 5.1. As long, as you only flash the firmwares without the CCM component, it will work.

Maybe a hint for everyone .... and HP.

BR

Florian

[taktosan]

Hi!

I also have this issue on a DL 380 G5 after using SPP 022014. After reanimating the NISs with the procedure described obove they are usable under Windows Server 2008 R2 but only with the standard Microsoft drivers. The naming is wrong ( now Broadcom ) and the original Broadcom driver won't fit.

Could you please send me the german document for complete reanimating these NICs.

Thanks in advance.

[GR89]

Could you please upload a NVRAM dump of a working NC373i ?

As I only own one DL380 G5 (which I was preparing for studying purposes) with both NICs bricked by the SPP 02/2014...

Thanks, would be much appreciated!