Lost all networking on HP Blade Servers.

[Eryk]

I have an HP C7000 enclosure with 8 blades (BL406c Gen 7) in it.  I updated one of the blades to  VMware ESX 5.5 U3 along with other patches. This blade will no longer connect to vCenter and I have found that this blade will no longer ping out to the network also.

 I am no longer able to ping my own gateway but I can ping loopback. I backed out of the updates and went to an earlier version of ESXi with no help either. I even attached a Linux Live CD to this blade and it says its connected to my VirtualConnect Fiber backend but I still am not able to ping my gateway or get anywhere on the network. 

Does anyone know why this would happen? I tested this issue on another blade to see if that blade was dying and this happened to both blades so now I have two of them down. Please help with any advice you may have.

[Robert_Jewell]

I am not sure about the ESX update, but did you check the status of the ports and connections from within Virtual Connect Manager?  Look to be sure that the networks are online and active and that the network settings match what you have in ESX.

-Bob

[Eryk]

I have checked all VC server profiles and Fiber connections and they report "OK" and connected. Even the Live Linux CD shows connected to the 2 10 Gig fiber adapters.

Everything shows as good but I am still disconnected.

[Robert_Jewell]

Perhaps check out these advisories:

http://h20565.www2.hpe.com/portal/site/hpsc/public/psi/advisoriesResults/?sp4ts.oid=5288507

If nothing, perhaps reset the VC modules (as in using the OA to reset the hardware modules)?

-Bob

[Eryk]

If there was a problem with the VC modules, wouldnt the other blades go down as well? I have 8 blades and this is only happeneing on the ones I upgraded.

[Robert_Jewell]

Yup, I agree there.  I didnt realize others were still working OK.  What you can try then, is to power off the affected servers, remove then reapply the VC profile to each blade slot.  Power on and try again.

-Bob

[Eryk · Accepted solution]

I should note that FINALLY I have a fix for this. I despise when people don't post the fix and just let the thread go unanswered so here it goes:

I noticed my blades, enclosure and ILO 3  firmware was low (2010 versions).

These are the thing I tried:

• Switch to different enclosure bays
• Re IP and test default gateways over and over
• Reinstall all different versions of ESXi to test functionality
• Check the Virtual connect Fiber Blades for issues

I noticed my blades, enclosure and ILO 3  firmware was low (2010 versions).

 I was able to get a hold of an HP SPP CD ( HP Service Pack for Proliant Server) which had all the 

Firmware, Updates, and Revisions need for that time period which brought my equipment Firmware to 2014. I retried the install of ESX 5.5 U3 and everything is functioning just fine!!!

[DanRobinson]

You should refer to http://vibsdepot.hpe.com in the future for the "Recipes"

There are known issues like this if you run a mismatched firmware and driver.

Most likely you updated the driver when you used the new ISO/ZIP to upgrade your software on your VMware Hosts, but did not update the firmware.

Then when you updated the firmware, things started to match again.

Around page 3-4 in each Recipe will be the SPP the recipe is based on and then shrtly after that a table that calls out which HPE Customized VMware Images align to that Recipe as well.

[Eryk]

Thank you for mentioning this. I also wanted to repost and add something I found that does coincide with checking ALL firmware version compatibility. After upgrading the blade with the SPP ISO I put them back into the cluster but noticed we had VM's that were on a network (VM Network VLAN ID = 0) that would not pass traffic still, but other VLAN ID networks (i.e. 10,20 ,etc) we had on the ESX host worked just fine.

I did notice that the ILO version Firmware errored out when I flashed everything else but I though nothing of it. Turns out that affected that VM Network for some reason. I managed to upgrade the ILO firmware (using older versions until I could upgrade to the latest) and all the ESX networks pass traffic. 

Hopes this helps someone in the same position!!