Network Interface Issue with HPE Synergy Blades

[robocp01]

We are having a very wierd issue and I have tried just about everthing to get this working.  I have an open case with HPE on this issue, but not getting much satisfaction from them.

Here is a synopsis of the issue.  We have some new and even existing Synergy Gen 10 servers which refuse to connect to the network.  Interfaces show as cable disconnected.  The previsouly existing servers were working before but have now stopped.  The newer servers we are trying to image and the interfaces refuse to connect.  

What is more confusing is a situation like this below:

More info on this.  I installed 5 new Synergy servers into Stacey Frame 4, Bays 1 – 5.  The servers in Bays 1 – 4 appear to all have network connectivity – I’m laying down an image on each one right now.  However, the one in Bay 5 (HITECMDPA1) won’t allow me to image b/c it doesn’t have network connectivity.  The ones in Bay 2 – 5 all were built using the same profile template, so should be the same. 

I haven’t had a chance to play with this one very much yet – but this is very strange behavior.  Maybe there’s some sort of hardware issue on some of these blades – not sure.

We have this occuring on two separate 2 frame Synergy systems in different buildings and the same symptoms are occuring.  We are right now testing these frames with new connectivity on the uplinks to new Nexus switches set up in redundant format on the uplinks.  I have my networking team investigating the issue as well.  

Some of the things we have tried,

Moving to a new enclousre slot.

Removing profiles and recreating them

Force removing profiles and recreating them.  

Nothing seems to work.

[ChrisLynch]

Can you go to the Interconnect inventory view, and examine the downlink ports?  Do they show "Linked" as their status for the servers in question.

[robocp01]

The downliink ports show as unlinked.  With this message.

The connection on downlik Port 12, subport a has failed The subport is unlinked (NWCN_033) Active

If the server Frame02-2SM0240005, bay 12 is power cycling or powered off, connectivity alerts may occur when the network adapter is either disconnected or negotiates connectivity with the interconnect. These alerts can either be ignored or manually cleared and will normally clear automatically. If the server is booted and running an operating system, the alerts indicate a loss of connectivity between the network adapter and the interconnect. Verify that the CNA supports the current interconnect downlink speed. Verify the configuration of the operating system, as well as the health and link status of the downlink ports on which the connection depends. Perform refresh of logical interconnect if this connection is associated with a Virtual Connect SE 100Gb F32 module or a Virtual Conn

Refresh of interconnects does not solve the issue.  

[ChrisLynch]

And the server is powered on, correct?  Can you private message me your case ID?

[robocp01]

Yes we have tried virtually everything i have gone over the configuration, relocated the servers, removed profiles, rebooted numerous times, checked MAC addresses,  looked at Firmware levesl, looked at NIC drives.  One thing that is interesting is using Intelligent provisioning, which we don't use but use MDT instead, you can go into the maintenance section and run an iventory.  That inventory at first says, no nics available, but when you allow it run in the network section you can see the interfaces defined.  If I had or subtract a network you can also see those networks come and go.  I will need to find out how to PM you the case numbe.r.

Another interesting fact is that problem has surfaced on two seperate Synergy systems in two building.  We from the server team made no changes, but we have hooked 2 frames out of 4 to a new 100 gig network, using Cisco nexus and I beleive it's MGLA technology.  Our network team is seperate from our server support team so we don't touch that aspect of it.  Fortunitely we are only in a testing phase right now with 100 gig connections, working toward transition to production.  I am waiting our main network guy to get back from vacation to see if he made any changes, his change control shows nothing being done.  

Thanks for your help.

[robocp01 · Accepted solution]

Ok, so HP identified the issue, turns out ILO 5 firmware version 2.41 is buggy.  Problem is we have servers that are running the same version that are working and others that are not.  So solution was to upgrade to newer ILO 5.0 firmware.

Here is what resolved it. 

As per the EA the fix for this issue is as given below:

1.Upgrade or downgrade the HPE Integrated Lights-Out 5 (iLO 5) firmware to a supported Synergy iLO 5 version which is ILO5 to 2.33 or ILO5 2.44

2.After performing the iLO 5 firmware upgrade or downgrade, unassign the Synergy OneView Server Profile from the server, and then reassign it back to the server

3.After reassigning the Synergy OneView Server Profile, boot the server and test connectivity

Thanks to those that provided me with help.  Really appreciated.

[ChrisLynch]

Thanks for the update, and glad it is working.  I will let the OneView R&D folks I had looking at your case know.

[ChrisLynch]

BTW, what iLO5 firmware version did you, or you are planning to update to?

[robocp01]

2.44 ILO version is now loaded.  

[ChrisLynch]

Thanks.