Blade Flex NICs not mapping

[Brian Mullin]

Hello All,

I'm having a problem which is preventing sleep for a week now , where 3 blades'  FLex NICs are permanently disconnected, can pointers rrreally gratefully recieved, sorry if it's long winded and i'm probably missing detail.

Background:

2 sites, c7000 on each . independent, no stretched clusters. kind of mirrored environments.

8 blades in each enclosure, 5 x BL660c G8   ( Windows 2008 R2, 2-node oracle cluster and 3-node sql cluster )

working fine . managment vLAN plus 2 others

and 3 BL465c Gen8 in each, installed ESXi 5.1 .

The 3 in one site 'a' are fine, connected to vLANS ok .

problem

But the 3 in site 'b'  were ok until the only thing i can think of that caused the problem ; I tried to add three more  vLAN networks along with the existing vLANs at the ''Ethernet Adapters'  Section and added them to the server profiles,  then brought the BL465c servers up and they henceforth have showed the Flex NICs 'disconnected' .log shows

Date1513:38:39+01:00 VCEX23C430401LC vcmd: [PRO:Profile_ESX1:6041:Warning] One or more profile connections are not mapped to the server ports. : At least 1 connection OK but not all connections OK, Previous: Enet Network state OK

DateT13:38:40+01:00 VCEX23C430401LC vcmd: [SVR:Profile_ESX1:5051:Warning] There are more Ethernet connections defined for the profile than exist on the server blade.

 I brought down the servers again, removed the vLANs ,  but now it doesn't matter what I do , they stay that way, the network doesn't reach the nics.  The HBAs are fine on all .  I fear i've corrupted something somewhere .

 Even if i bring one of them up without a profile, the 2 physical FLex NICs are hanginig there, disconnected

More details :

OA firmware version 3.60

VC firmware version 3.7

SPP run on blades  : HP_Service_Pack_for_Proliant_2013.09.0-0_744345 001_spp_2013.09.0-SPP2013090.2013_0830.30.iso 

vSphere 5.1 Update 2

HP Custom Image for ESXi 5.1.0   Update 2 (17 June 2014)

hp-build 5.66.28-434156

No driver/software updates   required. Image contains driver versions specified in this Recipe

 - VMware-ESXi-5.1.0-Update2-1483097-HP-5.66.28-Jun2014.iso

 - HP FlexFabric 10Gb 2-port 554FLB Adapter ,  installed 2013.09.0\HP Firmware Flash for Emulex Fibre Channel Host Bus and Converged Network Adapters – Lunix (x86_64) (CP019369.scexe)\active\available FW 4.6.247.5 driver be2net

Among (!) actions tried

-Vcutil showall showed no problem

 - Brought Servers down

 - Unassign profiles

 - Unassigned vLANxxx

 - Deleted vLANxxx

 - Wizard, new profiles for each blade ESX1,2,3

 - -Profile 3 , add network VLANyyy (working with the other blades ok )

 - Assigned to bay for ESX 3

 - Bring up

- NICs still disconnected

these may be transitory messages while I was upping and downing ...

2014-07-24T14:06:38+01:00 VCEX23C430401LC vcmd: [SVR:enc0:dev8:5012:Critical] Server state FAILED : Component is not operational due to an error, Previous: Server state OK

2014-07-24T14:06:38+01:00 VCEX23C430401LC vcmd: [ENC:enc0:2014:Minor] Enclosure state DEGRADED : Some Enet modules & servers not OK, Previous: Enclosure state OK

2014-07-24T14:06:38+01:00 VCEX23C430401LC vcmd: [VCD:IESENC3_vc_domain:1024:Minor] Domain state DEGRADED : 1+ enclosures & profiles OK, DEGRADED, UNKNOWN, NOT-MAPPED, Previous: Domain state OK

Many thanks for the time ..