Hello,
I am having a problem trying to get a couple of new blades connected to our SAN fabric.
I have a c7000 blade enclosure with 2 x HP VC FlexFabric 10Gb/24-Port Modules. The virtual connects have two flex10 (shared) uplinks and 2 x Fibre connections for the SAN fabric.
The Chasis has 10 working blades (BL460c's) already in that are connected to the fabric, 8 running VMware 5.5 and two running Server 2008 R2.
My problem is, when I add new blades (another BL460c Gen8 and a BL460c Gen9) they will not connect to the fabric.
I have copied the server profile for one of the working VMware blades and assigned it to a new bay. I then inserted and new blade and built the server using the Vmware 5.5 HP custome image , ran all the firmware updates on the blade. The Ethernet Network links are connecting as expected, the HBA's are present with the correct WWN's but they can't see any paths.
I arranged for the Network team to add the HBA ID's (WWN's) to the Nexus 5000's (acting as the SAN switch), I watched them add the configuration (where they copied and modified configuration for a working server) and entered the correct WWPN's. We have a TAC case with Cisco open and they have confirmed that the configuration is correct and should be connecting, but the Nexus is not seeing anything from the new blades.
On the Virtual connect the interconnect bays are showing "Logged In".
I forgot to mention. I moved the blades from Bays 5+6 to bay 13-14 after the firmware updates to try to eliminate the bays as the problem. This is why the logs below state Bay 5+6, but the screenshot above shows 13+14.
But on the Nexus 5000's they are not showing as active, the indicators are not showing up on the SAN for me to register and assign LUN's.
It looks like there is an issue either with Virtual Connect getting stuck and not forwarding requests to the Nexus or there is something missing on the Nexus configuration (that Cisco have missed) causing it not to connect to the fabric.
I have a ticket open with HP who asked me to upgrade the firmware on the OA and VC to 4.5. I upgraded to this and the problem is still the same.
Observations during pull tests and VC firmware updates ---- After the VC firmware updates I also physically removed the VC modules, one by one (as not to cause downtime), I noticed that when I pulled out VC module 2, all systems stayed up (even the Ethernet side of the two new blades). I did get some reduncancy warnings on the blades as expected. When I removed VC module 2 (after module 1 had come back up) all of the "working blades" stayed up (reporting redundancy warnings again) but the two new blades both stopped responding to Ping. When I reinserted the VC module 1 they started responding to Ping again.
I reported these findings and a new set of logs to the HP team on my case, whoe are looking through the new logs, but they are currently thinking "replace the virtual connect modules" and I am a bit worried this may not work, considering there are 10 other blades in the chasis all using those virtual connect modules without issue....
One error that HP did find in the logs before the firmware updates (that I can't find so it must be in the VC collects) was the following:
1. [VCM_OP_STATE_UNAVAILABLE] New Op State [VCM_OP_STATE_OK]
2. EnetModuleDownlinkPort::loadData: platform claims DCC is not enabled on port enc0:iobay2:d15 but model thinks otherwise.
2016-08-04T15:04:11.826+01:00 [Personality.c:4742] ERROR: CQHORD retrieval failed for server 5
EnetModuleDownlinkPort::loadData: platform claims DCC is not enabled on port enc0:iobay2:d15 but model thinks otherwise.
EnetModuleDownlinkPort::loadData: platform claims DCC is not enabled on port enc0:iobay2:d15 but model thinks otherwise.
I have had this case open with HP for months now, in various tickets and I am completely at a loss as to what is causing this.
Any help with my situation would be greatly appriciated.
Kind Regards,
James
EDIT: I have had a look at the Throughput Statistics and noticed that the HBA's don't appear to be sending anything:
Bay14 Config:
LOM1-1:b=>Bay 1:d14:v2
LOM1-2:b=>Bay 1:d14:v2
Stats for both:
