FC on BL460c G1 in C7000 Mystery

[CoryB]

We were having the same issues with two of our BL460c G1s that were running ESX where certain paths would go blind to the fabric. For us the problem was not SAN zoning but that the HBAs were not logging into the VC-FC switches, which is why I mentioned the VC Manager - the fix for us was a reset of the the two VC-FC interconnect modules. Note that we also had to reset the VC-Enet module that was the primary VC Domain Manager after we added some G6 blades to our chassis as well.

[Adrian Clint]

He doesnt have Virtual Connect. He has Cisco 9124 SAN switches.
But I can see why you did it for VC.

[Bob Firek]

Adrian, actually we are using Virtual Connect. VC is used to configure the the backend blade stuff, FC connections, ethernet ect. The enclousure is attacted to the MDS9124 through the VC-FC modules and the EVA6100 is connected to the MDS9124. So the issue that CoryB talked about is intresting to me. I'm just not sure I'm up to the task of pulling all the cards but I guess if that is what has to happen. CoryB can you explain this sitution a little more? Did this process come from HP support?

[CoryB]

When we added the two new blades to the existing enclosure they were having difficulty with iLO connectivity dropping. When I called HP, they did recommend bouncing the OBA modules which resolved the iLO issue.

When we ran into the FC connectivity problem, I bounced the VC-FC interconnects based on the success of the OBA resets and that resolved the FC problem as well, no support call for this part.

[Adrian Clint · Accepted solution]

Ah yes silly me I missed the bit where you started talking about VC. I've been quoting a blade solution with 9124 internal switches over the last 2 days and forgot the model no also refers to external ones.

As you mention you have 4,8,10 on the same uplink does that mean you have static distribution set to map uplinks>downlinks?
Uplink #1 - 1, 5, 11, 15
Uplink #2 - 2, 6, 12, (16)
Uplink #3 - 3, 7, 9, 13
Uplink #4 - 4, 8, (10), 14

I'm also assuming you gone thru the actions in Appendix B in the VC SAN Cookbook?
http://bizsupport1.austin.hp.com/bc/docs/support/SupportManual/c01702940/c01702940.pdf

I'd also ask for advice on the Virtual Connect User Group there are many of the VC design and support team on there.
https://h30340.leveragesoftware.com/default.aspx
Jeroen did a good list of VC commands you can type that will help you see more detail on the port status on the VC side
https://h30340.leveragesoftware.com/portfolio_detail.aspx?fileid=3ec7b55a121b45debbe6bd41258eaec2

[Bob Firek]

CoryB - logical. I curious. Did you bring down the blades before reseating the VC-FC modules or did you just yank it out?

Adrian - You've given me a lot to think about. I'll get back to you when I don't have "people" pulling on me. The nerve.

[Bob Firek]

Adrian - I believe we have static distribution set to map uplinks>downlinks. I didn't set up the configuration but was involved in the discussions. I recall that originally the mapping was set up for dynamic but we ran into the issue of the blade in bay 2 looking the fiber connection on port 1 to the SAN. We switched to static and initially it seemed to solve the problem but after a time it lost its connection to the SAN and no matter what we did we could not get it to reattach. About a month ago after applying some VMware updates and rebooting the blades I noticed that the blade in bay 10 lost its connection to the SAN on port 1. Also during these VMware updates I noticed that than when I brought down blade 4 and 10 I would receive errors on the Polyserve server on blade 8. Actually the errors had to do with fencing and I believe to prevent "split-brain" Polyserve issued a reboot. So to me that seems to indicate the connection to the SAN was lost but Polyserver saw both servers were up and communicating. It didn't want to corrupt the data so it rebooted the suspect server. So what I'm wondering is what causes the connection to the SAN to be interrupted when some blades in the uplink group and shut down. Any help or suggestions you can provide is greatly appreciated.