FC on BL460c G1 in C7000 Mystery

[Bob Firek]

I have a mystery and I'm not quite sure how to describe it. We have a C7000 connected to two Cisco MDS9124 connected to an EVA6100. The FC hab1 ports on the blades in device bay 2 and device bay 10 do not connect to the storage. I've checked the fabric, cables everything looks good. These two blades are running ESX3.5. The really strange thing is that if I shut down the ESX server in device bay 10 for maintenance it will reboot our Polyserve blade in device bay 8. This is just a basic overview. I can provide more detail if need. I'm just beating my head against the server rack now. Any help or suggestions will be greatly appreciated.

[Adrian Clint]

Sounds like a SAN switch issue to me.

Are your HBA driver,firmware and SAN switch firmware all in the EVA support matrix?

I'd also update the iLO, BIOS and Power controller firmware if you have power action issues.

[Bob Firek]

Adrian,

The HBA drivers and firmware on the blades are up to date. The SAN switch firmware is on the EVA support matrix but just barely. Currently switch 1 is running version 3.1(3a) and switch 2 version 4.1(1b). I want 4.1(3a) or what is the current version. But before I upgrade the switches I'd like to resolve the pathing issue. I guess I'm in a chicken and an egg sitution. Is the older code causing the problem and upgrading will fix the problem or could be something else. I have one more blade to upgrade the firmware and iLO on before I can upgrade the enclousure and OA cards. The last blade to upgrade is hard to get freed up to run the upgrade. So I'm thinking get the get the enclosure, iLO, Onboard Administrator and Virtual Connect first and then move to upgrading the fiber channel swithes and then upgrade the EVA. Any thoughts or suggestions?

thanks for your help

[Adrian Clint]

Bob,
Your config is as far as I know an invalid unsupported config. Having two switches with different firmware is probably your biggest issue. (especially major versions being different)
You need to get someone to confirm that the EVA firmware, Switch firmware, HBA firmware and HBA driver are in a supported matrix...not just the latest.
Eg it could be that the latest switch firmware is not in the supported matrix for your EVA firmware.
Get this confirmed in the SAN forum
http://forums11.itrc.hp.com/service/forums/categoryhome.do?categoryId=248

[TTr]

> The FC hab1 ports on the blades in device bay 2 and device bay 10 do not connect to the storage

I read this as blades 2 and 10 have FC HBAs in them but are not in use.
You mention port1 but what about port2?
What about the zoning in the two MDSes? Anything in there that is related to blade8?

What about the ESX OSes of blade10? Is there any VM that is tied up to the polyserve server at the OS or application level?

What do the reboot logs say on the polyserve?

[Bob Firek]

Adrian, good time. I have to check my notes but I think I did my due diligence a couple of months ago and I was in spec but I'll post in the SAN forum.

TTr,
Thanks for your thoughs. Intresting. When I checked in Fabric Manager blade 8 shares the same connection with blade 10 and blade 4 through the the HP 4Gb VC-FC module on both port 1 and port 2. Blade 10 shows connected under link status as does all of my other hosts. However when I got to the ESX host in blade 10 and do a esxcfg mpath -l show I do not see any storage on port 1. If I got to the ESX host on blade 4 and do the same command I see storage on both port 1 and 2. When I go to the Polyserve server in port 8 and look through HP MPIO DSM Manager I see storage on both port 1 and 2. Now here is the kicker. I didn't mention this before but I have another Polyserve server on blade 16 (which shares the fiber connection with blade 2) and when I go into HP MPIO DSM Manager I only see storage on port 2. Very strange. Do you think the mismatched software versions on the switches could be causing this anomaly.

[TTr]

I am not sure about what you mean by "seeing storage on port...". Do you have any zoning defined on the MDSes? Or is it a flat fabric and possibly getting interference between HBAs? A zone should have one initiator (server) port and one target (disk array) port in it. If you have more than one HBA in the same FC zone you will get target traffic interference across the HBAs. I would expect the firmware difference to be the last thing to cause this but it could do it as well.

[CoryB]

What fibre interconnects are you using?

If Virtual Connect modules, log into VC Manager and look under 'Hardware Overview' --> 'Interconnect Bays' --> Bay 3 and Bay 4. Scroll down to the Server Port Information section and verify the individual ports are logging into the VC modules.

[TTr]

> What fibre interconnects are you using?

The Cisco MDS9124 are fiber interconnect switches. I think the zoning needs to be verified so that there is no HBA interference across the blade servers.

[Bob Firek]

TTr - It appears that the Polyserve servers are seeing the correct luns that are present to them except Polyserve02 can only see the luns present on fabric b. It is blind to fabric a. Where as Polyserve01 can see the storage present on both fabric a and b. This is by design for redundency. This is the same for the VMware hosts. They see the correct luns present to then except that blade 2 and blade 10 are blind to fabric a. I've checked the zoneing in Fabric Manager as everything appears correct. The blades connect to two HP 4Gb VC-FC modules. These modules has four 4Gb uplink ports that one module goes to FC switch 1 (fabric a) and the other one goes to FC switch 2 (fabric b). The fibre channel ports in the blades are split between the two modules. Each fibre port in the VC-FC modules handles four blades. It seems to me that if it was an issues with a bad cable or dirty connector the all blades connecting through that fiber connection would not work. I'm at a lost to determain where the problem lies. I hate to upgrade a system that isn't a 100 percent because who knows what kind of problems I may create for myself.

CoryB, good suggestion but another problem I ran in to is that a while ago we had added four additional blades that happen to be ProLiant BL460c G6 model. We added then to the enclousure that had BL460c G1 blades and appearently that broke Virtual Connect. When I try and launch VC I just get a gray striped "candy cane" with a message,"Loading, please wait..." I'm told upgrading the OA will take care of that but I can't get Polyserve02 long enough to run the upgrade on the server.