topic Re: Hot-replace fibre switch in BladeSystem - General

Hot-replace fibre switch

Joshua Small_2 — Thu, 03 Feb 2011 06:11:54 GMT

Hi,

We have a C7000 with redundant B-series 8/24 SAN switches connected to a P2000 via fc. Each blade is running ESXi 4.1.

We have confirmed that, within each ESXi instance, each LUN presented is seen on both HBAs with two paths - ensuring our zoning, presentation, and multipathing is accurate.

During our availability testing, we confirmed that if you hot pull a SAN switch, there is no interuption to service and all active paths migrate over to the operating switch.

When plugging that switch back in, it appears to hot plug fine, in that it becomes accessible etc. It does not however appear to connect to the individual blades.

ESXi continues to report the uplnk as "dead" and viewing the connected WWN's on the switch interface shows you all the SAN interfaces, but none of the blades.

Rebooting a blade corrects this issue (for that blade). It does leave the installation a bit less "hot pluggable" than I expected however by the time you need to reboot 16 servers.

Any advise on this appreciated.

Re: Hot-replace fibre switch

Johan Guldmyr — Thu, 03 Feb 2011 08:13:02 GMT

Is it possible to do a refresh on the ESX blades? - of what they see via the FC HBA?

What do you mean it shows all the SAN interfaces? - Is that all the host ports on the P2000? What did a switchshow say?

Maybe an OA reseat, restart or maybe you can unassign/reassign the port mapping in OA?

Are you running a 'release set' of all the c-class firmwares? (Brocade switches are not listed in there - you can find the supported ones on hp spock).

Re: Hot-replace fibre switch

Jorge Pons — Thu, 03 Feb 2011 08:15:15 GMT

Hi

versions of FW are update?
The ESX dont see switches OK, but from Onboard, have the same problem or see OK.

Regards, Jorge

Re: Hot-replace fibre switch

Zygmunt Krawczyk — Thu, 03 Feb 2011 08:23:52 GMT

Which type of HBA do you have (QLogic, Emulex, Brocade)?

Re: Hot-replace fibre switch

Johan Guldmyr — Thu, 03 Feb 2011 08:37:35 GMT

And what kind of blades?

Re: Hot-replace fibre switch

Joshua Small_2 — Thu, 03 Feb 2011 22:59:53 GMT

Hi,

The blades are BL490c G6's. They were updated using Smartupdate 9.2 DVD with iLO supplemental update 2.15.

The OA is running 3.21 and it was reseated as part of our testing.

The SAN switches were upgraded from 6.2.2 to 6.3.2c during testing - problem existed before and after upgrade.

Yes, I do mean that the switch, after a hotplug, will see the SAN's HBAs, but not the Blade HBAs.

The OA lists everything as "connected" and a map/unmap does not help.

ESXi does have a vicfg-rescan command. It does discover recovered paths when, for example, we test removing a cable between the SAN switch and the SAN and reconnecting it. It does not however assist in this case.

Re: Hot-replace fibre switch

Joshua Small_2 — Thu, 03 Feb 2011 23:08:14 GMT

Interesting addition - the Flex-10 modules in use don't have this issue.
That is to say, I can pull one out, put it back in, and the network link on a blade fails over, then comes back up into redundant mode.

Re: Hot-replace fibre switch

Johan Guldmyr — Fri, 04 Feb 2011 08:15:37 GMT

Hey!

Do you have a VC-FC module as well?

I'm a bit stumped :p Is this supposed to work?

Re: Hot-replace fibre switch

Joshua Small_2 — Mon, 28 Feb 2011 23:49:05 GMT

No, I did not have a VC-FC module in place.

I just found a vmware discussion that seemed to place this as normal though:

http://communities.vmware.com/thread/304521?tstart=60

Re: Hot-replace fibre switch

Trygve Henriksen — Tue, 01 Mar 2011 09:23:31 GMT

Maybe the 6.4.1 update for the Switches can help?

Re: Hot-replace fibre switch

Trygve Henriksen — Tue, 01 Mar 2011 09:25:49 GMT

What's the port staus on the switch?
(Right-click on the ports while in the web administration interface)

Re: Hot-replace fibre switch

Michael Vaarning — Mon, 16 May 2011 07:18:40 GMT

Did you find a solution to this problem?

I seem to experiencing the same issues.

Environment:

ESXi 4.1 update 1
vCenter 4.1 update 1

C7000 Enclosure

HP B-series 8/24c SAN Switch BladeSystem c-Class
Firmware 6.4.1b

QLogic QMH2562 8Gb FC HBA

If you found a solution, I would be very glad to hear about it.

Re: Hot-replace fibre switch

Michael Vaarning — Mon, 23 May 2011 12:33:13 GMT

We researched the issue together with experts from BladeTeam, and found that the HBA didn't log back into the SAN switch, after being disconnected from the Brocade switch.

However when we forced the HBA to run at 4GB/s it did login. But at 8GB/s or auto negotiating the HBA was only able to login when the bladeserver was restarted.

We updated the qla2xxx driver on each ESXi host to 841.k1.16.2-1vmw and that solved the issue for us.

Re: Hot-replace fibre switch

Joshua Small_2 — Tue, 24 May 2011 03:33:57 GMT

That's great news Michael. We put this issue on the backburner ourselves once the servers were live.

I'll assume a VMware upgrade will come out with the required driver upgrade built in - can you link exactly the download you used to correct it for me?

Re: Hot-replace fibre switch

Devan LV — Tue, 24 May 2011 20:57:06 GMT

Thanks Michael. I've been working on the same issue and was glad to hear other people were experiencing this. I updated my builds with the driver you mentioned and this has fixed my issue as well.

I downloaded the driver from the qlogic site which links to the VMware site.

http://support.qlogic.com/support/oem_product_detail_vmware.asp