vSphere and multi-site

[Olvi_1]

I want to set up multi-site with two VIPs. The HP guide (Running VMware vSphere 4 on HP LeftHand P4000 SAN Solutions) is confusing me.

In page 7:
"It is important to note that native vSphere 4 multi-pathing cannot be used with HP P4000 Multi-Site SAN configurations that utilize more than one subnet and VIP (virtual IP)"

But in page 11:
"When connecting ESX or ESXi hosts to a Multi-Site SAN each of the virtual IP addresses (VIPs) of the SAN from each site should be listed in the discovery list of the ESX or ESXi software initiator. Path selection policy for Multi-Site SAN volumes should be set to fixed"

I have found many good guides how to setup vSphere iSCSI initiator to use Round-Robin but all of those guides are for one IP target.

So, does anyone have experience how this should be setup and can I use Round-Robin?

-Olvi

[teledata · Accepted solution]

Olvi,

I went through the very same thing so I think I can help clarify for you. (unless things changed with the SAN/iQ 8.5 release, but I haven't dug that deep into 8.5 yet)

Both page 7 and Page 11 are correct.

vSphere 4 DOES NOT support multiple subnets (or iSCSI routing) with multi-pathing This is simply a restriction on the way VMware implemented iSCSI multipathing

and you MUST have 2 VIPs (and they are required to be in unique subnets) in a LeftHand multi-site, this is a design requirement of HP LeftHand SAN/iQ software design.

Yes - these 2 features are mutually exclusive. You can EITHER do iSCSI multi-pathing (round robin) with VMware OR you can do Multi-Site SAN with LeftHand, but currently you can not do both.

You must use both VIPs in the discovery, but cannot bond multiple iSCSI kernels to the iSCSI initiator. Fixed Path will provide storage path redundancy, but not any load distribution.

I even tried creating both VIPs as stretched VLANs, and all sorts of routing trickery to try to setup multipathing with Multi-Site SAN, and whenever I did it it would break the redundancy and resiliency or cause storage pathing problems. If you find a way to fool it into working please post it here!

I will say that the customer was quite pleased however when we methodically tested node failure, link failure, and site failure, and their Exchange server stayed UP, and the information store stayed mounted and online. They were less concerned with storage throughput (as their iSCSI load wasn't that high), and more concerned with high availability.

I suspect HP will work with VMware and eventually this problem will likely be resolved (maybe through the vStorage API???), but for now you can either have iSCSI multipathing or Multi-Site failover, but not both.

If you would like additional assistance with this configuration I have a bit of experience with it, please feel free to send me a PM or contact me through our website: www.tdonline.com

[Olvi_1]

Thank you a lot teledata, that's exactly what I was suspecting!

There is one way to have both but I don't know is this supported or not:
Multi-Site cluster with single VIP. IOW, nodes in same subnet.

I read all manuals that I could found and although it's not best partices I haven't found anything that denies this configuration.

I can't think of a reason why this should not work as long as you have enough bandwidth between sites. I have tested this with VSAs and it works, VIP makes failover to secondary site.

Does anyone have experience or opinion why this should not work?


-Olvi

ps. Here is links to some better manuals for initializing iSCSI in vSphere
http://www.vmware.com/pdf/vsphere4/r40/vsp_40_iscsi_san_cfg.pdf
-Starting page 32

http://www.scribd.com/doc/24586958/Configuring-Left-Hand-ISCSI-and-VSPHERE-MPIO
-Good images, view fullscreen or download

[teledata]

The problem is the VIP often doesn't migrate fast enough for systems that are sensitive to connectivity loss (databases, exchange comes to mind).

The customer site I was referring to above was in that situation. When we first worked with them they had a (non ESX) single VIP Multi-Site SAN cluster, with only 1 subnet. If they had a site-link failure their fileservers would recover fine, but their Exchange server would almost ALWAYS loose the information store. (one time the unclean dismount caused db corruption, and you can imagine that was a fun weekend for them!)

We helped the customer because they were adding 4 more storage nodes (built a new cluster), and built it out as a multi-site SAN, then converted their OLD cluster into a true multi-site SAN as well, so they have 1 cluster with the FoM at SITE A, and a 2nd cluster with the FoM as Site B... this way they have systems that will stay online at their own primary location, yet both have multi-site failover capabilities. (basically we put 2 ESX hosts at Primary Site A, and 1 ESX host at secondary SITE B.

Exchange (and SQL Server) will often suffer from a single VIP Multi-Site configuration is the downside. I didn't test it, but somehow I suspect VM datastores would suffer the same fate in a VIP failover across sites... At the very least the VMs would probably HANG for 15-30 seconds during a failure, but if the connection to the datastore actually times out before the VIP comes back up then your VMs may crash.

My recommendation to them was, stick with the more reliable multi-VIP configuration to provide a higher level or resiliency and availability, and if HP/VMware make changes in the future I wouldn't be surprised if you get to add iSCSI storage multipathing in a future release.

Or maybe a plug-in to v-Storage API from HP-Lefthand? (Hint hint hint.... to HP Lefthand engineers!)

[Olvi_1]

Teledata, it's almost as like we have one brain. I realised the problem with one VIP maybe 5 minutes before you replied :)

True Multi-Site it is then. And back to my original question: How do I configure ESX with two VIPs? All HP manual says that pathing should set to "fixed". Teledata, you posted earlier that I can't bond VM kernels. I assume you mean esxcli-command but how about the rest? Do I still assign kernerports with IPs to different subnets? How about Override Switch failover policy, should it be left to default?

Teledata (or anyone), I beg you share your experience with me and others. I will probably figure out best way to configure ESX eventually but you could save A LOT of time and banging head against the wall.

I agree that vmware should work on the iSCSI multi-pathing. And HP should work on LH failover. Why could there be two (or more) VIPs in single site cluster? I don't see any problems there and won't be surprised if SAN/iQ 9.x will have this feature. Plus HP really needs to publish better manuals!

-Olvi

[teledata]

Sorry for the late response. I'm guessing you already figured it out:

for ESX (vSphere) you need to put BOTH VIPs in the discovery address tab for the iSCSI initiator. I think you are stuck with using the Network stack for failover of the iSCSI NICs for now since you can't bond the kernels.

For my customer I created a stretched VLAN, if that's not possible you'd have to configure routing between the 2 iSCSI subnets on your router or layer-3 switch.

[Olvi_1]

Hi, again.

Actually I haven't had change to test configurations so all hints are still highly welcome although I'm getting the idea how this should be done. Thank you, teledata, for that.

I created TODO list for summary:
- Create Kernel Port with IP address to subnet A (along with new vswitch)
- Add second Kernel Port with IP address to subnet B
- Add NICs to the vswitch
- Enable iSCSI initiator
- Add both VIPs to Dynamic Discovery list
- Set Path selection policy to Fixed
- Rock 'n' Roll

Am I missing some step? Any Comments?

-Olvi

[DB ADC]

Anyone have any further information? I am having issue whereas once a LUN has been connected to an ESX "site" no other esx server can connect to that LUN. by site, I mean IP subnet...all esx hosts are in same data center.

Individually, all esx hosts can connect to LUN presented. Two esx hosts in same subnet can connect but attempting to connect a second esx with diferent ip subnet fails. esx host cannot even see LUN that has been presented.

currently bleeding on this

[Paul Hutchings]

Ah! This could be my eureka thread :)

So in multi-site you can have failover/redundancy with two subnets, just not using multi-pathing!