StoreVirtual Storage
1748249 Members
3269 Online
108760 Solutions
New Discussion юеВ

Multi-Site SAN and vSphere 4.1, multiple subnets or not?

 
balblas
Advisor

Multi-Site SAN and vSphere 4.1, multiple subnets or not?

HP's documentation is conflicting. In the P4000 CMC's helpfile it says: "When using VMware ESX 4 or higher and its native MPIO in a Multi-Site SAN, you cannot configure more than one subnet and VIP. Multiple paths cannot be routed across subnets by the ESX/ESXi 4 initiator. With VMware ESX, the preferred configuration is a single subnet and with ESX servers assigned to sites in SAN/iQ."

 

However this whitepaper (http://www.vmware.com/files/pdf/techpaper/vmw-vsphere-p4000-lefthand-san-solutions.pdf) says: "HP P4000 Multi-Site SANs require virtual IP addresses (VIPs) in unique subnets. When connecting ESXi hosts to a Multi-Site SAN each of the virtual IP addresses (VIPs) of the SAN from each site should be listed in the Dynamic Discovery Send Targets list of the iSCSI Software Adapter. Path selection policy for Multi-Site SAN volumes should be set to fixed (default)."

 

I have followed the whitepaper, ie. multiple subnets, but a vmware host in site A is unable to connect to a volume for which the gateway connection happens to be a storage node in site B. The only way to resolve this is to manually add a vmkernel route to the host, or to change the default gateway to an address on the iSCSI vmkernel subnet.

 

I have setup sites in the P4000 CMC and moved the storage nodes and the vmware hosts into the correct sites. Nevertheless I doubt if the vmware hosts actually connect to local storage nodes only. Looking at the paths in vSphere clients only shows paths to the gateway connection node.

 

Anyone has experience with Multi-Site and VMware? Using multiple-subnets or a single subnet spanning both sites?

7 REPLIES 7
balblas
Advisor

Re: Multi-Site SAN and vSphere 4.1, multiple subnets or not?

After reading some more posts in this forum I start to wonder if I got it wrong. I currently have 2 subnets, one for each site. The storage nodes in site A are using an IP-address from subnet A, and a VIP from subnet A. Same for the storage nodes in site B (using addresses from subnet B). Both subnets are routed. The VMware hosts in site A have an software iSCSI interface with 2 nics bound to it, both using an IP-address from subnet A only.

 

But now I'm wondering, should I configure 2 iSCSI nics on the VMware hosts, one with an address from subnet A and the other with an address from subnet B? We can span the subnet vlans over both sites if we need to...

Paul Hutchings
Super Advisor

Re: Multi-Site SAN and vSphere 4.1, multiple subnets or not?

The VMware iSCSI Kernel can't route basically.

 

Your two options are either to run an iSCSI kernel on each host from each subnet and trunk the VLAN between the two locations, or just use a single VIP across both sites and operate a stretch cluster - obviously bandwidth and connection latency will determine whether you can do this, but we do and it works great.

balblas
Advisor

Re: Multi-Site SAN and vSphere 4.1, multiple subnets or not?

Hi Paul,

 

Although lots of people say the vmkernel can't route, it actually can, and is supported by VMware: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002662 - "Although VMware does not recommend that VMotion traffic be routed, both iSCSI and NFS traffic may be routed through a gateway to reach a storage server on a different subnet."

 

We initially used one subnet spanning both sites and a single VIP, and as you said it works, but it's far from optimal. The reason I think 2 subnets are recommended is 1. VIP failover is not always fast enough for critical apps like Exchange and 2. Using 2 subnets would allow hosts to only connect to local storage nodes.

 

Reason 2 is exactly what HP says in some of their documents for VMware, the only problem is it doesn't work that way. From what I can see now the VMware hosts don't connect to local storage nodes, they just connect to what ever node happens to be the gateway connection for a given volume. Also see this article: http://www.virtuallifestyle.nl/2011/11/hp-p4000-multi-site-san-and-vmware-vsphere-watch-out/

 

My biggest problem at the moment is that HP's documentation is not consistent about this subject, it's in fact conflicting. in the P4000 CMC's helpfile it says use a single subnet in a multi-site setup for VMware, in this article they say use 2 subnets: http://www.vmware.com/files/pdf/techpaper/vmw-vsphere-p4000-lefthand-san-solutions.pdf

 

I'm lost...

balblas
Advisor

Re: Multi-Site SAN and vSphere 4.1, multiple subnets or not?

Had a conversation with HP support. They acknowledged that the documentation is wrong. With VMware and a Multi-Site SAN, the recomendation is to use a single subnet spanning both sites. As there is no DSM for VMware, this means all VMware iSCSI traffic for all hosts goes via one storage node (the gateway connection for the volume) and might possibly travel the ISL.

L1nklight
Valued Contributor

Re: Multi-Site SAN and vSphere 4.1, multiple subnets or not?

Hey blahblas,

This could possibly be relevant to me. Let me see if I understand your situation correctly.

 

You have 2 subnets. Each subnet hosts some storage. You also have a VMware install and you want the ESXi server to have storage allocated from both of the subnets' storage systems. So you have a LUN being delivered from 172.16.20.0 subnet and a LUN being delivered via the 172.16.21.0 subnet. The time comes where the LUN on the 172.16.20.0 subnet has become IO saturated and you want to sVMotion Server01 from that subnet's storage volume to the storage volume on 172.16.21.0 because that storage volume is idle (or something). 

 

Is that kind of the goal?

balblas
Advisor

Re: Multi-Site SAN and vSphere 4.1, multiple subnets or not?

My goal is to have a Multi-site SAN, with volumes available in both sites, but with the application servers using those volumes only connecting to storage nodes in their local site. With Windows servers this is possible because the HP P4000 DSM for iSCSI takes care of this. As there is no P4000 DSM for VMware it does not work for VMware hosts. The result is a VMware host might talk to a storage node in the other site, and you have no control over that. This means the ISL (Inter Site Link) will not only be used for RAID traffic between the storage nodes, but also for direct VMware <-> storage node traffic.

Inks
Occasional Advisor

Re: Multi-Site SAN and vSphere 4.1, multiple subnets or not?