HPE StoreVirtual Storage / LeftHand
cancel
Showing results for 
Search instead for 
Did you mean: 

vmWare Volumes and Gateway

Peter J West
Frequent Advisor

vmWare Volumes and Gateway

Hi,

 

Is it normal when looking at the iSCSI connections for a volume which is assigned to an ESXi host to only see connections to a specific node?

 

I've done a little reading online and it seems there's a trade-off between using volumes provisioned through vmWare and provisioning them directly to the Windows OS and then using the DSM for MPIO.

 

When doing the latter it seems that the host will establish seperate paths to each of the nodes (at least on the same site) which to my mind should boost throughput.

 

Is it really the case that using a vmWare-based volume is going to cause a bottleneck at the gateway?

 

I think our current configuration probably needs rationalising a little if that's the case because we currently use a combination of the above, but we have something like 40 volumes defined on a 6 node setup.  From what i've read such a large number of volumes could indeed prove a bottleneck and we'd be better off reducing the volume count to a few large volumes and then using vmWare to slice the disk up into the required parts.

 

Any comments or views on this would be appreciated.  We don't see massive performance problems on a day-to-day basis but I did note today that copying data from one volume to another was relatively slow, with transfer peaking at around 18 MB/s.

 

Grateful for any feedback on this.  :)

 

Pete

 

22 REPLIES
Dirk Trilsbeek
Valued Contributor

Re: vmWare Volumes and Gateway

as far as i know all connections to a specific LUN are redirected to a single node which is gateway for that LUN. But as you probably have more than a single LUN, traffic from and to other LUNs will be redirected to other nodes, so in a production environment this will level out, as long as you don't have LUNs with lots of traffic and others with no traffic at all.

5y53ng
Regular Advisor

Re: vmWare Volumes and Gateway

Are you using a physical appliance or a VSA?

 

Yes this is normal in VMware. As Dirk pointed out, since you have multiple volumes, the load balancing feature of the P4000 will attempt to balance the gateway connections across each node in the cluster. You can also implement MPIO in VMware as well as configure the maximum number of IOPS per path.

 

Some other things I have learned since I started working with the VSA is that you should limit the number of virtual machines per volume to 4 (per HP supprt engineer recommendation). I was able to confirm this recommendation is effective using IOmeter.

 

Another trick to boost performance (VSA only) is to put a virtual machine on the same ESX/ESXi host as the gateway connection for its volume. If you do this your IO stays on the vswitch and you'll see a noticeable boost in throughput. Unfortunately you have to set this manually, but if you have a VM that you need a little better performance from this helps.

 

Using MPIO and limiting the number of IOPS in VMware along with following the maximum number of virtual machines per volume, you can achieve decent performance.

 

The performance you are seeing when copying between volumes is about the same as what I see in a six node cluster.

 

 

Peter J West
Frequent Advisor

Re: vmWare Volumes and Gateway

Thanks for the comments.

 

And we're not using the VSA - we have a total of 6 P4500 nodes.

 

Pete

 

Emilo
Trusted Contributor

Re: vmWare Volumes and Gateway

Is it normal when looking at the iSCSI connections for a volume which is assigned to an ESXi host to only see connections to a specific node?

 

Yes this is normal behavior, the node that has the connection is the gateway connection. It serves as the "host" iSCSi connection and is responsible for obtaining the data from the other nodes in the cluster. This architecture has proved to be very efficient and scales well as it distributes the load. In addition the more volumes you have the better the performance boost. See this document http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c03178470&jumpid=em_alerts_us-us_Feb12_xbu_all_all_1639858_105818_storage_routine_006_1

 

You can as has been mentioned implement Native Multi Path for Vmware.

To enhance the availability of iSCSI storage, implement network multi-pathing for the two or more physical network adapters in VMware hosts that are dedicated to iSCSI traffic, . Follow the instructions provided in the VMware document, iSCSI SAN Configuration Guide; refer to the section entitled, “Setting Up Software iSCSI Initiators.”

 

 

I've done a little reading online and it seems there's a trade-off between using volumes provisioned through vmWare and provisioning them directly to the Windows OS and then using the DSM for MPIO.

 

With the P4000 the best performing and most fault tolerant iSCSI solution is implenting the MS Initiator with DSM / MPIO. In order to get the most benefit from this you need to have a minimum of 2 Nics on the host iSCSI server. You should also have a minimum of 3 or more P4000 SAN's. This will need to be a HW based MS host or Hyper-V as there are some incompatibliteis with VMware.

 

Hope this helps.

 

 

Peter J West
Frequent Advisor

Re: vmWare Volumes and Gateway

Thanks for the feedback - it's all very helpful.

 

As we're running vmWare it seems our best option is to not use MPIO within the guest OS but instead just use VMDK volumes provisioned directly from within vmWare.

 

I read an article recently from 5y53ng about configuring the path selection policy to IOPS and reducing the default from 1000.  However this in itself raises a few questions:

 

1. Is it necessary to run the esxcli command against every volume on every ESX host?  If so is there any way to script or automate the process?

 

2. Is there any way to automate the process on newly created volumes or is there a default we can modify to ensure new volumes automatically have these values set?

 

Grateful for any feedback on this one.

 

Regards

 

Pete

 

5y53ng
Regular Advisor

Re: vmWare Volumes and Gateway

Hi Peter,

 

You can script the process. In prior versions of ESX/ESXi the IOPS setting was not persistant, so the script was placed in etc/rc.local, so it would persist across reboots. I am running ESXi 5 now, but I still leave it in rc.local.

 

# fix broken RR path selection
for i in `ls /vmfs/devices/disks/naa.6000eb* | grep -v ':[0-9]$'`; do
esxcli storage nmp psp roundrobin deviceconfig get --device ${i##*/} 2>/dev/null && \
esxcli storage nmp psp roundrobin deviceconfig set --type "iops" --iops=3 --device ${i##*/}

 

Giving credit where credit is due, I got this script, which works very well, from "Matt" in the comments section of the article below:

 

http://virtualgeek.typepad.com/virtual_geek/2009/09/a-multivendor-post-on-using-iscsi-with-vmware-vsphere.html

Peter J West
Frequent Advisor

Re: vmWare Volumes and Gateway

Haha, thanks.

 

It seems I already had this in place as i've got this (similar) line in my rc.local file already:

 

for i in `ls /vmfs/devices/disks/ | grep naa.600` ; do esxcli nmp roundrobin setconfig --type "iops" --iops=1 --device $i; done

 

as far as I can make out it does the same thing albeit with the volumes being parsed in a different way.

 

One thing I didn't realise is that this file is only executed when the ESX host first boots.  Is therea ny way to schedule the job to run periodically?  This would then help also with newly created volumes.

 

Pete

 

5y53ng
Regular Advisor

Re: vmWare Volumes and Gateway

Take a look at the directory, vmfs/devices/disks/ and make sure you are not trying to apply a PSP of IOPS to any local disks or partitions. Using grep for naa.600* might not weed out the local disks and partitions. I think that is really the only difference between what I posted and what you already have in your rc.local file.

 

You might be able to schedule a task using the vCLI or Power CLI along with the task scheduler in Windows, or maybe using cron in ESXi. I haven't played around with Cron in the ESXi environment, so I can't say for sure.

dcolpitts
Frequent Advisor

Re: vmWare Volumes and Gateway

I realize this is an older thread, but given that the connections are made to a single node, we've found that when we've had a node failure / node reboot / etc, those specific volumes (datastores) on that single node (and any running VMs) obviously go offline.

I'm searching for work arounds to keep those volumes online during a node reboot or node failure. Any thoughts?

dcc
5y53ng
Regular Advisor

Re: vmWare Volumes and Gateway

Hi dcc,

Are your volumes configured as network raid10?
dcolpitts
Frequent Advisor

Re: vmWare Volumes and Gateway

Yep - Network raid 10.

 

dcc

Dirk Trilsbeek
Valued Contributor

Re: vmWare Volumes and Gateway

then they shouldn't be offline. In rare cases the time needed for the remaining nodes to detect the missing node can cause applications to crash (especially MS SQL Server) but usually there is no disconnect. Do you have a failover manager in your cluster? How many nodes?

dcolpitts
Frequent Advisor

Re: vmWare Volumes and Gateway

I do have a failover manager.  I can replicate at will in both my lab (using VSA on ESXi), and in a customers P4300G2 production environment.  In both cases there are two nodes plus a failover manager.

 

When a node reboots, the datastore(s) that node is presenting goes offline.

 

dcc

Dirk Trilsbeek
Valued Contributor

Re: vmWare Volumes and Gateway

That definitly shouldn't happen. Are the volumes offline in the CMC?

dcolpitts
Frequent Advisor

Re: vmWare Volumes and Gateway

No. And that is why I find it weird. (btw - I didn't mean to hijack your thread Peter).

dcc
Dirk Trilsbeek
Valued Contributor

Re: vmWare Volumes and Gateway

but you did connect to these volumes using the cluster VIP, not the node IP address? What kind of client (VMWare ESX, Windows Server etc) does connect to these volumes?

dcolpitts
Frequent Advisor

Re: vmWare Volumes and Gateway

Yes - they are connected to the VIP.  All servers are ESXi 5.1.0, build 914609 (fresh install from VMware-ESXi-5.1.0-799733-HP-5.32.5.iso, then upgraded with VCUM to the current build). 

 

In my lab, I'm have DL380G6s using 1Gbe software iSCSI via two of the NC382i nics (Broadcom hardware iSCSI appears to be flakey at best so I ended up on software iSCSI) configured with dynamic target discovery pointing that VIP's IP (there is also a 4 port NC364T in each host for a total of 8 1GBe ports for VM Network, vMotion, etc). The hosts are connected to a Procurve 2848 and flow control is enabled on the ports that are part of the iSCSI vlan.  From the ESXi hosts, I can ping all IPs associated with the storage system (in it's vlan).

 

In the production environment, I have DL380G7s using NC550 10GBe to connect to the P4300G2 nodes (also at10GBe - a 2nd NC550 is used for the VM Network). Bonding is enabled.  Storage and servers are connected to a Procurve 5400ZL.

 

I pretty much tried to follow "HP LeftHand Storage with VMware vSphere: Design considerations and best pratices (4AA3-6918ENW.pdf)" as much as I could.

 

In my test lab, I've left Jumbo frames turned off (I do believe the VSA does not support enabling Jumbo frames), although in the customer's production environment, Jumbo frames are turned on.

 

To ensure I didn't miss something in my configuration, I used the HP LeftHand CLI Shell to create my volumes, servers, and assignments, and used esxcil to configure my ESXi hosts.

 

To create my volumes in CLI, I used this command:

 

cliq createVolume volumeName=p4000-datastore01 description=p4000-datastore01 clusterName=p4000-cluster01 size=300GB

To create my servers in CLI, I used this command (192.168.11.6 = vcenter, 192.168.222.131 = VSA node 1):

 

cliq createServer serverName=esxi01-vmhba37 description=esxi01-vmhba37 initiator=iqn.1998-01.com.vmware:esxi01-6db9xxxx:37 useCHAP=0 allowiSCSIAccess=1 vipLoadBalance=1 controllingserver=192.168.11.6 login=192.168.222.131 userName=xxxx passWord=xxxx


To assign my servers in CLI, I used this command:

 

cliq assignVolumeToServer volumeName=p4000-datastore01 serverName=esxi01-vmhba37 login=192.168.222.131 userName=xxxx passWord=xxxx

 

To add the dynamic targets to ESXi, I used this command (192.168.222.101 = VIP).

esxcli iscsi adapter discovery sendtarget add -A vmhba37 -a 192.168.222.101:3260

 

I obviously edited and repeated these commands as required...

 

dcc

5y53ng
Regular Advisor

Re: vmWare Volumes and Gateway

When you shut off a node do you see a loss of quorum in the CMC? That's the only thing I can think of based off what I have read in this thread. Make sure you have a manager running on each node.

 

dcolpitts
Frequent Advisor

Re: vmWare Volumes and Gateway

I haven't noticed loss of quorum...  The manager appears to be running.  See the attached text file that has the getNsmInfo output from both VSA nodes plus the FOM.

 

dcc

5y53ng
Regular Advisor

Re: vmWare Volumes and Gateway

When you reboot a storage node, do you see a red X next to each volume in the CMC?

dcolpitts
Frequent Advisor

Re: vmWare Volumes and Gateway

No.

 

dcc

5y53ng
Regular Advisor

Re: vmWare Volumes and Gateway

Do you see the gateway connections change when you reboot a node?

 

If the gateway connections are changing up, I would start looking at your VMware hosts. Make sure all of them have the same number of devices/paths on the iSCSI adapter properties. Make sure all the paths are active or active IO.