topic Re: Importance of path change settings in VMware in Array Setup and Networking

Importance of Path Change Settings in VMware

aherbert23 — Tue, 12 Mar 2013 15:23:54 GMT

With 1 gig networks it is important to tune per path settings for maximum throughput. The default of 1000 IOs per path can cause micro bursts of saturation and limit throughput. I've done a fair amount of testing and found that the best setting is to actually change paths based on the number of bytes sent per path. The reasoning is that it can be detrimental to change paths too often for small block IO. Setting the path change to bytes optimizes for both.

Here is a real world example from a demo I recently conducted. This was done with 4 1G interfaces on both the Nimble array and the ESX host connected through a Cisco 3750X stack.

SQLIO prior to path optimization :

Server	Tool	Test Description	IO/s	MB/s	Avg. Latency
SQL-06	SQLIO	Random 8k Writes, 8 threads with 8 qdepth for 120 sec	12375	97	4ms
SQL-06	SQLIO	Random 8k Reads, 8 threads with 8 qdepth for 120 sec	14456	113	3ms
SQL-06	SQLIO	Sequential 64k Writes, 8 threads with 8 qdepth for 120 sec	2130	133	29ms
SQL-06	SQLIO	Sequential 64k Reads, 8 threads with 8 qdepth for 120 sec	2147	134	29ms

SQLIO after path optimization :

Server	Tool	Test Description	IO/s	MB/s	Avg. Latency
SQL-06	SQLIO	Random 8k Writes, 8 threads with 8 qdepth for 120 sec	26882	210	1ms
SQL-06	SQLIO	Random 8k Reads, 8 threads with 8 qdepth for 120 sec	28964	226	1ms
SQL-06	SQLIO	Sequential 64k Writes, 8 threads with 8 qdepth for 120 sec	7524	470	8ms
SQL-06	SQLIO	Sequential 64k Reads, 8 threads with 8 qdepth for 120 sec	7474	467	8ms

Notice the large improvement in not only throughput but also the reduction in latency. The latency in the first test was due to the saturation of the 1G links.

The optimization is done with the following command from the ESX 5.x console:

for i in `esxcli storage nmp device list | awk '/Nimble iSCSI Disk/{print $7}' | sed -e 's/(//' -e 's/)//'`; do

esxcli storage nmp psp roundrobin deviceconfig set -d $i -I 0 -t iops;

done

This can also be done with a PowerShell script that is posted here :

Set VMware RoundRobin PSP through PowerCLI

Re: Importance of path change settings in VMware

pdavies34 — Tue, 12 Mar 2013 17:44:55 GMT

Why did you choose "-B 262144" as the bytes size, what other options/results did you test with? I assume this was the best overall balance between IOPS and Throughput?

Re: Importance of path change settings in VMware

aherbert23 — Tue, 12 Mar 2013 22:45:47 GMT

I did try several other options. Setting it to 256K per path seemed to be the sweet spot.

Re: Importance of path change settings in VMware

mandersen81 — Wed, 13 Mar 2013 05:00:36 GMT

This is great Adam Herbert greatly appreciated. I have seen this in a couple installs now, and will put this in my toolbox of changes for 1G installs.

Re: Importance of path change settings in VMware

dhamilton113 — Thu, 14 Mar 2013 03:02:21 GMT

really terrific and important post

Adam Herbert Thanks for this.

Re: Importance of path change settings in VMware

Thu, 14 Mar 2013 16:45:08 GMT

Can this command be run on an environment that already has VMs on the iSCSI LUNs without an interruption of service?

Re: Importance of path change settings in VMware

pdavies34 — Thu, 14 Mar 2013 16:50:31 GMT

Yes, it has an immediate effect and the way the script is written it only affects Nimble Volumes.

It is also relatively simple to modify the script to only affect an individual volume if you so wished.

Phil

Re: Importance of path change settings in VMware

aherbert23 — Thu, 14 Mar 2013 16:50:59 GMT

Yes, it is safe to run. No downtime needed.

Re: Importance of path change settings in VMware

wen35 — Thu, 14 Mar 2013 18:02:07 GMT

have you guys tried iops=0, this essentially ignore the # of IOPS per path before switching, and relies on queue depth. Essentially poor man's LQD on ESX! We are trying to do some testing in tech-marketing lab to get some results.

Re: Importance of path change settings in VMware

jwang131 — Thu, 14 Mar 2013 20:40:15 GMT

Yes setting the policy=iops with both iops=0 and bytes=0 may give better performance, where MPIO doesn't need to wait for 256K before switching path.

Re: Importance of path change settings in VMware

aherbert23 — Fri, 15 Mar 2013 02:18:21 GMT

I would be interested in seeing the results of the tests. When I tried using low IOPS per path numbers I saw small block random performance degrade. I did not try setting IOPS per path to 0. I didn't even know that would be a valid input!

Re: Importance of path change settings in VMware

Wed, 10 Apr 2013 16:21:15 GMT

These settings apply/benefit 10Gb?

Thanks

-Craig

Re: Importance of path change settings in VMware

wen35 — Wed, 10 Apr 2013 17:04:46 GMT

Yes, definitely. Assuming you have dual 10G bind to the iSCSI initiatior, PSP_RR will leverage both paths without having to wait for a given path to reach X IOPS or X bytes before the switch.

Re: Importance of path change settings in VMware

epedersen22 — Thu, 18 Apr 2013 17:06:29 GMT

Is this (or iops=0/bytes=0) going to end up in the Nimble VMware installation guide?

Re: Importance of path change settings in VMware

wen35 — Thu, 18 Apr 2013 17:55:17 GMT

it is making its way to the next edition of the vsphere on Nimble best practices guide - the same will go into the next edition of the vmware integration guide as well.

Re: Importance of path change settings in VMware

epedersen22 — Thu, 18 Apr 2013 22:09:44 GMT

Cool - thanks!

Re: Importance of path change settings in VMware

Sat, 20 Apr 2013 21:32:30 GMT

Made the change. My SQLIO test throughput on 10G on Reads went from ~470 MB/s to ~715 MB/s. Have a CS240G. NICE!

-Craig

Re: Importance of path change settings in VMware

wen35 — Mon, 22 Apr 2013 17:13:31 GMT

wow nice to hear Craig!

Re: Importance of path change settings in VMware

epedersen22 — Thu, 25 Apr 2013 23:40:59 GMT

I may be wrong, but when making the change to --iops=0 & --bytes=0, it looks like you have to set '--type' to 'iops'. I tried it using '--type=bytes' as written in the script above, but the iops limit didn't change.

Result when run with --type=bytes:

Device: eui.xxx

IOOperation Limit: 1000

Limit Type: Bytes

Use Active Unoptimized Paths: false

Byte Limit: 0

After, when run with --type=iops:

Device: eui.xxx

IOOperation Limit: 0

Limit Type: Bytes

Use Active Unoptimized Paths: false

Byte Limit: 0

From the help text:

-t|--type=<str>

Set the type of the Round Robin path switching that should be enabled for this device.

Valid values for type are:

bytes: Set the trigger for path switching based on the number of bytes sent down a path.

default: Set the trigger for path switching back to default values.

iops: Set the trigger for path switching based on the number of I/O operations on a path.

Cheers,

Eric

Re: Importance of path change settings in VMware

mallocarray12 — Fri, 03 May 2013 16:25:03 GMT

I just ran the command twice, one to set bytes, and then one to set IOPS. Since there is a Limit Type, I'm not sure if it matters if you changed bytes to 0 if the Limit Type is set to Iops.

My notes from another post:

In the SSH console on ESXi 5.1, this command will loop through each datastore setting Bytes to 0, IOPS to 0 and then display the current settings. For some reason, when listing disks, they show up twice, once with their regular ID and a second time with the ID ending in :1 and the settings can't be applied.

for i in `ls /vmfs/devices/disks/ | grep eui.` ; do echo $i ; esxcli storage nmp psp roundrobin deviceconfig set -d $i -t bytes -B 0; esxcli storage nmp psp roundrobin deviceconfig set -d $i -t iops -I 0 ;esxcli storage nmp psp roundrobin deviceconfig get -d $i; done

If you want it set back to defaults how it came out of the box, this will reset Bytes to 10485760, IOPS to 1000, and Type to default

for i in `ls /vmfs/devices/disks/ | grep eui.` ; do echo $i ; esxcli storage nmp psp roundrobin deviceconfig set -d $i -t bytes -B 10485760; esxcli storage nmp psp roundrobin deviceconfig set -d $i -t iops -I 1000 ; esxcli storage nmp psp roundrobin deviceconfig set -d $i -t default; esxcli storage nmp psp roundrobin deviceconfig get -d $i; done