StoreVirtual Storage
1748089 Members
4870 Online
108758 Solutions
New Discussion

Re: Performance issues when using VSA on ESX with VMXNET3 driver

 
fourg
Advisor

Re: Performance issues when using VSA on ESX with VMXNET3 driver

I might be missing something, but why not use a fixed path to the iSCSI virtual IP to handle the load balancing?

Princes
Advisor

Re: Performance issues when using VSA on ESX with VMXNET3 driver

"Most recent path used" seems to settle things down a little - storage performance is still poor though not crippling so. This and "Fixed path" still allows a path to be created locally to the VSA, the pathing distribution itself being managed by the VIP and the internal pathing logic employed by LeftHand OS (which you cannot manually change).

 

I have now tried using the NPAR featue of the adapter. As expected, each port was given an additional 3 virtual adapters that can be used to created dedicated vSwitches - one for VSA and one for iSCSI Adapter. In practice however I have found that the setup is unstable - it tended to work on only one host at a time in a cluster where two Active iSCSI paths could be sustained. On the other partner host in the cluster, one path would always be down. This seems more likely an artifact of how the cluster VIP interprets requests to establish paths from the adapter than anything else - it did not seem to like virtual adapters.

 

Thus I reverted back to the 2 x 10GbE 'real physical' Port arrangement. Then I discovered the article linked below, implemented the recommendation, and now storage runs like the wind with no slowdowns. It seems HP LeftHand is amongst those storage systems suseptable to this condition, especially when storage data is passed over the hypervisor stack and MTU9000 is used. Once enabled, the reverse situation seems to be the case - local VSA storage access is even faster than storage data accessed over the network from the partner VSA (which is still fast). According to the Performance Monitor in the CMC, I am now getting anything up to 600MB/s with latencies of 7-8ms. VM's are also reporting substantial storage performance via vCenter. Happy days!

 

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598

douglascountyit
Occasional Advisor

Re: Performance issues when using VSA on ESX with VMXNET3 driver

Thank you for sharing the link.

 

Just to confirm, you applied the work around "Disabling Delayed ACK in ESX/ESXi 4.x and ESXi 5.x" and now your HP VSA storage system is performing considerably faster, correct?

douglascountyit
Occasional Advisor

Re: Performance issues when using VSA on ESX with VMXNET3 driver

Did you see my last post? Please confirm the solution. Thank you.

 

:-)

Princes
Advisor

Re: Performance issues when using VSA on ESX with VMXNET3 driver

Sorry - only checked this in passing today.

Yes, if you are using Round Robin PSP then strongly consider the recommended advice of diabling DelayedAck on each ESXi host.

Princes
Advisor

Re: Performance issues when using VSA on ESX with VMXNET3 driver

Testing VSA v12.5 with MEM. Similar results to Default SATP with RR path policy, IOPS=1, DelayedAck disabled. Creates 12 rather than 4 paths to the VSA cluster however, with 2 Active I/O on each iSCSI Software Adapter.

Princes
Advisor

Re: Performance issues when using VSA on ESX with VMXNET3 driver

I have thoroughly tested the new MEM vib, however in a two node VSA / ProLiant combination at least, performance and especially latency is actually poorer than using the 'traditional' Defualt SATP with Round Robin and IOPS 1 arrangement.

Like other testers who have posted on here and in VMware's forums, I have noted that over and over again superior performance and the lowest storage latencies can always be achived when running on a single node in the two node cluster, e.g. when one node is shut down, and using NW RAID 10 volumes.

Turning off NW RAID 10 traffic (e.g. setting volume to RAID 0) does not seem to have a similar effect however - performance is lower and latencies the same as if running two nodes. Presumably only the control traffic that maintains cluster integrity is left behind after that, but why would this have such an impact on latencies?