** IMPORTANT -- DSM/MPIO + VMware Users, PLEASE READ **

[Craig A. Liess]

Hi folks -

There's another thread a little ways down about a slightly-related topic, but I wanted to make sure as many people read this as possible.

We have uncovered a serious bug in the 9.0 DSM/MPIO driver that causes significant hurt to VMware VMFS volumes. Here is the situation:

If you have a Windows host running virtual machine backup software (such as vRanger Pro) and you are doing LAN-free backups via iSCSI and you install/upgrade to the 9.0 DSM/MPIO driver, it causes latency on the attached VMFS volumes to skyrocket to 400-600ms during even moderate disk operations and vCenter begins throwing miscellaneous disk warnings.

Uninstalling the driver from your Windows host does not cure the issue. Logging off your hosts from the volumes, representing, etc. also do not solve the problem.

The ultimate fix was to carve out brand new volumes and migrate all of the VMs over COLD. Storage vMotion isn't an option since taking/committing snapshots becomes a complete crapshoot when your volumes are experiencing that kind of latency.

Additionally, we were able to reproduce this issue on a completely different set of servers, P4300 nodes and network gear.

After a number of weeks of prodding with support, I finally got HP to admit this was an issue. No idea if they're going to issue a technical bulletin on this or not, but I wanted to warn everyone here.

FWIW, the initial cluster we encountered this issue was on SANiQ 9.0, and the second was 8.5. Clearly, this issue is with the DSM/MPIO driver.

Please do not hesitate to ask if anyone has any questions, would like any more details, etc.

Hope this saves someone a few days of headaches!

-Craig

[Ben Drawbaugh]

Is there any way to verify a system is experiencing this problem? We have a similar setup to what you describe and I haven't noticed any issues or had any reports, but I'm afraid it is only a matter of time now.

Does it happen to every vmfs volume you connect to from the Windows backup server?

[Craig A. Liess]

Yes, it happens to 100% of the VMFS volumes we presented to our Windows hosts. Upgrade or fresh install of the DSM/MPIO driver did not matter.

How we were able to determine it was a storage issue was by firing up your CMC, and monitor the 'IO Latency Total' statistic.

Or, just try taking a snapshot of a VM on your suspect volume and committing it. If you get odd errors, then you surely are affected. Sometimes, vCenter would report the snapshot DID delete, but if you open up the Datastore Browser, you could clearly see multiple delta files existing for the VM in question.

Hope this helps. Please don't hesitate to ask if you have any other questions.

-Craig

[Ben Drawbaugh]

I just double checked and my vSphere snapshots are deleting. During a deletion, my array is averaging about 1400 IOPS and the average IO Total Latency is 3.7ms.

[Craig A. Liess]

Then you are not experiencing the issue. Trust me, you'd know. :)

[dazzagp]

Do you need to remove the HP MPIO and revert to the MS only?

[bublik]

Craig,

Are you installing DSM/MPIO on vmware guests? As my understanding DSM/MPIO supported only for physical hosts. Or am I mistaking?

Could you provide more detail where DSM/MPIO is installed and which verson of DSM/MPIO

[RonsDavis]

It is also important to remember connecting to a VMware volume with DSM is NOT supported, bugs or not.

[Steve Burkett]

VMware KB posted on this.

KB article 1030129 : HP Lefthand DSMs for MPIO can cause locking and LUN accessibility issues on LeftHand arrays

Basically says you'll get locking errors if your Windows server with the HP Lefthand DSM for MPIO installed can see the raw VMFS LUNs.