VMware LAN-free backups performance probs / MPIO 9.0 woes

[Craig A. Liess]

Hi Everyone -

Recently, we upgraded the DSM/MPIO driver on our backup server from 8.0 -> 9.0.

Our backup server does a number of LAN-free backups (snapshots are taken via the CLIq, mounted locally for backup, then discarded) in addition to running vRanger Pro 5.0 for backing up ESXi virtual machines.

Our VMFS volumes are also mounted locally to perform LAN-free backups of our virtual machines -- we do this with both iSCSI and FiberChannel-homed VMFS volumes. The performance vs. LAN backups is staggering.

Anyway, after upgrading to the DSM/MPIO v9.0, we began having massive latency issues on our VMFS volumes -- to the tune of 400-600ms. Ultimately, we ended up carving our new volumes and migrating the VMs to them, as it was deemed the volumes were corrupt and could no longer be trusted.

After consulting with both HP + VMware support, it was deemed the 9.0 DSM/MPIO driver was the culprit. Not wanting to corrupt the volumes again, we uninstalled it from our backup server.

This has caused performance of our LAN-free backups to tank.. especially those picking up VMs from our VMFS volumes. We went from 70-80MB/s down to topping out around 40. Our other snapshot-based backups of other NTFS volumes have also suffered a ~25-30% decrease in performance as well.

Has anyone else out there run into a situation like this? What kind of speeds are the rest of you seeing for LAN-free VM backups via iSCSI? Is this a known issue with the 9.0 DSM/MPIO driver?

Any help would be greatly appreciated. Thanks!

-Craig

[Bryan McMullan]

I've no experience with the LAN-free VM backups, but I have had issues with the upgrade of my 8.5 MPIO DSM to 9.0 DSM. For whatever reason a few systems did not take the install correctly and it only presented a single path to our clusters (causing a significant degrading of performance). You could check to ensure you have multiple paths in your initiator.

The only way I found to resolve the issue was to remove the MPIO software, reboot, reinstall the iSCSI software initiator, reboot, install the MPIO/DSM software, and finally reboot.

May be way off base, but thought I'd at least offer it up.

[Craig A. Liess]

Hi Bryan -

Thanks a ton for the response. At this point, we're staying as far away from that MPIO/DSM driver on our backup server as possible. We have confirmed with a separate set of servers + storage that it does indeed corrupt VMFS volumes. We can reproduce it 100% of the time.

Naturally, HP has entered 'radio silence' mode with us regarding this. I expect updated documentation and possibly even a tech bulletin regarding this sooner than later.

Either way, if you haven't begun doing LAN-free backups, you are really missing out -- the performance is staggering vs the traditional LAN method. If you have any questions regarding this, let me know and I'd be happy to assist.

Thanks again for the response.

-Craig

[Bryan McMullan]

Craig,

Guess I was off base....go figure.

What are you using to read the VMFS volume on your backup machine? We currently use the VMWare agent for Backup Exec to backup through the ESX boxes. I'd love to investigate the performance potential of moving away from it.

Hopefully there will be a patch soon...though not too soon, as how I despise upgrading the DSM.

Thanks,

Bryan

[Craig A. Liess]

Bryan,

We simply have our backup server zoned into our fiberchannel and iSCSI-based VMFS volumes. vRanger Pro runs on the backup server and reads the VMFS volumes directly over the SANs. If you open up Disk Manager, you can see the VMFS volumes in there listed as unknown disks.

I regularly see 130MB/s throughput backing up 3 VMs simultaneously via FC.

LAN-free NTFS volume backups (via iSCSI) are a bit more tricky as they require some scripting via the CLI, but the performance boost is very impressive. Additionally, for many cases, it nullifies the need for a software backup agent, which obviously is a nice bonus.

Since we've broken the topic of this thread, feel free to shoot me an email at cliess AT goctii.com. Always happy to help out if you have any questions!

-Craig

[kghammond]

We were using vRanger with the 8.5 DSM/MPIO and it was working great via iSCSI with LAN-free.

We had all of our iSCSI volumes mounted R/O though on the backup server.

Performance was better than over the LAN, but probalby not as spectacular as over fiber. One step away from NDMP.

Anyway, we have been hesitant to try vRanger over iSCSI with 9.0. This is not the first time I have heard about corruption doing this.

Did you mount your iSCSI vollumes R/W or R/O?

I am still trying to figure out if vRanger can corrput a VMFS when it is R/O...

We may just skip this though as we are upgrading EMC Networker and adding Data Domain sometime this year so we are debating about just leaving vRanger in over the LAN mode until it is retired...

[Craig A. Liess]

Previously, we were on 8.0 MPIO/DSM on our backup server, prior to "upgrading" it to 9.0. Based on what you are saying, something clearly changed in 9.0.

Like I said in my previous message, we can kill iSCSI volumes at will with the 9.0 MPIO/DSM driver. We confirmed this with a completely new setup of servers, storage and network gear. I would recommend even against trying unless you are willing to accept the consequences :)

We had them mounted as R/O, just like all of our FC-based VMFS volumes we have mounted on our backup server.

We're 100% ESXi now, so we doing backups via the LAN isn't a realistic option.

If you plan on staying with ESX classic for a bit, then you surely could continue the LAN route..

-Craig

[kghammond]

Yes we are fighting the ESXi transition as long as we can.

Something about being able to "officially" putty into a server and do stuff makes us feel better. :)

And it seems as much as VMware wants to say ESXi is as good or better than ESX, you keep hearing stories of you can't do x, y, or z using ESXi...

Oh well, sooner or later it will be ESXi.

[Craig A. Liess]

We've been 100% ESXi since January and really haven't run into any of those "gotchas" yet. With ESX classic going away, we chose to blow all of our hosts away and install 4.1 from scratch.

Since it's an inevitable thing, we figured that getting it over with sooner than later was the way to go...

-Craig