topic Re: Replication and backup? (VMware VVols) in HPE Nimble Storage Solution Specialists

Replication and backup? (VMware VVols)

k_erik — Thu, 30 Jan 2020 09:06:37 GMT

Hello,

I have had a couple of cases with Nimble support where they are surprised that we both replicate our VMs *and* do backups of them. For me replication is for DR scenarios, and backups are to recover older versions.
For DR I want to be able to spin up the latest version of the VM as quickly as possible and with low amount of data loss. Don't need several versions.
For backups I need history, customers want to go back months in time sometimes. So these have to be pulled off the Nimble array.
We use VMware VVols for our VMs and run NimbleOS 5.1.4.

The problem is that when a VM snapshot is created in vCenter the Nimble arrays puts it into the DR replication queue and all kinds of issues arises. I have logged a request for enhancement to treat vCenter snapshots differently, AS-104088.
Issues we have seen are:
1) Snapshots from vCenter are replicated to downstream array, when vCenter removes it only knows/cares about upstream array. So, there will be a lot of extra snapshots left on the downstream array. Time to live on unmanaged snapshots, introduced in 5.1.x helps here.
2) If the Nimble array is still working on the replication of the snapshot it will report an error back to vCenter, failing to remove it. Since it is VVol vCenter doesn't care anymore, and it has become orphaned. Nimble array doesn't care either since it was triggered by vCenter. We have scheduled a script daily to clean this up.
3) Nimble replication of a VM can sometimes be stuck for days. According to support it is the config vvol. It is temporarily resolved by support restarting replication services but reappears on a handful of VMs every night after backup has been run.

I have had a chat with someone on the RMC team, hoping that HPE native backup software could do Nimble snapshots instead of vCenter snapshots, but no luck. Apparently, it is quite complicated for RMC to use Nimble native snapshots. Since this is VVol, Veeam doesn't have any useful integration either and behaves the same way using vCenter snapshots.

If you are planning to use VMware VVol and to do Nimble replication while also using a third party backup software I urge you to contact support or your HPE representative and add your voice to feature request AS-104088

Re: Replication and backup? (VMware VVols)

mamatadesaiNim — Thu, 30 Jan 2020 19:15:23 GMT

For vVol VMs, when there is a vCenter snap created on the upstream (e.g. created by backup software), and then deleted, NimOS vasa provider will mark it for deletion and it will get reaped (deleted) in 6 hours, from both upstream and downstream arrays. If this is NOT happening, we'd like to know.

No extra deletion scripts should be needed.

Re: Replication and backup? (VMware VVols)

k_erik — Fri, 31 Jan 2020 08:24:11 GMT

Nice feature, though it is not working in our environment. Don't think support is aware of this feature.

Source/upstream array with 4 orphan vCenter snapshots:

Destination/downstream array also with 4 orphan vCenter snapshots:

vCenter shows no snapshots on the VM:

Information about one of the orphaned vCenter snapshots on the 001 array, *snip* is me censoring data. In this case it only contains the config vvol, the vmdk vvols were sucessfully deleted.

Nimble OS $ snapcoll --info vvol-snapcoll0082 --volcoll *snip*-010-138f0810b9cf9d76-0001
Name: vvol-snapcoll0082
Description:
Volume collection name: *snip*-010-138f0810b9cf9d76-0001
Origination group name/ID: *snip*-001
Is replica: No
Is complete: No
Is manual: Yes
Is externally triggered: No
Is unmanaged: No
Replication status: Complete
Replication started: Jan 30 2020 18:17:53
Replication completed: Jan 30 2020 18:20:10
Replication bytes transferred: 171049
Created: Jan 30 2020 18:02:31
Snapshots:
Volume: *snip*-010
Snapshot: vvol-snapcoll0082
Nimble OS $

Re: Replication and backup? (VMware VVols)

k_erik — Fri, 31 Jan 2020 08:53:46 GMT

Another example, here with orphaned VMDKs left in the collection.

No snapshots registered on the VM in vCenter

Source/upstream array with lots of orphaned snapshots:

Destination/downstream array, also lots of snapshots:

Information about the collection, more than 48 hours old:
Nimble OS $ snapcoll --info vvol-snapcoll0190 --volcoll *snip*-280-138f0810b9cf9d76-0002
Name: vvol-snapcoll0190
Description:
Volume collection name: *snip*-138f0810b9cf9d76-0002
Origination group name/ID: *snip*-001
Is replica: No
Is complete: Yes
Is manual: Yes
Is externally triggered: No
Is unmanaged: No
Replication status: Complete
Replication started: Jan 29 2020 05:01:07
Replication completed: Jan 29 2020 05:08:28
Replication bytes transferred: 16029484
Created: Jan 29 2020 05:01:06
Snapshots:
Volume: *snip*-280
Snapshot: vvol-snapcoll0190
Volume: *snip*-280-1.vmdk
Snapshot: vvol-snapcoll0190
Volume: *snip*-280-2.vmdk
Snapshot: vvol-snapcoll0190
Volume: *snip*-280.vmdk
Snapshot: vvol-snapcoll0190
Nimble OS $

Re: Replication and backup? (VMware VVols)

mamatadesaiNim — Fri, 31 Jan 2020 18:55:52 GMT

Thank you for the confirmation. We have logs from Dec 29 from you. Looking into those now.

Re: Replication and backup? (VMware VVols)

mamatadesaiNim — Fri, 31 Jan 2020 21:54:06 GMT

Could you please re-open your support case? Support & Engineering can work with you directly.

Re: Replication and backup? (VMware VVols)

k_erik — Fri, 31 Jan 2020 22:10:27 GMT

Great, thank you so much! I will do that.

topic Re: Replication *and* backup? (VMware VVols) in HPE Nimble Storage Solution Specialists

Replication *and* backup? (VMware VVols)

Re: Replication *and* backup? (VMware VVols)

Re: Replication *and* backup? (VMware VVols)

Re: Replication *and* backup? (VMware VVols)

Re: Replication *and* backup? (VMware VVols)

Re: Replication *and* backup? (VMware VVols)

Re: Replication *and* backup? (VMware VVols)

topic Re: Replication and backup? (VMware VVols) in HPE Nimble Storage Solution Specialists

Replication and backup? (VMware VVols)

Re: Replication and backup? (VMware VVols)

Re: Replication and backup? (VMware VVols)

Re: Replication and backup? (VMware VVols)

Re: Replication and backup? (VMware VVols)

Re: Replication and backup? (VMware VVols)

Re: Replication and backup? (VMware VVols)