Around the Storage Block

Accidental VM Deletion of VVol VMs – One click recovery with Nimble Storage vCenter Plugin

Blog written with George Costea - VMware Integration Lead Developer at Nimble Storage

Accidental deletion of VMs from the vCenter UI or deletion of the disk files from datastore is something that can easily turn a good day into a terrible day. An important VM can far too easily get deleted by mistake due to an inadvertent VM rename in the past or multiple clones of VMs in the inventory.

For most VMFS based deployments, admins have a way out of the situation either using storage snapshots or using backup software. With Nimble Storage arrays, most customers enable scheduled snapshots of VMFS datastores. Using the array management UI or the Nimble vCenter plugin, datastores snapshots can easily be scheduled to quiesce the VMs on the datastore using VMware Tools VSS integration or to create storage consistent snapshots.

The scheduled snapshots act as the safety net around accidental deletion of a VM from VMFS datastore. The last storage snapshot can be cloned and presented to the vSphere cluster using the Nimble vCenter Plugin. The VM can then be registered and migrated back to the source datastore.

The situation is quite a bit different with VVol VMs. Each VMDK, including the virtual machine’s configuration directory, maps to a VVol on the underlying storage array.  So when a VMDK is deleted, VMware notifies the storage vendor software to delete the VVol.  How this is implemented is left to the storage vendor.

This might sound complicated but rest assured that with Nimble recovering from an accidental deletion is much simpler.

When creating a VVol VM, storage snapshots can easily be configured for using VM Storage Policies in the vCenter UI. It is highly recommended to configure at least one protection schedule. Local retention and remote replication can also be configured.

VM Storage Policy.png

When a VVol VM gets deleted, vCenter invokes VASA API calls to delete the VM from the storage system and removes it from the vCenter inventory as described above. Since the VVol VM’s disks map to a Nimble volume, the volume, and all its snapshots would get deleted. If the VM snapshots were replicated, those replicas would also get deleted. This would leave no trace of the VVol VM on the storage system!

Obviously, this is not a situation we want to leave any customer in. So, we added several safety nets to make recovery from this scenario simple and instantaneous.

1. Nimble vCenter Plugin provides VM deletion as a value-add option.

When this option is used, a new snapshot is created of the VM just before deletion, and the volumes are offlined instead of deleted. If there was any need for this VM again, there would be no data loss.

Local VMs.png

Delete Confirm.png

2. When a VM is deleted using any other method (vCenter API, API, PowerCLI etc), the volumes backing that VM are offlined on the Nimble array with an expiry time of 72 hours.

This gives enough time for the VM to be recognized as missing and recovered, even if the deletion happened on a Friday.

The VMs can be restored back to the most recent snapshot. Note that VMDKs with no protection schedule settings can also be restored to the latest state of the volume. However, the VM configuration files will reset to the initial state of the virtual machine. This means that any VMDKs added or removed after initial VM creation will no longer be associated with the VM.

3. The deleted VMs are easily accessible in the Nimble vCenter Plugin.

Once deleted, the VMs are removed from the vCenter inventory. But vCenter Plugin provides a list of such VMs similar to a recycle bin in Windows. Time until cleanup in the screenshot below indicates the time at which the VM will be permanently deleted.

Deleted VMs.png

 4. The deleted VM can be recovered in one click!

This operation starts a background task whose progress can be monitored using the vCenter Tasks panel. The volumes are restored to most recent snapshot and the VM is registered back into the vCenter. Power it on and you are good to go!

Undelete Confirm.png


5. For VMs with replication configured, the user can also choose if the volumes should be retained on the replication partner array even if the 72 hour grace period expires on the primary storage system.

This option can be seen in the VM Storage Policy screenshot above (“Delete replicas from the partner”).

6. From the Replicated VMs tab on the vCenter Plugin of the replication partner array, the VM can be claimed and registered back into the vCenter.

These safety nets and the UI workflows simplify and protect the VVol VMs from any accidental deletions, and make it easier to recover from such a situation.

About the Author


VMware Integration Architect at Nimble Storage