How Nimble Storage Uses VSS Part 3

Chris_Lionetti · ‎03-06-2018

To see Part 1 in this series, see; https://community.hpe.com/t5/Array-Performance-and-Data/How-Nimble-Storage-uses-VSS/td-p/6998526

To see Part 2 in this series, see; https://community.hpe.com/t5/Array-Performance-and-Data/How-Nimble-Storage-uses-VSS-Part-2/m-p/6998620/highlight/true#M1133

How to restore a Single Hyper-V VM on an alternate machine (or site).

The assumption is that you have a Cluster on a site (site A) that is hosting a large number of VMs. The process described below can be used to test the VM recovery procedure on Site A, or it can be used on the al alternate site if we have configured Nimble replication.

There are a few steps we need to accomplish to ensure that failovers can be done without losing ANY configuration that exists for each VM. We need to invoke a configuration Export to ensure that we can later import those VMs, and we want the set of export files to be as small as possible.

The first assumption is that you use a folder to isolate your VM files into specific directories. i.e. If I want to create a new VM Names TestVM1, I would use the following as the base location for that VM.

C:\ClusterStorage\Volume1\TestVM1

This is a good best practice for Hyper-V as otherwise all of your VM configuration files, Snapshot files, etc. are saved conglomerated and identified only by their GUID instead of their VM Owners name. This makes isolation of VMs and VM file cleanup a significant task.

The creation process will install a new subdirectory under that location to place all of the VM files;

C:\ClusterStorage\Volume1\TestVM1\TestVM1\{files go here}

If your existing VM is not in the correct location, you can use the Live Storage Migration process to move these VMs to conform to the standard process. This move process can be completed without downtime.

We will want to create an export of that VM to the location below;

Mkdir C:\ClusterStorage\Volume1\TestVM1\Exports

Export-VM -name TestVM1 -path C:\ClusterStorage\Volume1\TestVM1\Exports

If the VM is an existing VM, this process can be done by waiting for an outage window, then shut down the VM. Detach the VHDx from the VM, take the export, and then re-attach the VHDx and restart the VM. This process only takes minutes of downtime, and a very small amount of space, since the hard drives are not part of the export. If the export is done with the VHDx attached the process requires double the space and the time required to duplicate the VHDx to the export directory.

If the VM is a new VM, simply create the export prior to attaching the first hard drive.

A few pieces of information you will need are the Nimble Volume Name that hosts these VMs. An assumption is made that the Volume Name on ‘Nimble-A’ will match the destination volume name on ‘Nimble-B’ which is also best practice in the Replication Guide in Infosight.

Commands you will need to find out the Volume Name on Host-A1 that matches C:\ClusterStorage\Volume1.

Get-NimVolume | format-table WindowsVolumes,NimbleVolumename

Let’s assume that I want to import a VM on Site-B without bringing the Site-A VM down, or without interfering with the ongoing replication between Nimble-A and Nimble-B. In the next session I will show how you can use the replication feature to actually failover between site A and site B, but the failover process is much simpler than the more complex auditing process shown in this section.

The additional assumption is also that the Replication is configured to use a Snapshot schedule such that VSS snapshots are occurring naturally every few hours.

From this point on, all commands will be run on Site-B, using either Host-B1 or Host-B2.

At this point I can verify that snapshots on the secondary site exist, and I can choose to select the most recent one only using the following command;

Get-NimSnapShot -NimVolumeName MyNimVol1 -MaxObjects 1

Now both the volume and the snapshot are offline on the target of the replication and cannot be used, however I can choose to create an instant clone of that offline snapshot which can be used and assigned to hosts on Site-B. I would take the above command and pipe it to the Invoke-NimVolumeClone command as such

%previous command% | Invoke-NimVolumeClone -suffix test -addtocsv

You will notice that command gives an option to add a suffix to the VolumeName since you don’t want the test clone to have the same name as the real volume. The command also understands how the local cluster is configured, and will add the storage to both nodes of the Cluster, as well as importing the usable volume to the Cluster as a new CSV.

Once we have created this volume and attached it to the cluster, we must find the directory, which is made much simpler by searching for the suffix we added;

Get-NimVolume | where { $_.NimbleVolumeName -like “*test” }

I can now explore the filesystem and change directory to the right correct VM directory and run the import-VM command.

CD C:\ClusterStorage\Volume2\TestVM1\Exports\

Dir *.vmcx -recurse # this will return the full name of the import config file

The filename for the virtual machine configuration file will not look anything like the name, in fact, it is a long GUID type number similar to the following format; 2714AB2F-74D6-4ABD-9CCC-67B85A4041D0.vmcx.

(dir *.vmcx -recurse).FullName

Import-VM (dir *.vmcx -recurese).fullname

Once the VM is imported you will need to reattach the HDD from the running VM; The key her is that you will be adding the hard drive from the running VM which is not in the exports directory.

Add-VmhardDiskDrive -vmname TestVM1 -path C:\ClusterStorage\Volume2\TestVM1\TestVM1\TestVM1.vhdx

Now that the VM has the correct HDD, you can start the VM. Note that the VM in running from the wrong directory since the import command leaves the configuration info in the wrong location. Now we need to delete all of the files from the correct location to clear the way to move the VM.

Dir -recurse | where { -not ($_.name -like “*.vhdx”) } | remove-item

Move-vmstorage -name TestVM1 -virtualmachinepath C:\ClusterStorage\Volume2\TestVM1\TestVM1

Move-vmstorage -name TestVM1 -SmartPagingFilePath C:\ClusterStorage\Volume2\TestVM1\TestVM1

Move-vmstorage -name TestVM1 -SnapshotFilePath C:\ClusterStorage\Volume2\TestVM1\TestVM1

Now that the VM is running on the local machine, and in the right location, we need to add it as a clustered resource;

Get-vm -name TestVM1 | add-clustervirtualmachinerole

To remove a CSV from a Cluster;

Get-clustersharedvolume -name “Cluster Disk 2” | remove-clustersharedvolume

Then to remove the disk from the cluster;

Get-ClusterResource |where {$_.ResourceType -like “Physical Disk”} | Remove-ClusterResource

To remove the disk from the Windows Server;

Get-nimvolume | where {$_.NimbleVolumeName -like “*test”} | remove-nimvolume

To restore multiple VMs on the same CSV

This process is identical to the above process, with the exception that the Invoke-NimCloneVolume command only needs to be run once to import the CSV, and then for following VM importations, that step can be skipped.

Another special note when importing a large number of VMs. Since each VM exists in a separate folder, and are somewhat isolated, once the Invoke-NimCloneVolume command has been run, all of the VM Import operations can be run in parallel. Once the VM is imported and the VHD has been attached, each VM may be started and the entire process should take only minutes. The additional commands to relocate the VMs operational files back to the correct directories using Live Storage Migration can only be performed a few at a time, and as such will take time to complete. If each VM is processed completely serially the process could take hours to complete as each VM would be blocked by the previous step completion.

The steps to complete multiple VM Import operations from a CSV;

Invoke the Clone operation to bring the CSV online.
Import the VM
Attach the VM HDD
Start the VM
Repeat Steps 2-4 until ALL VMs are started
Issue Migration Command on the VM to relocate to proper location
Repeat Step 6 on each VM until all VMs are relocated

In the above process, steps 1 to 5 will take only minutes to complete even with a large number of VMs. Once Step 5 is complete, the environment is up and running and usable. The additional steps 6 and 7 will naturally queue up and will take the time required to move the configuration files including the memory files.

I can now explore the filesystem and change directory to the right correct VM directory and run the import-VM command.

CD C:\ClusterStorage\Volume2\TestVM1\Exports\

Dir *.vmcx -recurse

# this will return the full name of the import config file