Why Microsoft Performance Monitor returns vastly difference results than NImble Performance for CSVs

[Chris_Lionetti]

You may have found that getting Performance Monitoring Logs from your Server and matching them to the Nimble Performance differs greatly. This is actually because the IOPS that a VM produce are not always (or all) satisfied from the Nimble Array, and in many cases can be short-circuited. Below are the most common reasons for this, but not all of these can be overcome. In most cases, the performance of the VM and the performance of the parent Volume are not a 1-to-1 performance relation when using ReFS and CSVs.

Viewing all nodes as a group

Windows Performance Monitor commonly view the performance from the perspective of a single Machine. When you deploy a CSV, it is common for that CSV to be shared among 4 or more physical servers, and for the workloads (or VMs) to be distributed. This means that to gain a true view of the performance of a CSV, you must gather the combined Performance Monitor counters from all nodes of the cluster. The problem is mitigatable since you can direct Performance Monitor to gather the performance data for all nodes and combine them.

Understanding CSV Caching

By default the Windows File System will maintain a cache which can (and commonly is) enlarged using extra system DRAM. As this is one of the least expensive ways to increase VM performance we should expect this as common practice. While the CSV and shared access, each individual workload (VM) is locked (leased) to a single node. That single node will use its local DRAM as a read cache for that VMs files as long as the lease is valid. The act of moving a VM from one node to another node will cause this Cache to be invalidate on the source node, and the cache to slowly repopulate on the target node. All requests that can be satisfied from the READ cache no longer need to be requested from the Nimble target array which in turn increases the performance stats (higher IOPS) for the VM (as seen from Performance Monitor) while decreasing the performance stats (lower IOPS) reported from the Nimble Array perspective.

ReFS Deallocated Blocks

The expectations of NTFS are no longer valid regarding how deallocated blocks work. In the case of NTFS, once a volume has been created, a read operation will always generate a read to the Nimble Array since the MFT (Master File Table) is a simple flat table that refers to a set of LBAs on the Nimble Volume. The newer ReFS (which is the basis for CSVFS) acts as a Allocate-On-Write file system as opposed to a FlatMap employed in NTFS. In a flatmap filesystem (NTFS) when you choose to read a LBA, that read operations is directly converted to a SCSI read operation and the result is returned. The ReFS filesystem maintains a RefCount (Reference Counter) for each block on the drive, and if the Reference Counter equals zero, the ReFS filesystem knows that the block has either been deallocated or never been written to, and can short circuit the read procedure and return all zeros. This is beneficial in a number of ways for ReFS. When a file was deleted in NTFS, the OS would delete the entry in the MFT, but the blocks that represent the file would still exist on the disk in non-zero’ed form which meant that with the correct approach you could request to read those free blocks and obtain data for which you were not meant to see. ReFS solves this problem since the deletion of the file reduces the RefCount to zero and any attempt to read those blocks immediately returns zeros without generating any Nimble Array traffic.

This is most easily encountered using Benchmarks or other tools that commonly read blocks that have never been allocated, and can produce wildly inflated performance results.

ReFS RefCounters and software Snapshots

This also alters the way that Hyper-V creates Snapshots, since Allocate-on-Write behaves very differently than Copy-On-Write (Default for NTFS). The ReFS approach means that creating snapshots doesn’t create more workload, and in the case of Differencing Disks (DiffVHDx) there will be a natural deduplication aspect built into the workload. The combination of Differencing VHDs, ReFS and CSV Caching can more than double the performance some VMs. All of this added performance will NOT be exposed as any extra workload on the Nimble Storage Volumes, in fact you may see a decrease in Nimble Storage Volume performance since more of the workload is satisfied from the upper layers of the solution.

How these features flow thru the VM

The key unexpected behaviour here is that if your VM in running the Benchmarks against a VHD which is housed on the CSV, the VM may see the raw drive (which is really the VM) and may appear to be formatting it using NTFS or leaving it unformatted, but all of these behaviours will now change to what is expected of ReFS. This gives the person running the benchmark tool the false sense that they are running against a raw device and should expect raw device type performance which is not the case.

How to run a benchmark to limit the effect of these ReFS enhancements

If you plan to run a Benchmark against a VHD assigned to a VM, the only way to defeat the RefCount = 0 short circuit is to first write to the entire drive space first. Additionally there exist powershell commands which let you disable the CSV cache. See the PowerShell command (Get-Cluster).BlockCacheSize=0

You can also ensure that you are using a Fixed VHDx, and ensure that no software snapshots are in effect.

I should point out that to follow these actions is NOT recommended in production since these enhancements that ReFS offers actually assist the customers in running more VMs on the same hardware. In essense you are de-tuing your enviornment.

Conclusion

Unfortunately Microsoft publishes very little in regards to how the ReFS file system works and the majority of the publically available insight is via reverse engineering done by security experts for forensic investigation. Additionally your ability to monitor CSVs also depend on your Operating System since very few Perfmon objects exist in the older versions ss such I recommend using Windows Server 2016 and 2019 ONLY. The performance of the VMs should always be faster that the hosting Nimble Array, and you can view this higher added IOPS as bonus efficiencies.