MSA 2062FC with Windows 2022 Hyper-V Cluster

[ERoberge]

The problem :

We are currently testing the performance of the new configuration and we realized this :

We setup 2 VM (one on each Hyper-V host) and start an IOMeter to perform an access test to the MSA.

VM A was running on Hyper-V Host HV-04

VM B was running on Hyper-V Host HV-03

We discovered that the VM A was getting more than 40 000 IOPS (Excellent)

And the other VM B was getting only 8 000 IOPS. (Poor performance compared to the other test).

By doing some verifications, we realized that the Host Hyper-V HV-03 was using A LOT of network bandwidth to talk to the Hyper-V HV-04, and if we stopped the IOMeter on VM B, the network bandwidth returned to normal.

The question :

It seems that VM B was writing to the disk using the network to talk with the Host Hyper-V.

Why the VM B on the Hyper-V Host HV-03 was not using the direct HBA Path to write directly to the SAN MSA 2062 FC ?

Our setup :

We are currently configuring a new windows 2022 Hyper-V cluster with 2 nodes with a HP MSA2062 FC with full firmware and driver updates and Windows updates for both servers.

Hardware :

Servers :

HV-04 = DL360 Gen9

HV-03 = DL360 Gen10

Both servers are using  “SN1100Q 16Gb 2p FC HBA” to connect to the SAN.

SAN :

HP MSA 2062 FC with 2 controllers.

How we connected SRV to MSA:

We choose to use a “Direct” connect approach without FC Switch.

HV-04 -> HBA P1 -> MSA CTRL A P1

HV-04 -> HBA P2 -> MSA CTRL B P1

HV-03 -> HBA P1 -> MSA CTRL A P3

HV-03 -> HBA P2 -> MSA CTRL B P3

Ref : https://support.hpe.com/hpesc/public/docDisplay?docId=a00128579en_us&docLocale=en_US

Figure24Hostconnect: directattach-twoservers/oneHBAperserver/dualpath

MSA 2062 FC Setup :

We configured RAID, Volumes and present the disk group to both host on both path of the MSA.

Windows Server configuration :

We added the MPIO windows feature.

We tried to setup the MPIO with the command line “mpclaim –n –I –d “HPE   MSA 2060 FC” but we received “Invalid switch specified”.

Ref : Page 5 -> https://www.hpe.com/psnow/doc/a00105260enw

We decided to go to the Control panel, we opened the MPIO application, and under “Discover Multi-paths” we added the “HPE    MSA 2060 FC” on both servers.

And we added the disk to the Windows Cluster configuration.

When we run the Powershell command “Get-PhysicalDiskStorageNodeView |ft disknumber, ismpioenabled,pathstate,loadbalancepolicy”

Under HV-03 we get this :

DiskNumber : 1
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : True
LoadBalancePolicy : Round Robin with Subset
PathId : {0000000077010000, 0000000077020000}
PathState : {Active/Unoptimized, Active/Optimized}
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

DiskNumber : 0
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : False
LoadBalancePolicy :
PathId :
PathState :
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

DiskNumber : 2
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : True
LoadBalancePolicy : Round Robin with Subset
PathId : {0000000077010000, 0000000077020000}
PathState : {Active/Unoptimized, Active/Optimized}
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

Under HV-04 we get this :

DiskNumber : 1
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : True
LoadBalancePolicy : Round Robin with Subset
PathId : {0000000077000000}
PathState : Active/Optimized
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

DiskNumber : 0
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : True
LoadBalancePolicy : Least Blocks
PathId : {0000000077030001}
PathState : Active/Optimized
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

DiskNumber : 5
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : False
LoadBalancePolicy :
PathId :
PathState :
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

DiskNumber : 6
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : False
LoadBalancePolicy :
PathId :
PathState :
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

DiskNumber : 2
IsPhysicallyConnected : True
HealthStatus : Healthy
OperationalStatus : OK
IsMpioEnabled : True
LoadBalancePolicy : Round Robin with Subset
PathId : {0000000077000000}
PathState : Active/Optimized
StorageNode : MSFT_StorageNode (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)
PhysicalDisk : MSFT_PhysicalDisk (ObjectId = "{1}\\HVCluster02\ROOT/Microsoft/Windows...)

Questions :

#1: Why are we getting the “PathState: Active/Unoptimized”?

Is this the issue with the performance?

#2: How can we be sure that both Host Hyper-V are using their HBA to talk to the MSA?

#3: Is there something missing in the configuration?

Note : We are aware the the server DL360 Gen9 is not supported under Windows 2022, but we got the same problem when we tried with Windows 2019 Server Standard.

Thank you for your help !

[ERoberge]

To add information:

From both Host Hyper-V (HV-03 and HV-04) we added a VHDX drive saved on the Cluster Shared Storage (C:\ClusterStorage\VMs-Storage\Virtual Hard Disks).

From both Host Hyper-V (HV-03 and HV-04) we started a IOMeter performance and we realize that the problem is following the Owner of the shared disks.

On the Hyper-V Host, If we go to the "Failover Cluster Manager" and under "Storage - Disks" and we change the Owner of the VMs-Stroage disk to the other Hyper-V Host (from HV-03 to HV-04), the performance flip to the owner server.

ex: 

If the "Owner Node" of the share disk is the HV-03 :

- From the HV-03, IOMeter will be performing the 40 000 iops

- From the HV-04, IOMeter will be performing only about 8000 iops.

- The HV-03 will receive a lot of bandwidth from HV-04.

If we go to the "Failover Cluster Manager" and we change the "Owner Node" to the HV-04 :

- From the HV-04, IOMeter will be performing the 40 000 iops

- From the HV-03, IOMeter will be performing only about 8000 iops .

- The HV-04 will receive a lot of bandwidth from HV-03.

Why both servers don't read/write to the VHDX drive directly to the SAN?

Microsoft Reference about CSV Storage : 

https://learn.microsoft.com/en-us/windows-server/failover-clustering/failover-cluster-csvs#:~:text=S... 

About I/O synchronization and I/O redirection in CSV communication

I/O synchronization: CSV enables multiple nodes to have simultaneous read-write access to the same shared storage. When a node performs disk input/output (I/O) on a CSV volume, the node communicates directly with the storage, for example, through a storage area network (SAN). However, at any time, a single node (called the coordinator node) "owns" the physical disk resource that is associated with the LUN. The coordinator node for a CSV volume is displayed in Failover Cluster Manager as Owner Node under Disks. It also appears in the output of the Get-ClusterSharedVolume Windows PowerShell cmdlet.

[ERoberge · Accepted solution]

I found the answer !

By executing the PowerShell "Get-ClusterSharedVolumeState" I got the answer !

BlockRedirectedIOReason     : NotBlockRedirected
FileSystemRedirectedIOReason : FileSystemReFs
Name                       : VMs-Storage
Node                       : HV-03
StateInfo                   : FileSystemRedirected
VolumeFriendlyName         : VMs-Storage
VolumeName                 : \\?\Volume{43ec002a-fbb6-4a8a-8ceb-c79d1e8f5371}\

BlockRedirectedIOReason     : NotBlockRedirected
FileSystemRedirectedIOReason : FileSystemReFs
Name                       : VMs-Storage
Node                       : HV-04
StateInfo                   : FileSystemRedirected
VolumeFriendlyName         : VMs-Storage
VolumeName                 : \\?\Volume{43ec002a-fbb6-4a8a-8ceb-c79d1e8f5371}\

Solution :

Reformat the CSV with NTFS !

Reference :

https://learn.microsoft.com/en-us/windows-server/failover-clustering/failover-cluster-csvs 

Cluster Shared Volumes (CSV) enable multiple nodes in a Windows Server failover cluster or Azure Stack HCI to simultaneously have read-write access to the same LUN (disk) that is provisioned as an NTFS volume. The disk can be provisioned as Resilient File System (ReFS); however, the CSV drive will be in redirected mode meaning write access will be sent to the coordinator node.

[Sunitha_Mod]

Hello @ERoberge, 

That's excellent! 

We are glad to know you found the solution and we appreciate you for keeping us updated.