MSA 2060/2062 - VMWare Capped at 600MB?

[MBCIT]

I have a MSA2062 12GB SAS, all FLASH and RAID 10.  When I hook any version of VMWARE up to it I can only get 600MB read/700MB write and it doesn't matter what Version of VMWARE, it cap's at 600/700MB. 

I run the same setup Windows Server 2022 and I get 2675MB read and 2010MB write.

Is there some type of Cap in VMWare or in the MSA when hooked to a VMWare server?

VMWare:  8000 IOPS - MAX

Windows Server 2022:  over 60000 IOPS - MAX

The setups are 100% the same except loading Windows on the boot drive verses loading VMware 6.5/6.7/7.0.

So far, not impressed with the MSA's over my EMC's

[support_s]

System recommended content:

1. Advisory: HPE MSA 1050, 2050, 2052, 1060, 2060 and 2062 Storage - Hosts May Not Have Access to MSA Volumes When Updating to VMware ESXi 7.0 Update 2

2. Advisory: (Revision) HPE MSA 1060/2060/2062 Storage - IN110R001 - Rare Single Controller I/O Stall Could Occur Due to I/O Abort on VMware Write Same I/O

Please click on "Thumbs Up/Kudo" icon to give a "Kudo".

Thank you for being a HPE valuable community member.

[MBCIT]

MSA 2060 SAS  W/Firmware:  IN110P001

HPE Advanced Data Suite - Tiering

dgA01   MSA-DP+    (12)  10k 2.4TB SAS

dgA02   RAID10        (8) 1.92TB SSD SAS

VMWARE 6.5, 6.7 and 7.0 System IOPS:

Same setup running Windows 2022 Server:

If I configure it in RAID 10 and only use all flash, I get 60000 IOPS with Windows 2022.

I have tried the Stock VMware install .iso, I have tried the "Custom" HPE .iso's.  I have tried loading the HPE Smart Array HPCISSS3 Vmware Raid driver too (which only slowed it down more).

I have tried 2 different HP Servers (Proliant DL380 G10/Proliant DL360 G10), (6) different SAS cables.

I expect 600MB/sec out of my MSA 2062 iSCSI Array, but not the MSA 2060 SAS Array.

What am I missing with VMWARE???

[JonPaul]

@MBCIT 
600MB/sec sounds suspiciously like you are only using a single MSA port running @ 6Gb/sec. If so then it is a VMware multipathing issue.
Check the port statistics:
CLI>  show host-port-statistics
Also check the port speeds
CLI>  show ports
You mentioned the HPCISSS3 driver, what SAS HBA are you using?  The E208 and P408 should be using the SmartPQI driver.

[MBCIT]

show ports

Ports Media Target ID Status Speed(A) Health Reason Action
---------------------------------------------------------------------------------
A1 SAS 500c0ff664e4c000 Up 12Gb OK

---------------------------------------------------------------------------------
Ports Media Target ID Status Speed(A) Health Reason Action
---------------------------------------------------------------------------------
B1 SAS 500c0ff664e4c400 Up 12Gb OK

---------------------------------------------------------------------------------

show host-port-statistics

hostport_A1 0B 0 3938676 14195271 258.0GB 930.2GB 0 0 0 0 2022-11-15 08:52:37
hostport_B1 0B 0 192383 800232 12.5GB 52.4GB 0 0 0 0 2022-11-15 08:52:36

I am currently using HPE Smart Array P441 Controller until my P408 comes in.  It is on back order until  10/31/2023 (Yes 2023) due to a Chip Export issue with China. (from what CDW tells me).

They assured me that the P441 is 100% compatible with the MSA2060 SAS before I purchased it.

Is there a specfic driver I should be using for VMWARE?

I have tried every RAID level, drive combination etc... with no avail. 600 MB.  (My MSA 2062 iSCSI gets 800/mb, btw)

Thank you for all the help!!!

[JonPaul]

@MBCIT 
Interesting...
The ports show that the link is @ 12Gb/s
I assume that since the other ports are not shown you are only connecting the one server with the P441 HBA.  The one port should have the ability to run 12Gb/sec on the 4 SAS lanes so a full 48Gb/sec and ~4.8GB/sec.
The P441 is not qualified with the MSA 2060 see HPE SPOCK (Single Point Of Connectivity Knowledge) nor is the P441 supported in Gen10 servers see HPE QuickSpecs for the specific server.
SPOCK:  https://www.hpe.com/storage/spock
The HPE E208 and HPE P408 have had supply chain issues and are still on long lead times.
The Host Port statistics are more interesting during the test but as we would only be looking at a single port they won't tell much.  Things to look for there are: Bps, I/O resp Time and queue depth
Bps - how much data is flowing through the port  (This should roughly match what you are seeing on the host)
I/O resp Time - how long is it taking the system to respond to an I/O (30000 == 30 msec is good but a lot is dependent on your disks)
Queue depth - how many I/Os are being pushed  (depends on workload, sequential should result in <20, random can be in the 100s but again can be influenced by backend)

My guess is that during the VMware run the array will not be showing much stress and that the limit is somewhere else.  One thing that VMware likes is to run an 'Eager Zero fill' for file systems.  This writes Zeros out to every block and can affect performance if make a smaller volume and let that finish does the performance imporve?  The MSA will write out all those Zeros and will then go back and reclaim the zero pages, slowly.
Can you also check the cache configuration?  CLI>  show unwritable-cache 
The cache can be consumed by data intended for volumes which we abruptly removed which can happen in testing.  IF you don't care about the data it is safe to 'clear cache'  IF you think this data may be valuable and there is a volume, disk-group, pool offline then clearing cache will delete data.

[MBCIT]

The ports show that the link is @ 12Gb/s
I assume that since the other ports are not shown you are only connecting the one server with the P441 HBA.  The one port should have the ability to run 12Gb/sec on the 4 SAS lanes so a full 48Gb/sec and ~4.8GB/sec.  Correct, adapters are backordered, we plan to have (4) Servers hooked up to it in the future. 
The P441 is not qualified with the MSA 2060 see HPE SPOCK (Single Point Of Connectivity Knowledge) nor is the P441 supported in Gen10 servers see HPE QuickSpecs for the specific server.  Correct, I did look them up in SPOCK and saw that as well, my vendor said they would indeed work.
SPOCK:  https://www.hpe.com/storage/spock

I did test the P441 in a HPE DL380 G9 Server as well to see if it made a difference and it did not help.

The HPE E208 and HPE P408 have had supply chain issues and are still on long lead times. I purchased one on EBAY today and it should be here by the end of the week, so hopefully we'll have apples-to-apples results.

My guess is that during the VMware run the array will not be showing much stress and that the limit is somewhere else.  One thing that VMware likes is to run an 'Eager Zero fill' for file systems.  This writes Zeros out to every block and can affect performance if make a smaller volume and let that finish does the performance imporve?  The MSA will write out all those Zeros and will then go back and reclaim the zero pages, slowly.

I will gearing up to switch back from Microsoft Hyper-V Server 2022 (which everything works as intended and I get full speed.) So, I'll have VMWARE results for you shortly.

I can tell you for sure from all my tests, that if I use Windows Server 2022/Hyper-V eveything works as intended speed wise.

My end goal is VMWare for the Failover and HA as we also use a HP MSA2062 iSCSI in our Cluster as well for other VM's.

I appreciate all your time and help!!!

[JonPaul]

@MBCIT 
Asked around and got one suggestion about a possible Queue Depth (QD) throttle from VMware side:

During the test, look at the QD of each MSA port, #show host-port-statistics    

If the QD is pegged at 32 then its vmware throttling.

#esxcli storage core device list
Then look at the “No of outstanding IOs with competing worlds:”  for each MSA volume. 
To increase it, use the following command.
#esxcli storage core device set -d msa_volume_id  -O xx

As for the HBA, sorry if I put you in a panic,  I would not expect any issues with the P441.  On the other hand getting support from the standard HPE process would be problemmatic. 

[MBCIT]

During the test, look at the QD of each MSA port, #show host-port-statistics

It doesn't appear to be maxing, the most I have seen is:  1   As you can see it only max's out at 598.4MB / Bps  I do see only Hostport_A1 getting data under load though.  Should it be using both for 1200MB/ Bps?  I did try unplugging HP_A1 to manually fail it over and it transfers at the exact same speed on HP_B1: 599MB /Bps.

I have tried to manually update all HPE nhpsa drivers to the most current and reboot: Same speeds

Manually changed the Path Selection Policy from: VMW_PSP_MRU to VMW_PSP_RR.

Manually changed the iops from 1000 to 1.

[root@localhost:~] esxcli storage nmp device list
naa.600c0ff0006642efe656736301000000
Device Display Name: HPE Serial Attached SCSI Disk (naa.600c0ff0006642efe656736301000000)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; alua_followover=on; action_OnRetryErrors=off; {TPG_id=0,TPG_state=AO}{TPG_id=1,TPG_state=ANO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=iops,iops=1,bytes=10485760,useANO=0; lastPathIndex=0: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba1:C1:T1:L1
Is USB: false

As for the HBA, sorry if I put you in a panic, I would not expect any issues with the P441. On the other hand getting support from the standard HPE process would be problemmatic.  No worries here I have 4 on back order and just got lucky to find a cost effective one.  I plan to switch to my ordered ones when if/when actually get them.

I'm just trying to go production on this array to replace an aging HP Storage 1660 that is quickly running out of space.

Thanks again for all the help!

[MBCIT]

Finally got the HPE Smart Array E208e-p SR 10, updated the Firmware to V5.32 (latest) and all my tests show the same result capped at 600MB/s

I am reaching out to VMWARE now, to see if they see anything.