MSA 2052 - High write response times in Host Port performance

[JustinN1]

We're in the process of proving out our new MSA 2052 and I noticed in the "Performance" tab on the left that the host port performance is showing rather high (for storage) latencies of up to 25ms under "Write Response".  Is this considered normal?  Most of our Nimble (Hybrid) response times are sub 3ms.  This MSA is hosting Hyper-V VMs and none of the VMs are indicating there's any sort of high latency from storage.

Our MSA 2052 is also configured with just a Performance tier and an Archive tier.  Where does new data, such as a new 300GB VHDX being transferred from another storage system, get written to in terms of tiers?  All of them?  Standard?  Our two pools are configured :

Pool A - Performance disk group of 12x 3.2TB SSDs in Raid 6 and an Archive disk group of 12x 12TB MDL LFF disks

Pool B - Performance disk group of 11x 3.2TB SSDs in Raid 6 and an Archive disk group of 11x 12TB MDL LFF disks

[JustinN1]

I was just reminded to look at the best practices guide by HPE support, this is what they directed me to:

To optimize workloads, sequential writes are initially written to the fastest spinning drive tier with free capacity. A random write workload is initially written to the fastest tier with free capacity. In a three-tiered MSA, the random write workload is written to the performance tier.

So in our case it sounds like seqential would be written to archive and random would be written to SSD.  I'm assuming there's no way to override that and have it all written to SSD by default.

[HPSDMike]

Correct, no way to override that behavior (AFAIK).
Your options would be:

1) Separate SSD in one pool and archive in another pool

2) Add a higher performance set of spinning drives as a "standard" tier

Is the period of high sequential writes just a temporary thing due to a migration? Maybe you'll be ok once everything is settled.
Double check to make sure your latency isn't being introduced between the host and controller ports somewhere. Check all your multipathing settings to make sure everything is as best practice indicated. Also, volumes in Pool B may (or may not) perform as good as pool A because those disk groups do not meet the data set size "power of 2 rule." It really depends on the workload.

If you are truly just testing right now then consider recreating your pools with only the SSD and see how that performs. If you still have high latency there then it might point you back to something else in the environment.

[JustinN1]

I could move the VMs to the same pool and use one as a test, but it would take a while.  There's ~20 VMs on each pool atm.  I can't actually see any sort of latency in any of the VMs in regards to storage, which is why it's strange to me that the writes on the host ports are showing latency that high.  When I run Microsoft Diskspd tests I'm getting very good results, and while the test is running the write latency drops to sub 2ms for the host port that's getting the IO from that VM.  It seems like all of that load for those tests is going to the SSD tier.

As far as I can tell everything is performing well, I was just troubled when I saw the latency that high.

[HPSDMike]

Interesting. You may want to consider creating a support case and uploading your logs for analysis. Have these exact hosts shown good consistent performance on other arrays in the past or is it a new set of hosts? Wondering maybe if you have some sort of power saver thing happening. Considering checking your server BIOS settings to ensure the power profile is set to "maximum" or whatever is similar in the make/model server you are using.

[JustinN1]

I had actually opened a support case with HPE and they looked over my logs and saw no issues.  It's just very strange.  When I run disk benchmark tools on the VMs everything comes back great and the latency actually decreases drastically on that statistics page.  The benchmarks seem to be running against the cache judging by the results I get back, which shows the MSA being much faster than our all-flash 3par.

The hosts are currently connected to a 3Par and a pre-HPE Nimble and do not appear to be exhibiting any issues, but the hosts also don't appear to have any issues with the MSA.  It's just the MSA itself that is reporting the write latency, and the only place I see it is in the Host Port performance tab.