topic MSA2052 slow performance in MSA Storage

MSA2052 slow performance

shang1983 — Sun, 27 Jan 2019 14:17:07 GMT

Hello,

My MSA2052 have a setup of 18 SSD drives and 20 SAS drives where SSD disk group is configured with RAID5 and SAS drives are configured as RAID6. All the volumes are configured as "no affinity". The firmware version of the storage is VL100P001. The Hosts mapping to Storage are through FC SAN Switches.

There are some DB VMs reside on the storages. Its quite write and read intensive. I check the SMU on the Tier I/O information at the performance tier there's only 148 IOPS at around 6MB/s. Thats very slow. At the Hosts port IOPS and MB/s also quite low.

I browse to the performance tab, and select the volumes for the performance on current data. The report comes out also few hundreds of IOPS.

There's no error or warning on the storage. Any idea why the storage is giving such slow performance?

Re: MSA2052 slow performance

SUBHAJIT KHANBARMAN_1 — Sun, 27 Jan 2019 17:04:54 GMT

In order to troubleshoot Performance issue, there are many factors involved and it's not straight forward task. Some of the best practice to follow can be no hardware issue should exist, firmware need to be up to date, Connected system like Servers, SAN Switch all need to be up to date with driver/firmware as well.

Need to check what is the block size set at the Host and depends on that we should check if you want high IOPs or high throughput. Specifically, the smaller the I/O size, the more I/Os per second (IOPS) the SAN can process. However, the corollary to this is a decrease in throughput (as measured in MB/s). Conversely, as I/O size increases, IOPS decreases but throughput increases. When an I/O gets above a certain size, latency also increases as the time required to transport each I/O increases such that the disk itself is no longer the major influence on latency.

Typically, workloads can be defined by four categories—I/O size, reads vs. writes, sequential vs. random, and queue depth.
A typical application usually consists of a mix of reads and writes, and sequential and random.
For example, a Microsoft® SQL Server instance running an OLTP type workload might see disk IO that is 8k size, 80 percent read, and 100 percent random.
A disk backup target on the other hand might see disk IO that is 64k or 256K in size, with 90 percent writes and 100 percent sequential.

The type of workload will affect the results of the performance measurement.

Please find the link to get latest Controller firmware for MSA2052 which is VL270R001-01,

https://h41111.www4.hpe.com/storage/msafirmware.html

You can also go through Page no 9 onwards for MSA 2052 quickspec to get more details about performance,

https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00008277enw

As you haven't mentioned what is the Host operating system where MSA volumes presented where you are facing perfomance issue, I could have add more information to improve performance.

Hope this helps!
Regards
Subhajit

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************************

Re: MSA2052 slow performance

shang1983 — Mon, 28 Jan 2019 06:31:11 GMT

All the hosts OS are vsphere6.5U2.

Re: MSA2052 slow performance

SUBHAJIT KHANBARMAN_1 — Tue, 29 Jan 2019 07:16:38 GMT

I would suggest to go through the below best practice document,

https://h20195.www2.hpe.com/v2/getpdf.aspx/4aa4-7060enw.pdf

Hope this helps!
Regards
Subhajit

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************************

Re: MSA2052 slow performance

PrashantS — Tue, 29 Jan 2019 13:39:50 GMT

Hi,

Why not use a tool to generate the load and check

http://www.iometer.org/doc/downloads.html

Either it will give performance or throw error somewhere.

Prashant S.

Re: MSA2052 slow performance

shang1983 — Tue, 12 Feb 2019 07:00:12 GMT

I log a case to HPE and they say there's more load on this 4 disks as they analyze the disk statistics. But those disks are SSDs, the IOPs are mostly around 370 - 400 IOPS, its still consider low IOPS for SSDs. HPE also mention there are many IO errors on the Hosts end. What can we check from Hosts end?

# show disk-statistics

Location Serial Number Pwr Hrs Bps IOPS Reads Writes Data Read Data Written Lifetime Read Lifetime Written Reset Time

---------------------------------------------------------------------------------------------------------------------------------------------------------------

1.1 === 7735 826.8KB 49 5421431344 1507500252 287.2TB 54.1TB 0B 0B 2018-03-15 14:18:11

1.2 === 7735 772.0KB 46 5440882530 1528188074 287.9TB 54.8TB 0B 0B 2018-03-15 14:18:11

1.3 === 7703 721.4KB 44 5450720796 1535364828 288.0TB 54.9TB 0B 0B 2018-03-15 14:18:11

1.4 === 7703 768.5KB 47 5426011172 1512886554 287.2TB 54.1TB 0B 0B 2018-03-15 14:18:11

1.5 === 7703 4685.8KB 377 4477575906 253074870 267.0TB 5343.8GB 0B 0B 2018-03-15 14:18:12

1.6 === 7703 4895.7KB 383 4474787601 253899699 267.0TB 5394.2GB 0B 0B 2018-03-15 14:18:12

1.7 === 7703 4857.8KB 405 4478893030 256822817 267.1TB 5444.9GB 0B 0B 2018-03-15 14:18:12

1.8 === 7703 4670.9KB 379 4476948517 256095994 267.0TB 5372.8GB 0B 0B 2018-03-15 14:18:12

Re: MSA2052 slow performance

Radan011 — Tue, 15 Oct 2019 17:15:23 GMT

We were faced to the same issue with slow disk performace on RAID6 on Hyper-V cluster. Finally we decided to change RAID to RAID10 which is much more faster on MSA2050. I'm afraid there is some bug on MSA which slow down disk performace on RAID6. The disk performace we saw on RAID6 was variable from 6MB/s to 200MB/s. On RAID10 is consistent 1000MB/s (We used Crystaldiskmark)

Radan

Re: MSA2052 slow performance

RayHahn — Wed, 13 Nov 2019 20:17:17 GMT

Shang1983 - did your performance problem with your MSA 2052 ever get resolved? We're seeing performance problems on our MSA 2052, although we're not using SSDs. Performance was very high when we first installed it, but after a few months we started having Exchange databases fail over to an Exchange server that was on a different array. We quickly found out that performance on our MSA 2052 was now totally wacky - sometimes it's very good indeed, and at other times horrifically slow - 40 to 1000 times slower (depending on the benchmark) than the MSA 2050 that we have in our other location.

So far, HPE support has had us reconfigure and rebuild the entire array, add redundant iSCSI connections, and stop using the SSD read-only cache, and we're still having problems. Sometimes it's blazingly fast, and at other times it's the worst drive array I've ever touched in my 30 years in IT.

So, if yours is fixed, maybe what you did might point me in the direction of a solution for us as well. Thanks!

Re: MSA2052 slow performance

RayHahn — Wed, 13 Nov 2019 20:28:33 GMT

Radan,

We're having problems with our MSA 2052 in RAID 6 in a way that is similar to what you described when yours was set up with RAID 6, and you said that switching to RAID 10 fixed it for you. Are things still good for you? Our MSA 2052 was fine for months until it suddenly turned wacky - sometimes really fast, but far too often it's horrifically slow. When we make a configuration change it seems to be fast for a while, but then slows down to a small fraction of what it's supposed to be. I can move all VMs off of one disk group that's slow, save one left for testing, and that disk group will still be dog slow. Like... dead dog slow.

Unfortunately, switching from RAID 6 to RAID 10 is not a budgetary option for us right now.

Re: MSA2052 slow performance

SUBHAJIT KHANBARMAN_1 — Tue, 01 Feb 2022 08:09:05 GMT

@RayHahn .....May I know some details about the MSA2052 that you are using and not satisfied with Performance,

--> What type of drive you are using? How many each type?

--> Are you using both Pools?

--> How many VDG present in each Pool? I hope both pool contains VDG with different drive types.

--> Are you using Automated Tiering feature?

--> Are you using SSD read-cache?

--> Are you using Volume-affinity feature ?

--> What kind of performance issue you are facing? read or write? Sequential or random?

--> Was there anything running at the time of performance issue such as Scheduled backup, Antivirus, etc...?

--> What is the application you are running where you are facing performance issue?

--> What is Host OS, HBA firmware/driver version?

--> Host and SAN Switch up to date with respect to firmware or driver or any Microsoft Hotfix missing?

--> When you say performance issue what is you are checking and from where? is your expectation more IOPs or more throughput? both can't be achieved at the same time.

Please collect the Performance Statistical data 10 to 20 times (run one set of commands and then wait for 2 minutes) at the time the issue occurs:

# show controller-statistics

# show disk-statistics

# show host-port-statistics

# show vdisk-statistics

# show volume-statistics

Share the stat output here. Make sure any of the serial number of any MSA component shouldn't be visible in public.

You can also check best practice whitepaper and verify your setup,

https://h20195.www2.hpe.com/v2/getpdf.aspx/A00015961ENW.pdf?

Note: RAID 6 and RAID10 can't be compared with respect to performance measure. RAID10 is always give better performance over RAID6.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

Re: MSA2052 slow performance

MadS2 — Wed, 29 Jun 2022 12:58:21 GMT

Hi Mr. Khanbarman!

Was a soultion found for this topic? - I am currently fighting the same problem on my MSA 2052.

I am able to copy 7.5 GB in 46 Seconds between two Volumes (Source in Pool B, Destination in Pool A) and on the next attempt i need 24 minutes for 12 GB in the same direction.

Pool A
2 x SSD Raid1
5 x SFF Raid5
11 x LFF Raid6

Pool B
2 x SSD Raid1
6 x SFF Raid6

On this MSA Pool A is also used as destination for replications coming from another MSA2052.

One the dashboard it seems that the replication prevents my pool A from doing some nice autotireing... SSD and SFF are complete filled up and just on the archive tire space is left. - On my other systems there is always some free space on the different tiers...

Any comments are welcome

Thanks a Lot

Tobi

Statistics during slow-copy

# show controller-statistics

Durable ID CPU Load Power On Time (Secs) Bps IOPS Reads Writes Data Read Data Written Num Forwarded Cmds Reset Time Total Power On Hours

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

controller_A 7 8469011 13,9MB 89 733529643 430071675 81,8TB 76,6TB 0 2022-03-23 14:01:23 28434.61

controller_B 2 8468922 0B 0 0 0 0B 0B 0 2022-03-23 14:01:23 28434.68

# show disk-statistics

Location Serial Number Pwr Hrs Bps IOPS Reads Writes Data Read Data Written Lifetime Read Lifetime Written Reset Time

--------------------------------------------------------------------------------------------------------------------------------------------------------------

1.1 X990A0MXT0PR 20728 29,1MB 74 57971451 335336458 48,9TB 35,6TB 0B 0B 2022-03-23 14:02:36

1.2 X990A0MYT0PR 20728 29,7MB 74 57971245 335336458 48,9TB 35,6TB 0B 0B 2022-03-23 14:02:36

1.3 08GL5P8A 47920 1545,2KB 25 299163787 23652571 148,8TB 5821,1GB 0B 0B 2022-03-23 14:02:36

1.4 08GL8X2A 47968 847,3KB 4 299141589 24258121 148,9TB 5907,9GB 0B 0B 2022-03-23 14:02:36

1.5 08GL8DDA 47968 1601,0KB 26 297977802 23041902 148,9TB 5886,9GB 0B 0B 2022-03-23 14:02:36

1.6 08GAVS6A 50538 850,9KB 3 296669338 21540473 148,8TB 5837,3GB 0B 0B 2022-03-23 14:02:36

1.7 08GB75XA 50534 787,9KB 2 297346456 22432459 148,9TB 5871,7GB 0B 0B 2022-03-23 14:02:36

1.8 08GKV4BA 47968 0B 0 769258919 813511 50,4TB 17,4MB 0B 0B 2022-03-23 14:02:36

1.16 WBM4TCMZ 7943 188,6MB 2878 1018677803 756968 66,7TB 17,2MB 0B 0B 2022-03-23 14:02:36

1.17 0RVAK76A 51791 4134,9KB 11 110576428 73148697 87,4TB 17,8TB 0B 0B 2022-03-23 14:02:46

1.18 0RVAK7WA 51526 4156,9KB 11 110576225 73148697 87,4TB 17,8TB 0B 0B 2022-03-23 14:02:46

1.19 WBM4TN8T 7943 157,3MB 251 358568552 26214904 210,9TB 10,1TB 0B 0B 2022-03-23 14:02:46

1.20 WBM4TPGC 7943 157,3MB 251 358941693 26181596 211,0TB 10,1TB 0B 0B 2022-03-23 14:02:46

1.21 WBM4TNAJ 7943 157,4MB 252 358805947 25842784 211,2TB 10,1TB 0B 0B 2022-03-23 14:02:46

1.22 WBM4TPH2 7943 157,4MB 251 358258116 25492438 211,0TB 10,1TB 0B 0B 2022-03-23 14:02:46

1.23 WBM4T9FH 7943 157,5MB 252 357810459 25441768 210,9TB 10,1TB 0B 0B 2022-03-23 14:02:46

1.24 WBM4TQBL 7943 157,4MB 252 357933117 25444381 210,9TB 10,1TB 0B 0B 2022-03-23 14:02:46

2.1 2YGYNN4D 33711 19,5MB 243 1260219514 283359981 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.2 2YGUXVWD 33711 19,5MB 242 1259675480 282724782 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.3 2YGESKGD 33711 18,6MB 220 1259709646 282874891 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.4 VJG3W3AX 46694 18,6MB 209 1259375384 281102447 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.5 VJG3VJ3X 48161 18,6MB 208 1259727152 281567143 326,7TB 19,1TB 0B 0B 2022-03-23 14:02:36

2.6 VJG3UJBX 48161 18,4MB 207 1259197954 281006273 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.7 VLKZ9WZV 23087 18,7MB 211 1257763353 279443985 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.8 JEHSX80N 17181 18,8MB 230 1259666012 282915742 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.9 VJGA9AWX 23087 18,6MB 209 1258604732 280273090 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.10 R5G09RBV 23114 18,5MB 205 1258838680 280534379 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.11 VLKZ761V 23087 18,6MB 208 1259009655 280661994 326,7TB 19,0TB 0B 0B 2022-03-23 14:02:36

2.12 JEHWLSZN 17181 0B 0 3418698723 813511 224,0TB 17,4MB 0B 0B 2022-03-23 14:02:36

--------------------------------------------------------------------------------------------------------------------------------------------------------------

# show host-port-statistics

Durable ID Bps IOPS Reads Writes Data Read Data Written Queue Depth I/O Resp Time Read Resp Time Write Resp Time Reset Time

hostport_A1 7233,0KB 45 733530612 357443283 81,8TB 58,5TB 2 42752 368 49219 2022-03-23 14:01:23

hostport_A2 0B 0 0 0 0B 0B 0 0 0 0 2022-03-23 14:01:23

hostport_A3 23,7MB 93 0 72642712 0B 18,0TB 8 77772 0 77772 2022-03-23 14:01:23

hostport_A4 0B 0 0 0 0B 0B 0 0 0 0 2022-03-23 14:01:23

hostport_B1 0B 0 0 0 0B 0B 0 0 0 0 2022-03-23 14:01:23

hostport_B2 0B 0 0 0 0B 0B 0 0 0 0 2022-03-23 14:01:23

hostport_B3 0B 0 0 0 0B 0B 0 0 0 0 2022-03-23 14:01:23

hostport_B4 0B 0 0 0 0B 0B 0 0 0 0 2022-03-23 14:01:23

Re: MSA2052 slow performance

SUBHAJIT KHANBARMAN_1 — Wed, 29 Jun 2022 13:33:28 GMT

As per the output you have shared I can see that hostport_A1 and hostport_A3 both shows very high write response time which indicates definately Performance issue persists somewhere which needs investigation. I can also see that hostport_A1 only getting use for read operation and hostport_A3 not participating in read operation which is putting load on hostport_A1. So multipath configuration with respect to IO operation needs to be verified.

It's possible that replication is happening through hostport_A1 (I am not sure but guessing.) only which is again loading this port along with normal IO operation.

I don't know which drive belongs to which pool and what all features that you are using in this MSA as part of best practice.

I can see that drive 1.19 to 2.11 shows IOPS more than 200 so you need to check what type of drives are they and what is their safe IOPS value. If they are Midline SAS drives then anything more than 75 to 80 IOPs looks very high. This indicates load is high that point of time.

Regarding data copy, I would suggest you to copy same data twice and monitor the statistics both times and then compare both. Every data can be unique in nature so we can't compare two different data and measure performance. Still there are other factors also need to consider like when you start data copy that time how much hostport busy in handling replication traffic.

Performance issue can't be troubleshooted in public forum so its always better to log a support case with HPE support and get the troubleshooting done.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

Re: MSA2052 slow performance

MadS2 — Thu, 30 Jun 2022 15:10:42 GMT

Hello Mr. Khanbarman!

Thank you very much for the detailed reply. - I think we have directly found my problem...

Disks 2.1 to 2.12 are Midline SAS, so the IOPS are to high.

I think my major mistake is not using "the power of two"... i've got:

5 SAS in Raid 5 / 1.3 to 1.7 --> ok
6 SAS in Raid 5 / 1.19 to 1.24--> not ok
11 MDL in Raid 6 / 2.11 to 2.11 --> not ok

When i got it right i should use:
Raid 5 with 3 / 5 / 9 Disks
Raid 6 with 4 / 6 / 10 Disks
Right?

I think i should correct this an test again before making a support ticket (my fault)

One further question regarding the use of the MDL disks as target for replication and backup: should i choose 10 MDL-Disks in Raid 6 or would i gain noticable performance when using 10 or 12 disks in Raid 10 instead?

Thank you very much

Re: MSA2052 slow performance

SUBHAJIT KHANBARMAN_1 — Thu, 30 Jun 2022 15:28:38 GMT

@MadS2 Yes you should follow all best practice in order to claim good performance.

The write performance on RAID10 will be significantly faster than RAID6. As far as deciding whether or not you need the increased fault tolerance, that's more of a business decision than a technical decision.

There are noticeable performance improvements with RAID arrays. Here's the formula:

N = number of drives

RAID10 = N times faster read, (1/2)N times faster write

RAID6 = N-2 times faster read, no write improvement

Please see this RAID performance calculator for more info:

http://www.raid-calculator.com/default.aspx

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************