MSA2052 slow performance

[shang1983]

Hello,

My MSA2052 have a setup of 18 SSD drives and 20 SAS drives where SSD disk group is configured with RAID5 and SAS drives are configured as RAID6. All the volumes are configured as "no affinity". The firmware version of the storage is VL100P001. The Hosts mapping to Storage are through FC SAN Switches.  

There are some DB VMs reside on the storages. Its quite write and read intensive. I check the SMU on the Tier I/O information at the performance tier there's only 148 IOPS at around 6MB/s. Thats very slow. At the Hosts port IOPS and MB/s also quite low. 

I browse to the performance tab, and select the volumes for the performance on current data. The report comes out also few hundreds of IOPS.   

There's no error or warning on the storage. Any idea why the storage is giving such slow performance? 

[SUBHAJIT KHANBARMAN_1]

In order to troubleshoot Performance issue, there are many factors involved and it's not straight forward task. Some of the best practice to follow can be no hardware issue should exist, firmware need to be up to date, Connected system like Servers, SAN Switch all need to be up to date with driver/firmware as well.

Need to check what is the block size set at the Host and depends on that we should check if you want high IOPs or high throughput. Specifically, the smaller the I/O size, the more I/Os per second (IOPS) the SAN can process. However, the corollary to this is a decrease in throughput (as measured in MB/s). Conversely, as I/O size increases, IOPS decreases but throughput increases. When an I/O gets above a certain size, latency also increases as the time required to transport each I/O increases such that the disk itself is no longer the major influence on latency. 

Typically, workloads can be defined by four categories—I/O size, reads vs. writes, sequential vs. random, and queue depth.
A typical application usually consists of a mix of reads and writes, and sequential and random.
For example, a Microsoft® SQL Server instance running an OLTP type workload might see disk IO that is 8k size, 80 percent read, and 100 percent random.
A disk backup target on the other hand might see disk IO that is 64k or 256K in size, with 90 percent writes and 100 percent sequential.

The type of workload will affect the results of the performance measurement.

Please find the link to get latest Controller firmware for MSA2052 which is VL270R001-01,

https://h41111.www4.hpe.com/storage/msafirmware.html

You can also go through Page no 9 onwards for MSA 2052 quickspec to get more details about performance,

https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00008277enw

As you haven't mentioned what is the Host operating system where MSA volumes presented where you are facing perfomance issue, I could have add more information to improve performance.

Hope this helps!
Regards
Subhajit

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************************

[shang1983]

All the hosts OS are vsphere6.5U2. 

[SUBHAJIT KHANBARMAN_1]

I would suggest to go through the below best practice document,

https://h20195.www2.hpe.com/v2/getpdf.aspx/4aa4-7060enw.pdf

Hope this helps!
Regards
Subhajit

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************************

[PrashantS]

Hi,

Why not use a tool to generate the load and check

http://www.iometer.org/doc/downloads.html

Either it will give performance or throw error somewhere.

Prashant S.

[shang1983]

I log a case to HPE and they say there's more load on this 4 disks as they analyze the disk statistics. But those disks are SSDs, the IOPs are mostly around 370 - 400 IOPS, its still consider low IOPS for SSDs.  HPE also mention there are many IO errors on the Hosts end. What can we check from Hosts end?

# show disk-statistics

Location   Serial Number         Pwr Hrs Bps      IOPS Reads       Writes      Data Read Data Written Lifetime Read Lifetime Written Reset Time

---------------------------------------------------------------------------------------------------------------------------------------------------------------

1.1        ===  7735    826.8KB  49   5421431344  1507500252  287.2TB   54.1TB       0B            0B               2018-03-15 14:18:11

1.2        ===      7735    772.0KB  46   5440882530  1528188074  287.9TB   54.8TB       0B            0B               2018-03-15 14:18:11

1.3        ===              7703    721.4KB  44   5450720796  1535364828  288.0TB   54.9TB       0B            0B               2018-03-15 14:18:11

1.4        ===             7703    768.5KB  47   5426011172  1512886554  287.2TB   54.1TB       0B            0B               2018-03-15 14:18:11

1.5        ===             7703    4685.8KB 377  4477575906  253074870   267.0TB   5343.8GB     0B            0B               2018-03-15 14:18:12

1.6        ===             7703    4895.7KB 383  4474787601  253899699   267.0TB   5394.2GB     0B            0B               2018-03-15 14:18:12

1.7        ===             7703    4857.8KB 405  4478893030  256822817   267.1TB   5444.9GB     0B            0B               2018-03-15 14:18:12

1.8        ===             7703    4670.9KB 379  4476948517  256095994   267.0TB   5372.8GB     0B            0B               2018-03-15 14:18:12

[Radan011]

Hi

We were faced to the same issue with slow disk performace on RAID6 on Hyper-V cluster. Finally we decided to change RAID to RAID10 which is much more faster on MSA2050. I'm afraid there is some bug on MSA which slow down disk performace on RAID6. The disk performace we saw on RAID6 was variable from 6MB/s to 200MB/s. On RAID10 is consistent 1000MB/s (We used Crystaldiskmark)

Radan

[RayHahn]

Shang1983 - did your performance problem with your MSA 2052 ever get resolved?  We're seeing performance problems on our MSA 2052, although we're not using SSDs.  Performance was very high when we first installed it, but after a few months we started having Exchange databases fail over to an Exchange server that was on a different array.  We quickly found out that performance on our MSA 2052 was now totally wacky - sometimes it's very good indeed, and at other times horrifically slow - 40 to 1000 times slower (depending on the benchmark) than the MSA 2050 that we have in our other location.

So far, HPE support has had us reconfigure and rebuild the entire array, add redundant iSCSI connections, and stop using the SSD read-only cache, and we're still having problems.  Sometimes it's blazingly fast, and at other times it's the worst drive array I've ever touched in my 30 years in IT.

So, if yours is fixed, maybe what you did might point me in the direction of a solution for us as well.  Thanks!

[RayHahn]

Radan,

We're having problems with our MSA 2052 in RAID 6 in a way that is similar to what you described when yours was set up with RAID 6, and you said that switching to RAID 10 fixed it for you.  Are things still good for you?  Our MSA 2052 was fine for months until it suddenly turned wacky - sometimes really fast, but far too often it's horrifically slow.  When we make a configuration change it seems to be fast for a while, but then slows down to a small fraction of what it's supposed to be.  I can move all VMs off of one disk group that's slow, save one left for testing, and that disk group will still be dog slow.  Like... dead dog slow.

Unfortunately, switching from RAID 6 to RAID 10 is not a budgetary option for us right now.

[SUBHAJIT KHANBARMAN_1]

@RayHahn .....May I know some details about the MSA2052 that you are using and not satisfied with Performance,

--> What type of drive you are using? How many each type?

--> Are you using both Pools? 

--> How many VDG present in each Pool? I hope both pool contains VDG with different drive types.

--> Are you using Automated Tiering feature?

--> Are you using SSD read-cache?

--> Are you using Volume-affinity feature ?

--> What kind of performance issue you are facing? read or write? Sequential or random?

--> Was there anything running at the time of performance issue such as Scheduled backup, Antivirus, etc...?

--> What is the application you are running where you are facing performance issue?

--> What is Host OS, HBA firmware/driver version?

--> Host and SAN Switch up to date with respect to firmware or driver or any Microsoft Hotfix missing?

--> When you say performance issue what is you are checking and from where? is your expectation more IOPs or more throughput? both can't be achieved at the same time.

Please collect the Performance Statistical data 10 to 20 times (run one set of commands and then wait for 2 minutes) at the time the issue occurs:  

     # show controller-statistics

     # show disk-statistics

     # show host-port-statistics

     # show vdisk-statistics

     # show volume-statistics

Share the stat output here. Make sure any of the serial number of any MSA component shouldn't be visible in public.

You can also check best practice whitepaper and verify your setup,

https://h20195.www2.hpe.com/v2/getpdf.aspx/A00015961ENW.pdf?

Note: RAID 6 and RAID10 can't be compared with respect to performance measure. RAID10 is always give better performance over RAID6.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

[MadS2]

Hi Mr. Khanbarman!

Was a soultion found for this topic? - I am currently fighting the same problem on my MSA 2052.

I am able to copy 7.5 GB in 46 Seconds between two Volumes (Source in Pool B, Destination in Pool A)  and on the next attempt i need 24 minutes for 12 GB in the same direction.

Pool A
2 x SSD Raid1
5 x SFF Raid5
11 x LFF Raid6

Pool B
2 x SSD Raid1
6 x SFF Raid6

On this MSA Pool A is also used as destination for replications coming from another MSA2052.

One the dashboard it seems that the replication prevents my pool A from doing some nice autotireing... SSD and SFF are complete filled up and just on the archive tire space is left. - On my other systems there is always some free space on the different tiers...

Any comments are welcome

Thanks a Lot

  Tobi

Statistics during slow-copy

# show controller-statistics 

Durable ID     CPU Load   Power On Time (Secs)   Bps                IOPS             Reads            Writes           Data Read        Data Written     Num Forwarded Cmds  Reset Time                Total Power On Hours 

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

controller_A   7          8469011                13,9MB             89               733529643        430071675        81,8TB           76,6TB           0                   2022-03-23 14:01:23       28434.61

controller_B   2          8468922                0B                 0                0                0                0B               0B               0                   2022-03-23 14:01:23       28434.68

# show disk-statistics      

Location   Serial Number         Pwr Hrs Bps      IOPS Reads       Writes     Data Read Data Written Lifetime Read Lifetime Written Reset Time                

--------------------------------------------------------------------------------------------------------------------------------------------------------------

1.1        X990A0MXT0PR          20728   29,1MB   74   57971451    335336458  48,9TB    35,6TB       0B            0B               2022-03-23 14:02:36       

1.2        X990A0MYT0PR          20728   29,7MB   74   57971245    335336458  48,9TB    35,6TB       0B            0B               2022-03-23 14:02:36       

1.3        08GL5P8A              47920   1545,2KB 25   299163787   23652571   148,8TB   5821,1GB     0B            0B               2022-03-23 14:02:36       

1.4        08GL8X2A              47968   847,3KB  4    299141589   24258121   148,9TB   5907,9GB     0B            0B               2022-03-23 14:02:36       

1.5        08GL8DDA              47968   1601,0KB 26   297977802   23041902   148,9TB   5886,9GB     0B            0B               2022-03-23 14:02:36       

1.6        08GAVS6A              50538   850,9KB  3    296669338   21540473   148,8TB   5837,3GB     0B            0B               2022-03-23 14:02:36       

1.7        08GB75XA              50534   787,9KB  2    297346456   22432459   148,9TB   5871,7GB     0B            0B               2022-03-23 14:02:36       

1.8        08GKV4BA              47968   0B       0    769258919   813511     50,4TB    17,4MB       0B            0B               2022-03-23 14:02:36       

1.16       WBM4TCMZ              7943    188,6MB  2878 1018677803  756968     66,7TB    17,2MB       0B            0B               2022-03-23 14:02:36       

1.17       0RVAK76A              51791   4134,9KB 11   110576428   73148697   87,4TB    17,8TB       0B            0B               2022-03-23 14:02:46       

1.18       0RVAK7WA              51526   4156,9KB 11   110576225   73148697   87,4TB    17,8TB       0B            0B               2022-03-23 14:02:46       

1.19       WBM4TN8T              7943    157,3MB  251  358568552   26214904   210,9TB   10,1TB       0B            0B               2022-03-23 14:02:46       

1.20       WBM4TPGC              7943    157,3MB  251  358941693   26181596   211,0TB   10,1TB       0B            0B               2022-03-23 14:02:46       

1.21       WBM4TNAJ              7943    157,4MB  252  358805947   25842784   211,2TB   10,1TB       0B            0B               2022-03-23 14:02:46       

1.22       WBM4TPH2              7943    157,4MB  251  358258116   25492438   211,0TB   10,1TB       0B            0B               2022-03-23 14:02:46       

1.23       WBM4T9FH              7943    157,5MB  252  357810459   25441768   210,9TB   10,1TB       0B            0B               2022-03-23 14:02:46       

1.24       WBM4TQBL              7943    157,4MB  252  357933117   25444381   210,9TB   10,1TB       0B            0B               2022-03-23 14:02:46       

2.1        2YGYNN4D              33711   19,5MB   243  1260219514  283359981  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.2        2YGUXVWD              33711   19,5MB   242  1259675480  282724782  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.3        2YGESKGD              33711   18,6MB   220  1259709646  282874891  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.4        VJG3W3AX              46694   18,6MB   209  1259375384  281102447  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.5        VJG3VJ3X              48161   18,6MB   208  1259727152  281567143  326,7TB   19,1TB       0B            0B               2022-03-23 14:02:36       

2.6        VJG3UJBX              48161   18,4MB   207  1259197954  281006273  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.7        VLKZ9WZV              23087   18,7MB   211  1257763353  279443985  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.8        JEHSX80N              17181   18,8MB   230  1259666012  282915742  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.9        VJGA9AWX              23087   18,6MB   209  1258604732  280273090  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.10       R5G09RBV              23114   18,5MB   205  1258838680  280534379  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.11       VLKZ761V              23087   18,6MB   208  1259009655  280661994  326,7TB   19,0TB       0B            0B               2022-03-23 14:02:36       

2.12       JEHWLSZN              17181   0B       0    3418698723  813511     224,0TB   17,4MB       0B            0B               2022-03-23 14:02:36       

--------------------------------------------------------------------------------------------------------------------------------------------------------------

# show host-port-statistics 

Durable ID           Bps                IOPS             Reads            Writes           Data Read        Data Written     Queue Depth      I/O Resp Time    Read Resp Time   Write Resp Time  Reset Time                

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

hostport_A1          7233,0KB           45               733530612        357443283        81,8TB           58,5TB           2                42752            368              49219            2022-03-23 14:01:23       

hostport_A2          0B                 0                0                0                0B               0B               0                0                0                0                2022-03-23 14:01:23       

hostport_A3          23,7MB             93               0                72642712         0B               18,0TB           8                77772            0                77772            2022-03-23 14:01:23       

hostport_A4          0B                 0                0                0                0B               0B               0                0                0                0                2022-03-23 14:01:23       

hostport_B1          0B                 0                0                0                0B               0B               0                0                0                0                2022-03-23 14:01:23       

hostport_B2          0B                 0                0                0                0B               0B               0                0                0                0                2022-03-23 14:01:23       

hostport_B3          0B                 0                0                0                0B               0B               0                0                0                0                2022-03-23 14:01:23       

hostport_B4          0B                 0                0                0                0B               0B               0                0                0                0                2022-03-23 14:01:23       

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[SUBHAJIT KHANBARMAN_1]

As per the output you have shared I can see that hostport_A1  and hostport_A3 both shows very high write response time which indicates definately Performance issue persists somewhere which needs investigation. I can also see that hostport_A1 only getting use for read operation and hostport_A3 not participating in read operation which is putting load on hostport_A1. So multipath configuration with respect to IO operation needs to be verified.

It's possible that replication is happening through hostport_A1 (I am not sure but guessing.) only which is again loading this port along with normal IO operation. 

I don't know which drive belongs to which pool and what all features that you are using in this MSA as part of best practice.

I can see that drive 1.19 to 2.11 shows IOPS more than 200 so you need to check what type of drives are they and what is their safe IOPS value. If they are Midline SAS drives then anything more than 75 to 80 IOPs looks very high. This indicates load is high that point of time.

Regarding data copy, I would suggest you to copy same data twice and monitor the statistics both times and then compare both. Every data can be unique in nature so we can't compare two different data and measure performance. Still there are other factors also need to consider like when you start data copy that time how much hostport busy in handling replication traffic.

Performance issue can't be troubleshooted in public forum so its always better to log a support case with HPE support and get the troubleshooting done.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************

[MadS2]

Hello Mr. Khanbarman!

Thank you very much for the detailed reply. - I think we have directly found my problem...

Disks 2.1 to 2.12 are Midline SAS, so the IOPS are to high.

I think my major mistake is not using "the power of two"... i've got:

5 SAS in Raid 5 / 1.3 to 1.7 --> ok
6 SAS in Raid 5 / 1.19 to 1.24--> not ok
11 MDL in Raid 6 / 2.11 to 2.11 --> not ok

When i got it right i should use:
Raid 5 with 3 / 5 / 9 Disks
Raid 6 with 4 / 6 / 10 Disks
Right?

I think i should correct this an test again before making a support ticket (my fault)

One further question regarding the use of the MDL disks as target for replication and backup: should i choose 10 MDL-Disks in Raid 6 or would i gain noticable performance when using 10 or 12 disks in Raid 10 instead?

Thank you very much

[SUBHAJIT KHANBARMAN_1]

@MadS2 Yes you should follow all best practice in order to claim good performance.

The write performance on RAID10 will be significantly faster than RAID6. As far as deciding whether or not you need the increased fault tolerance, that's more of a business decision than a technical decision.

There are noticeable performance improvements with RAID arrays. Here's the formula:

N = number of drives

RAID10 = N times faster read, (1/2)N times faster write

RAID6 = N-2 times faster read, no write improvement

Please see this RAID performance calculator for more info:

http://www.raid-calculator.com/default.aspx

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************