MSA2052 slow performance

[shang1983]

Hello,

My MSA2052 have a setup of 18 SSD drives and 20 SAS drives where SSD disk group is configured with RAID5 and SAS drives are configured as RAID6. All the volumes are configured as "no affinity". The firmware version of the storage is VL100P001. The Hosts mapping to Storage are through FC SAN Switches.  

There are some DB VMs reside on the storages. Its quite write and read intensive. I check the SMU on the Tier I/O information at the performance tier there's only 148 IOPS at around 6MB/s. Thats very slow. At the Hosts port IOPS and MB/s also quite low. 

I browse to the performance tab, and select the volumes for the performance on current data. The report comes out also few hundreds of IOPS.   

There's no error or warning on the storage. Any idea why the storage is giving such slow performance? 

[SUBHAJIT KHANBARMAN_1]

In order to troubleshoot Performance issue, there are many factors involved and it's not straight forward task. Some of the best practice to follow can be no hardware issue should exist, firmware need to be up to date, Connected system like Servers, SAN Switch all need to be up to date with driver/firmware as well.

Need to check what is the block size set at the Host and depends on that we should check if you want high IOPs or high throughput. Specifically, the smaller the I/O size, the more I/Os per second (IOPS) the SAN can process. However, the corollary to this is a decrease in throughput (as measured in MB/s). Conversely, as I/O size increases, IOPS decreases but throughput increases. When an I/O gets above a certain size, latency also increases as the time required to transport each I/O increases such that the disk itself is no longer the major influence on latency. 

Typically, workloads can be defined by four categories—I/O size, reads vs. writes, sequential vs. random, and queue depth.
A typical application usually consists of a mix of reads and writes, and sequential and random.
For example, a Microsoft® SQL Server instance running an OLTP type workload might see disk IO that is 8k size, 80 percent read, and 100 percent random.
A disk backup target on the other hand might see disk IO that is 64k or 256K in size, with 90 percent writes and 100 percent sequential.

The type of workload will affect the results of the performance measurement.

Please find the link to get latest Controller firmware for MSA2052 which is VL270R001-01,

https://h41111.www4.hpe.com/storage/msafirmware.html

You can also go through Page no 9 onwards for MSA 2052 quickspec to get more details about performance,

https://h20195.www2.hpe.com/v2/getdocument.aspx?docname=a00008277enw

As you haven't mentioned what is the Host operating system where MSA volumes presented where you are facing perfomance issue, I could have add more information to improve performance.

Hope this helps!
Regards
Subhajit

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************************

[shang1983]

All the hosts OS are vsphere6.5U2. 

[SUBHAJIT KHANBARMAN_1]

I would suggest to go through the below best practice document,

https://h20195.www2.hpe.com/v2/getpdf.aspx/4aa4-7060enw.pdf

Hope this helps!
Regards
Subhajit

If you feel this was helpful please click the KUDOS! thumb below!

***********************************************************************************

[PrashantS]

Hi,

Why not use a tool to generate the load and check

http://www.iometer.org/doc/downloads.html

Either it will give performance or throw error somewhere.

Prashant S.

[shang1983]

I log a case to HPE and they say there's more load on this 4 disks as they analyze the disk statistics. But those disks are SSDs, the IOPs are mostly around 370 - 400 IOPS, its still consider low IOPS for SSDs.  HPE also mention there are many IO errors on the Hosts end. What can we check from Hosts end?

# show disk-statistics

Location   Serial Number         Pwr Hrs Bps      IOPS Reads       Writes      Data Read Data Written Lifetime Read Lifetime Written Reset Time

---------------------------------------------------------------------------------------------------------------------------------------------------------------

1.1        ===  7735    826.8KB  49   5421431344  1507500252  287.2TB   54.1TB       0B            0B               2018-03-15 14:18:11

1.2        ===      7735    772.0KB  46   5440882530  1528188074  287.9TB   54.8TB       0B            0B               2018-03-15 14:18:11

1.3        ===              7703    721.4KB  44   5450720796  1535364828  288.0TB   54.9TB       0B            0B               2018-03-15 14:18:11

1.4        ===             7703    768.5KB  47   5426011172  1512886554  287.2TB   54.1TB       0B            0B               2018-03-15 14:18:11

1.5        ===             7703    4685.8KB 377  4477575906  253074870   267.0TB   5343.8GB     0B            0B               2018-03-15 14:18:12

1.6        ===             7703    4895.7KB 383  4474787601  253899699   267.0TB   5394.2GB     0B            0B               2018-03-15 14:18:12

1.7        ===             7703    4857.8KB 405  4478893030  256822817   267.1TB   5444.9GB     0B            0B               2018-03-15 14:18:12

1.8        ===             7703    4670.9KB 379  4476948517  256095994   267.0TB   5372.8GB     0B            0B               2018-03-15 14:18:12

[Radan011]

Hi

We were faced to the same issue with slow disk performace on RAID6 on Hyper-V cluster. Finally we decided to change RAID to RAID10 which is much more faster on MSA2050. I'm afraid there is some bug on MSA which slow down disk performace on RAID6. The disk performace we saw on RAID6 was variable from 6MB/s to 200MB/s. On RAID10 is consistent 1000MB/s (We used Crystaldiskmark)

Radan

[RayHahn]

Shang1983 - did your performance problem with your MSA 2052 ever get resolved?  We're seeing performance problems on our MSA 2052, although we're not using SSDs.  Performance was very high when we first installed it, but after a few months we started having Exchange databases fail over to an Exchange server that was on a different array.  We quickly found out that performance on our MSA 2052 was now totally wacky - sometimes it's very good indeed, and at other times horrifically slow - 40 to 1000 times slower (depending on the benchmark) than the MSA 2050 that we have in our other location.

So far, HPE support has had us reconfigure and rebuild the entire array, add redundant iSCSI connections, and stop using the SSD read-only cache, and we're still having problems.  Sometimes it's blazingly fast, and at other times it's the worst drive array I've ever touched in my 30 years in IT.

So, if yours is fixed, maybe what you did might point me in the direction of a solution for us as well.  Thanks!

[RayHahn]

Radan,

We're having problems with our MSA 2052 in RAID 6 in a way that is similar to what you described when yours was set up with RAID 6, and you said that switching to RAID 10 fixed it for you.  Are things still good for you?  Our MSA 2052 was fine for months until it suddenly turned wacky - sometimes really fast, but far too often it's horrifically slow.  When we make a configuration change it seems to be fast for a while, but then slows down to a small fraction of what it's supposed to be.  I can move all VMs off of one disk group that's slow, save one left for testing, and that disk group will still be dog slow.  Like... dead dog slow.

Unfortunately, switching from RAID 6 to RAID 10 is not a budgetary option for us right now.

[SUBHAJIT KHANBARMAN_1]

@RayHahn .....May I know some details about the MSA2052 that you are using and not satisfied with Performance,

--> What type of drive you are using? How many each type?

--> Are you using both Pools? 

--> How many VDG present in each Pool? I hope both pool contains VDG with different drive types.

--> Are you using Automated Tiering feature?

--> Are you using SSD read-cache?

--> Are you using Volume-affinity feature ?

--> What kind of performance issue you are facing? read or write? Sequential or random?

--> Was there anything running at the time of performance issue such as Scheduled backup, Antivirus, etc...?

--> What is the application you are running where you are facing performance issue?

--> What is Host OS, HBA firmware/driver version?

--> Host and SAN Switch up to date with respect to firmware or driver or any Microsoft Hotfix missing?

--> When you say performance issue what is you are checking and from where? is your expectation more IOPs or more throughput? both can't be achieved at the same time.

Please collect the Performance Statistical data 10 to 20 times (run one set of commands and then wait for 2 minutes) at the time the issue occurs:  

     # show controller-statistics

     # show disk-statistics

     # show host-port-statistics

     # show vdisk-statistics

     # show volume-statistics

Share the stat output here. Make sure any of the serial number of any MSA component shouldn't be visible in public.

You can also check best practice whitepaper and verify your setup,

https://h20195.www2.hpe.com/v2/getpdf.aspx/A00015961ENW.pdf?

Note: RAID 6 and RAID10 can't be compared with respect to performance measure. RAID10 is always give better performance over RAID6.

Hope this helps!
Regards
Subhajit

I am an HPE employee

If you feel this was helpful please click the KUDOS! thumb below!

*************************************************************************