MSA2012FC Poor Performance

[Andrew Ide]

Hi, we are running an HP MSA2012FC (dual enhanced version) with 11 300GB disks in a RAID5 as a dedicated storage device for a SQL2005 server configured in an active passive cluster.

Since this device went live we have suffered performance problems where windows indicates requests are not being satisfied by the disk subsystem quick enough and high queues are seen in Perfmon among other counters that normally indicate a disk bottleneck. HP informed me that I can not go by these stats as they do no reflect the true activity on the MSA.

However, the performance displays in the MSA manager indicate figures as low as 1000 I/O's and 3 MB's at the times this is occuring.

HP are adament there is nothing wrong with the device, and the disk array is not being hammered. They asked me to run I/O meter which produced results of around 8000 I/O's and 500MB's on sequential read and write tests.

We have tried our hardest to hammer the array using I/O meter, while our DBA also hammers the disks through SQL to see if we can get the device to log an error to indicate a disk bottleneck but have not been able to do so.

How on earth are you suppost to monitor this device and decide if an array is causing a bottleneck?! Any reccomendations on a third party application would be welcome as HP Support can not give me any answers.

Thanks in advance!!

[Patrick Terlisten]

Hello,

have you updated to the latest windows storport and HBA driver?

Best regards,
Patrick

[Andrew Ide]

We have updated the storport driver to the latest from Microsoft, this was the only thing HP support picked up on.

I can confirm that I am running all the latest drivers from HP.

Andy

[Patrick Terlisten]

Hello,

do you have the same problems with fileservices? How many disks are included in the Vdisks on which are the volumes for your SQL reside?

Best regards,
Patrick

[Andrew Ide]

The MSA is only used for this single SQL server so am unable to comment on file services.

We have just have 11 disks configured in a raid 5, which has 3 virtual disks. One for the quorum, and two for data.

Im aware this isnt the ideal configuration for SQL server, and need to be able to prove the array is not coping so our business can make a decision as to whether another disk shelf will make a difference.

Andy

[Patrick Terlisten]

Hi,

was the IOmeter run against the same disks as SQL or was it another vdisk and volumes?

SQL is doing smal IOs, the logging operation is small sequential write. RAID 5 is not optimal for such workloads (write penalty and short writes). The long queues can be a result of lots of small IOs.

Can you try a IOMeter run with the following parameters: 8 KB IOs, sequential write with two or four workers?

Best regards,
Patrick

[Andrew Ide]

IO Meter was run against the same disks the databases are stored on using 64k read, and 64k write sequential tests as per HP's reccomendation.

The partiton sector alignment is set to 64K as per MS reccomendations, and the array has a 64k stripe.

These resulted in the 8000 I/O's and 500Mb's figures.

I can try it with the 8k test. I have 8 3GH'z CPU's in the node. Is it worth increasing the workers to match?

Andy

[Andrew Ide]

IO Meter results with an 8K sequential write are 5200 I/O's and 40MB's on the main vdisk our SQL servers DB is on.

The 'last queue depth' value on the MSA was 2, and the last sector size read 16.

Andy

[Patrick Terlisten]

Hello,

the 3 MB/s at 1000 IOPS result in a average IO size of 3 KB. So it's hard for me to belive that the SQL is doing 64 KB.

Best regards,
Patrick

[Andrew Ide]

Hi Patrick, could you explain how you would use the stats the MSA gives to figure out how an array was too heavily utilised?

I really dont see how this can be done as the snapshot only occurs every 30 seconds, and the device can output any stats to a log file for charting?

Im dont really understand the values the device gives me too well.

Andy