MSA 1000 - Devastating Performance on SQL Backup to Disk

[Chris Lawrence]

Hello,

We have had our MSA 1000 setup on a 2 node cluster (ML370 G3, Windows 2000 Sp4) for 14 months in production. We have a 19 disk array RAID0+1 for SQL data, a 4 disk array RAID0+1 for SQL Transaction Logs, a 2 disk RAID 1/0+1 for the Quorum drive, and a 6 disk RAID 0 for SQL Disk backups (as well as some file backups for our batch cycle).

We have always seen a serious system degradation during the SQL disk backups, which initially drove us to use the RAID 0 instead of RAID 5 for the backup volume. Due to the nature of our application, the amount of data in the database has actually been reduced over the past 12 months. Over the past weekend, we experienced issues during our SQL backups where the server could not even respond to SNMP polls, causing the monitoring system to page out. The backups completed successfully, however I can't imagine this is an acceptable situation.

I have spent the better part of the last 24 hours reviewing the forums, so I think I need to run a show tech_support and upload the results to get anywhere. Is this something that is safe to do during operational hours?

If it helps at all, the entire RAID 0 is in the 2nd MSA 30, and the remainder of the disk arrays are split between the MSA1000 chassis and the 1st MSA 30. If I need to link my support info in order to get help, please let me know.

[John Kufrovich]

Which HBAs are you using? Emulex or Qlogic.

It is safe to do a show tech_support.

jk

[Chris Lawrence]

Qlogic HBA's. I have attached the show tech_support output, unfortunately it does not seem to provide the same info I have seen on some of the others (specifically negotiated SCSI transfer rate).

Let me know where to go from here if you can, much appreciated.

[John Kufrovich]

Chris,
You really should upgrade your MSA FW to 4.48. We address some write performance problems. One way to know for sure, monitor the MSA LUN you are using for SQL backups using perfmon. physical disk / write bytes/sec. You will have to scale your counter to fit appropriately on the screen. Tell me if you see 10sec pauses. If so, then install 4.48.

[Chris Lawrence]

I figured that would be the first step. Is there any way to find out if the disks in the array are operating at the same speed without interrupting service?

I didn't see 10 second gaps, but I certainly saw gaps, oddly enough I saw 512 bytes being written every few seconds despite there being no activity on that volume.

We also saw the same result (system brownout) during the weekly anti-virus scan, which we have never seen before. We do have the SQL backup destination excluded from the real-time protection, so that is ruled out.

[John Kufrovich]

More likely the cluster accessing the Quorum drive.

Perhaps 10sec is a little long. With the perfmon counter I mentioned. With 4.32 FW and SQL backups, you will see a burst of writes. Then pause, then another burst.

The >show disks cli command displays drive speed. I'm not positive if it was included in 4.32.

I'll have to try out my system in lab and report back.