MSA1500/MSA20 Performance Problems

[Kerry Classen]

I have an MSA 1500 (single controller) with 2 MSA20 disk cabinets full of 250gig SATA disks. I have a HP SAN Switch/8 and the HP/Qlogic 2340 HBA. The MSA1500 is configured with a 2TB Raid 5 LUN assigned to a Windows 2003 server recently upgraded to storport drivers to see if they help performance. My primary application running is Backup to Disk from other servers.

I can currently only get a aggregate throuput of 3 to 5 MB/sec on the MSA LUNS. I have upgraded the Drivers and Firmware on the HBA and the MSA1500 is running a 4.82 firmware. The MSA20's are version 1.24.

The FC switch is basically running an out of the box configuration with just an IP addresss added. No zoning has been setup and all ports are running at their defaults.

When observing LUN performance with PerfMon, the MSA's LUN is regularly at 0% idle and over 50 queued requests (I have seen 250 queued requests). It is truly saturated at less than 5 MB/sec.

For a test I simultaneously assigned another LUN from a different SATA disk based SAN to this same host using the same HBA and switch. I can easily sustain over 40 MB/Sec aggregate read/Write performance with the other SAN while PerfMon is reporting over 50% idle time and no queued requests for this LUN. This "other SAN" is supposed to be a low end system and only operates on 1GB Fiber Channel.

I installed SAN Surfer to check for FC errors. It is showing no errors on the link to the MSA. I would like to reference SNMP performance statistics from the MSA, but this capability is sadly missing from the product.

Obviously the bottleneck is not in the Host, HBA or FC switch since they are all used in both senarios. What could make the MSA1500 perform so poorly by comparison? I must have something setup incorrectly, but I don't know what it might be.

Any feedback is appreciated!

KC

[Kerry Classen]

It turns out we had a failed hard disk, but the system did not detect it or show any alerts in any of the management screens. This worries me a lot about the probability of future un-detected problems having an even more dramatic impact on our production environment.

Users of the MSA1000/1500 should be aware that not all drive failures are detected by the system. An un-detected failure is as big a risk as an administrator just ignoring the warnings.

This issue has cost us over a month of VERY compromised performance and if another drive would have failed, we would have lost over 1.5TB of data.

Shouldn't someone look into why the system never reported the failed drive?

KC

[Kerry Classen]

Update:

Last week we upgraded the firmware on the MSA1500 and MSA20 to the April 07 version 4.94-2 release. Even with this release the failed drive was still undetected but was causing severe performance problems.

This firmware release is no "fix" for this acknowledged "Critical Issue."

[sazp]

Hi Kerry
we have just one MSA20 and we are also
disapointed with its dataflow performance
we bought it in dec.2005
msa20 failed 2 times (march, may)
along with bateries warning.
HP recommended to upgrade firmware
so we did firm.upgrade of msa20
but the write performance fall down to 7MB/s
so we did firm.upgrade of smartarray6402
but write performance is still about 30MB/s.
Product specification declares transfer data rates greater than 100MB/s...
Please HOW did you realize disk failure if the system does not recognize it ?
thank you
unix@sazp.sk

[Greg Carlson]

The MSA 20 and SATA drives are designed for non-mission critical and non-enterprise usage. They are designed to be used for online disk backups for a mission critical deployment.

The MSA1500 page says, "The MSA1500 cs is perfect for those customers looking for less expensive storage solutions using HP SATA enclosures to store data that doesn't require a high degree of I/O performance and/or is viewed as non-mission critical to business operations."

http://h18006.www1.hp.com/products/storageworks/msa1500cs/index.html

Also here is a HP whitepaper documenting SATA's vs SCSI disks and their performance and MTBF: ftp://ftp.compaq.com/pub/products/servers/proliantstorage/drives-enclosures/sata-vs-sas.pdf

Expecting enterprise performance on SATA drives is never stated by HP. If you need higher performance then the MSA30 w/ SCSI drives is what is needed.

For TS the MSA20 however, establish a hyperterminal into the MSA20. Use the same serial cable from the MSA1500/1000 controller w/ the hyperterm connections. There isn't any viewable CLI commands from this connection, however if an error is generated, it will output to the CLI. Then post the output here or call HP support for assist.

HTH,
Greg