MSA1000 low perfomance

[Alexander_97]

We purchased HP server DL380G3 and disk storage MSA1000. During application installation and configuration we discovered that it takes too much time to copy a 2Gb file from one logical disk to another(raid10). Disk subsystem test showed a dramatically low performance (~20Mbps). Same tests made on an old server are much better.
We tried to get help from HP business partner and system integrator, HP Service and Support Center. No one could help us or explain problem.
We checked everything: version BIOS, firmware, drivers and connections. We saw posts with same problems -no solutions. Internal DL disks are more than 3 times faster.
External storage 2-raid10 8x72gb, 4x36gb;1- raid1 2x36Gb. Linked to server FC HBA2214.

[Greg Carlson]

Alexander,

Are you running SQL or a database? Are you at 4.32 fw for the MSA, 1.34 bios on the 2214 with driver version 9.0.0.13?
Running single path, dual path? Also do you have a 2/8 or 2/6 switch or the single i/o module. If you have a switch involved, lock down the speed to 2GB from auto-negotiate as auto-negotiate can cause issues also.

Let us know your fw levels..

Ciao,
Greg

[Dean Simiana]

Hi Alexander,
Do you have two additional storage enclosures on your MSA? If so, how are your disks, and array's on these disks distributed?
I have been advised that better performance can be achieved by spreading the disks in an array across all enclosures. I have also experienced performance issues with the MSA under extreme loads, such as Domino mail systems. I did some experiments (not 100% quantitive) but did notice improved performance when the disks are spread around.
I understand it is do to with splitting the I/O over multiple SCSI Busses in the MSA. The controller shelf has two busses (first 7 and second 7 slots) and one for each other enclosure.
You can move the disks without having to rebuild the arrays; although you should make a backup and you must swap disks when the MSA and enclosures are powered off.

[Alexander_97]

Server:
PROC X3.06/533-512 â 2, RAM â 4Gb
Duplex mode.
1 Channel 0,1 2x32Gb RAID1 SYSTEM
2 Channel 0,1,3 3x72Gb RAID5 ARHIVE
FCA 2214 Slot1 (133Mhz)
Storage: MSA1000 (Drive bay numbers)
1,2,3,4,8,9,10,11 8x72Gb RAID10 ndf
5,6,12,13 4x36Gb RAID10 mdf
7,14 2x36Gb RAID1 log
Storage linked to server by Fiber Channel 2m LC/LC Cable

Installed by SMARTSTARTом, during POST writes - optimyzed for W2000 и 2003 server.
Windows2003 Standart (all HF appl),
SQL2000 Enterprise SP3, Navi

[Alexander_97]

Navision 3.70 with SQL option, 1.34 bios on the 2214 with driver version 9.0.0.13.
Single i/o module.
>Running single path, dual path?< ??

[Ted Buis]

The MSA1000 can have either an internal fC switch 7 host ports or a 4 port FC hub. This might make a performance difference. My guess is that the switch would provide a faster solution, particularily if more than one server was connected to the array. So please advise if you have the hub version. Also, often ProLiant servers often are set up with internal hardware mirroring which would be faster than software mirroring in the array. No mirroring is even faster. The cache in the array that helps with random I/Os is of no value to when copying a 2GByte file are any file larger than the cache, because it will fill up before the job is complete. Likely, it would actually slow things down, because there is the extra action of putting the info in cache, then mirroring the write to the second controller for protection against controller failure, and then actually doing the write to disks. However, the array may be faster for smaller random I/Os due to the cache, and data is protected.

[Alexander_97]

MSA: 1- controller, 1- 1port FC adapter. There are no switches or hubs.

[Ted Buis]

Only one controller!!! Not sure how the firmware works, but without a mirror cache, you would need the second controller, your cache may be useless, as it won't report back anything until it is written to disk. That is the default policy for other disk arrays such as AutoRAID and VA arrays. You may be able to change that policy but it is more risky. I don't have time now, but maybe you can check the array manual to put it in immediate report mode. There may even be a setting within Windows (properties on the disk) to set this if I remember correctly. If so, that will improve performance but increase the chances of data loss if power is lost to the array, but not to the computer. Best to have both on the same UPS to mitigate this if you need the performance.

[Uwe Zessin]

Each cache module has two small batteries on-board.

[Alexander_97]

HP supp eng:"- second controller can be placed for redundancy only. It does`t improve perfomance.

[Ted Buis]

Is Windows set up to enable the write cache?

[Alexander_97]

Write caching on disk enabled.

[John Kufrovich]

Alexander,
(~20Mbps), do you mean 20MB/s.

Are you copying from MSA LUN to MSA LUN?

[Alexander_97]

Yes, we tried to copy from-to RAID's10 8&4 disks. 20Mb per second by IOMETER.

[John Kufrovich]

You should expect ~50MB/s.

If you have anti-virus software with realtime protection, you will see performance degradation with copies and backups.

Let me know if it fixes your problem.

[Ted Buis]

Did you try enabling the "advanced performance"? If you have the server and MSA1000 on a UPS you may wish to use that, but regardless, it might be worth testing.

[Alexander_97]

John, we didnâ t install any antiviral software.
All,
as we were told Moscow HP SERVICE â â the measured activities are design features of MSA1000. By the way, you will not find anywhere tech data for MSA with MBs benchmarksâ .
That is true, only one doc where got - 30000IOPs. Try to fin

[John Kufrovich]

Interesting.
I can tell you it is not a design feature of the MSA.

The ~50MB/s I stated was from a actual 2G file D2D copy (MSA to MSA LUN). Both drives were RAID 1+0.

Using a DL380G3, Windows 2003. My 2G copy took 40sec.

Right, 30000 IOPS. All array manufacturers quote IOPS and all IOPS are 100%
cache hits. High MB/s depend on your file size. Small transactions, small MB/s but high IOPS. Large transactions, large MB/s but low IOPS. Databases are small transaction typically 8k.

If you have the time, can we make sure we are comparing the same numbers. Let's create a 2G file. Find a directory for the file. At a cmd prompt Enter the following,
fsutil file createnew c:\yourdir\test.txt 2147483648. Open your clock and just time the copy from one lun to the other.

[Alexander_97]

John,
we did as you told. Found out amazing results - 25 sec., But Perf monitor does not show any READ disk activity. Earlier we copied real database arh file. First we copied it to the source disk, reloaded server and storage (to flush cache) and then started copy to target disk. (see atth). We will try to make our exe once more on weekend. I`ll reply on monday.

[teplov][karatsc][ru]

[Alexander_97]

John,
We did as you told. Found out amazing results - 25 sec., But Perf monitor does not show any READ disk activity. Earlier we copied real database arh file. First we copied it to the source disk, reloaded server and storage (to flush cache) and then started copy to target disk. (see atth). We will try to make our exe once more on weekend. I`ll reply on monday.

[teplov][karatsc][ru]

[John Kufrovich]

That's good news.

Looking at your perf chart, this is not a MSA problem.

Lets begin by looking at your HBA settings.
regedit - locate hp2300 -> devices -> parameters. Tell me what is there.

[Alexander_97]

Hello John,
Are you shure? > "regedit - locate hp2300 -> devices -> parameters". HP2300 it`s printer and it is not connected or configured on the server. So there is nothing in the registry. AS for me: better read manuals, consult friends before buing anything. HP - good brand, but no better than others. Tread closed. Thanks all.

[Alexander_97]

power off!