MSA-1500 is an expensive dissapointment

[Gene Leybzon]

MSA-1500 performance is a BIG disappointment
I do not know what others experience with MSA-1500 disk array and HP support has been but mine is one disappointment after another.
This is the story. We purchased MSA-1500 CS with 8 HP 300GB 15k hard drives. Installed, configured and connected to HP DL580G5 connected to MSA 1500 CS
with HP FC1142SR 4Gb PCI-e HBA controller via fiber link. According to HP it is a supported and recommended configuration for Red Head Linux 4. All brand new and shiny, can connect from the server to disk array. I am happy, management - happy, HP representative happy. All until I do a simple test for the read performance:


time dd if=/data/4GBfile
of=/dev/null bs=512
8192016+0 records in
8192016+0 records out

real 1m19.991s
user 0m1.417s
sys 0m8.211s


I am screeching my head – 4 GB in 80 seconds? 50MB/sec? No way!
Naturally check if I did something wrong, play with different file, play with block size – still 50MB/sec. OK, calling HP, creating support case. And fun starts. Every day for the last day I am either asked to something against MSA over serial connection or run diagnostic on the server. Well, I do all of these (still do not know why all of these cannot be done in one day without midnight phone calls, but I am still cool with that). And finally I am told that everything seems to be according to the specs (that I knew all along) we need to PAY for the next level of support if we want them to investigate it farther. What? We should pay HP to fix performance problems in recommended configuration? I expected better from HP.

[John Kufrovich · Accepted solution]

Gene,

Most likely the OS doesn't send enough request to keep the MSA busy.

Attach the cli cable. Start your test. At the cli, type "show taskstats" This is a snapshot of the number of request the MSA is working on. Cycle through it several times. I bet you see no more than 1-2 commands in progress. What this means, the OS isn't sending enough commands to keep the MSA running. Believe me, the MSA can handle more than a single read request.

[Torsten.]

I'm not this familiar with the MSA boxes, but I would do a test with another command like a copy. dd was never a good benchmark.

[Gene Leybzon]

dd command turns to be as good as copy...
For the copy commands it takes even longer to copy the same file to /dev/null

real 2m28.791s
user 0m0.380s
sys 0m7.927s

[Gene Leybzon]

John,
You are right. Number of tasks gauge stay at 1 under the load:
CLI-1> show taskstats

Current task stats for Initiator All and LUN All:
Counter total tasks in set: 1
Executing tasks: 0
Executing tasks proxied: 0
Originated tasks proxied: 0
Head of queue tasks: 0
Simple queue tasks: 0
Ordered queue tasks: 0
Untagged tasks: 0
Dormant tasks: 0

Global tasks stats
Parse running: Yes
Unprocessed HQ commands: 0
Unprocessed non-HQ commands: 1
Enabled commands: 0
Incomming data phases ready: 0
Logical requests ready: 0
Logical data phases ready: 0
Total commands in progress: 1
Total commands not complete: 1
Active initiators:
INITIATOR INDEX QUEUE DEPTH
0000 0512

CLI-1>



Is it an indication of problem with Linux driver on server side?

[Víctor Cespón]

Can you connect a Windows Server to that MSA? We usually use iometer (http://www.iometer.org/) to measure disk performance. It can launch several threads at the same time and use different block sizes.

[Gene Leybzon]

Do not have Windows :(
We are 100% Linux shop

[Torsten.]

Don't know this tool, but there are sources and builts for linux too.

[David Tapper_1]

Hello Gene,

Can you tell us more about your configuration?

Dual Controller?
- Active/Active?
- Controller firmware version?

FCA
- Native Red Hat driver or HP-driver?
- Queue-depth?

Regards,
David

[John Kufrovich]

Gene,
There are two schools of thought. Create a more elaborate read ahead algorithm on the storage system but sacrifice storage bandwidth. Or, have the HOST OS tell us exactly what is needed and burst it. Unfortunately, for reads they don't. One thing you could try but I don't think it will matter much. With the linux 2.6 kernel, you can change the elevator queue. And depending on the 2.6 kernel version you could get finer granularity and use a particular elevator on a per LUN basis.

Trust me, if the OS could burst the reads and the block size is 64k or 128k the read performance of the MSA will be well over 100MB/s. And I'm being conservative
jk