raid5 performance

[grobelny]

What is the expected performance of RAID5 solution on Compaq 431 Array with 5 disks in relation to single disk? On Linux I tried "dd if=/dev/zero of=test bs=1M count=1024;sync;" and on single disk it shows 70MB/s, on RAID1 it shows 40MB/s (which is quite disappointing) and on RAID5 only 10MB/s (which is totally unacceptable). I'd surly get better results with software RAID. Am I doing something wrong or it just works this way? Thanks in advance,

[Eric de Lange (MSE)]

Hi Grobelny,

You are testing sequential write performance of the solution, is that what you want to test ?

Raid 5 write performance will be slower because of the parity calculation cycles that RAID 5 has to go through. If write perfromance is what your application needs then stick to RAID1

Cache does not help you either here I think as you are doing a sequential data test.

Does you application use 1M blocks as well ? If not, try matching the test blocksize to what you application will use.

best regards,
-Eric

[grobelny]

You are testing sequential write performance of the solution, is that what you want to test ?
[TG] That's the first test I managed to run. Others will follow. But this one provides a nice comparision between one disk and RAID.

Raid 5 write performance will be slower because of the parity calculation cycles that RAID 5 has to go through. If write perfromance is what your application needs then stick to RAID1
[TG] Ok, I understand that RAID5 needs extra computations but to slow the disk access 7 times is just ridiculous for a hardware array controller.
And in theory RAID1 should affect performance only slightly and not cut transfer almost in half.

Does you application use 1M blocks as well ? If not, try matching the test blocksize to what you application will use.
[TG] The dd test uses 1M blocks. The actual applications will have various characteristics when it comes to disk access.

[TG] The main question is: is this normal for this controller? Maybe I did something wrong with array setup? Maybe one disk is at fault (I haven't checked all of them separately yet)? Maybe one of the cables is wrong? So once again: what is the expected performance of this controller in relation to single disk performance.

[Eric de Lange (MSE)]

Grobelny,

I do not know if this is normal behavior for this controller. I do not have one of these.

Then again, there may be other issues at hand. Are you using using a cable with twisted pairs or just a flat ribbon cable ? Is there termination at the end of the SCSI chain, etc. Are there code updates for the disks, etc, ect. This all depends on how you set the configuration up on your machine.

hth,
-Eric

[grobelny]

The machine is not new, but it is new to me. So I would like to know what can I expect from this hardware and what should I check to get best performance possible. Thanks for the tips.

[TTr]

> on single disk it shows 70MB/s

What kind of disk (Ultra3 , ultra320?) and with what SCSI controller?

What kind of disks do you have connected on the 431? The 431 is a single bus controller so any of the disks in the bus will run at the speed of the slowest disk. This controller came out at aroung 2000 so 8 years later I would not expect much from it.

[grobelny]

> on single disk it shows 70MB/s

What kind of disk (Ultra3 , ultra320?) and with what SCSI controller?
What kind of disks do you have connected on the 431?
[TG] That same Compaq Smart Array 431 and I believe Ultra160 disks (not sure now) on all the tests.

The 431 is a single bus controller so any of the disks in the bus will run at the speed of the slowest disk.
[TG] All the disks are supposed to be the same so all of them should sustain 70MB/s write speed. I'll check if that indeed is the case.

This controller came out at aroung 2000 so 8 years later I would not expect much from it.
[TG] I know it's not new. What is highly suspicious is the fact that transfer rate drops 7 times. Even 8 years ago it would be unacceptable for me. In fact I was very surprised that the disks are so fast. But the controller is a disappointment for me.

[gregersenj]

Looking at the speed of the scsi bus.

The SA431 run Wide Ultra 3 speed = 160 MB/sec

When mixing WU2, WU3 and WU320, unites will run at their native speed, but not faster than the speed of the controller.

If You connect a Wide Ultra device, the entire bus will fallover to WU speed = 40 MB/sec.

If you don't use the correct twisted pair cable, then i won't work at all (For sure if you are using the hot plug drive cages).

Write perfomance is slow on RAID 5.
If you got a 3 drive RAID 5, and you if you write 1 chubk only.
Then the controller has to read, the old parity, and the data of the other disk. modify the parity, write new data and new parity.
With 5 disks:
Read
Read
Read
Read
Modify
Write
Write
This is known as the write penalty.

The Smart Array controllers is known to be faster than its competitors. And is faster than software RAID, for sure.

RAID 1 should give you the write performance of the slowest disk, and the read performance of the fastest disk, or faster.

I'm sorry, I can't pinpoint where you're problem is.
I have seen slow perfomance from a Smart Array, only once. And reinstalling the OS (Windows) fixed it.

[Hein van den Heuvel]

This is all very normal and to be expected behaviour for a 'dumb', no-battery-backed-up-write-back-cache controller.

http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/smartarray431/questionsanswers.html

"Q14. Does the SA-431 support write-back cache?
A14. No, Compaq believes that data integrity is the most important feature of any of our array controller products. Write-back cache is vulnerable to power drops. With higher-end Compaq array controllers (e.g., the SA-5300, SA-and SA-4250 ES), write-back cache is protected by a unique removable battery backed cache daughter board. Since this is a costly feature to implement, standard with higher end array controllers, the SA-431 does not support battery backed write-back cache."

So every write has to go out there... and be waited for. And for a raid 0+1 you will be waiting for both transfers to complete, each competing for the scsi bus. the controller might even be transferrig from the OS buffer twice.

The raid-5 performance suggests that the controller failed to recognize whole-chunk writes and those had to read the old data first before it can calculate the new parity. That would be a super slow down.
1) receive new 2) read old data 3) read old parity 4) calculate new parity) 5) write new data 6) write new data 7) tell OS to send more.

You may want to look for flag/switch in the ACU to overwrite enabling write-back cache... but just 16MB will not last too long.

You may want to retry with a smaller, more application typical write blcok size (try 64K, 128K and 256K?)

You may want to focus on some, repeatable, application based benchmark.
I see many applications with 90% or more reads, and those might just be happy with the raid-5 and very happy with 0+1

hth,
Hein.

[grobelny]

It seems to me like the controller is running in Ultra2 mode (80MB/s) and not Ultra3 mode (160MB/s). I tried to measure performance of two disks at the same time and both were near 40MB/s (alone they are both over 65MB/s). None of these disks is Ultra2 so it cannot be the reason for slowdown. So what else may slow down the scsi bus as if it was Ultra2?