raid5 performance

[grobelny]

What is the expected performance of RAID5 solution on Compaq 431 Array with 5 disks in relation to single disk? On Linux I tried "dd if=/dev/zero of=test bs=1M count=1024;sync;" and on single disk it shows 70MB/s, on RAID1 it shows 40MB/s (which is quite disappointing) and on RAID5 only 10MB/s (which is totally unacceptable). I'd surly get better results with software RAID. Am I doing something wrong or it just works this way? Thanks in advance,

[Eric de Lange (MSE)]

Hi Grobelny,

You are testing sequential write performance of the solution, is that what you want to test ?

Raid 5 write performance will be slower because of the parity calculation cycles that RAID 5 has to go through. If write perfromance is what your application needs then stick to RAID1

Cache does not help you either here I think as you are doing a sequential data test.

Does you application use 1M blocks as well ? If not, try matching the test blocksize to what you application will use.

best regards,
-Eric

[grobelny]

You are testing sequential write performance of the solution, is that what you want to test ?
[TG] That's the first test I managed to run. Others will follow. But this one provides a nice comparision between one disk and RAID.

Raid 5 write performance will be slower because of the parity calculation cycles that RAID 5 has to go through. If write perfromance is what your application needs then stick to RAID1
[TG] Ok, I understand that RAID5 needs extra computations but to slow the disk access 7 times is just ridiculous for a hardware array controller.
And in theory RAID1 should affect performance only slightly and not cut transfer almost in half.

Does you application use 1M blocks as well ? If not, try matching the test blocksize to what you application will use.
[TG] The dd test uses 1M blocks. The actual applications will have various characteristics when it comes to disk access.

[TG] The main question is: is this normal for this controller? Maybe I did something wrong with array setup? Maybe one disk is at fault (I haven't checked all of them separately yet)? Maybe one of the cables is wrong? So once again: what is the expected performance of this controller in relation to single disk performance.

[Eric de Lange (MSE)]

Grobelny,

I do not know if this is normal behavior for this controller. I do not have one of these.

Then again, there may be other issues at hand. Are you using using a cable with twisted pairs or just a flat ribbon cable ? Is there termination at the end of the SCSI chain, etc. Are there code updates for the disks, etc, ect. This all depends on how you set the configuration up on your machine.

hth,
-Eric

[grobelny]

The machine is not new, but it is new to me. So I would like to know what can I expect from this hardware and what should I check to get best performance possible. Thanks for the tips.

[TTr]

> on single disk it shows 70MB/s

What kind of disk (Ultra3 , ultra320?) and with what SCSI controller?

What kind of disks do you have connected on the 431? The 431 is a single bus controller so any of the disks in the bus will run at the speed of the slowest disk. This controller came out at aroung 2000 so 8 years later I would not expect much from it.

[grobelny]

> on single disk it shows 70MB/s

What kind of disk (Ultra3 , ultra320?) and with what SCSI controller?
What kind of disks do you have connected on the 431?
[TG] That same Compaq Smart Array 431 and I believe Ultra160 disks (not sure now) on all the tests.

The 431 is a single bus controller so any of the disks in the bus will run at the speed of the slowest disk.
[TG] All the disks are supposed to be the same so all of them should sustain 70MB/s write speed. I'll check if that indeed is the case.

This controller came out at aroung 2000 so 8 years later I would not expect much from it.
[TG] I know it's not new. What is highly suspicious is the fact that transfer rate drops 7 times. Even 8 years ago it would be unacceptable for me. In fact I was very surprised that the disks are so fast. But the controller is a disappointment for me.

[gregersenj]

Looking at the speed of the scsi bus.

The SA431 run Wide Ultra 3 speed = 160 MB/sec

When mixing WU2, WU3 and WU320, unites will run at their native speed, but not faster than the speed of the controller.

If You connect a Wide Ultra device, the entire bus will fallover to WU speed = 40 MB/sec.

If you don't use the correct twisted pair cable, then i won't work at all (For sure if you are using the hot plug drive cages).

Write perfomance is slow on RAID 5.
If you got a 3 drive RAID 5, and you if you write 1 chubk only.
Then the controller has to read, the old parity, and the data of the other disk. modify the parity, write new data and new parity.
With 5 disks:
Read
Read
Read
Read
Modify
Write
Write
This is known as the write penalty.

The Smart Array controllers is known to be faster than its competitors. And is faster than software RAID, for sure.

RAID 1 should give you the write performance of the slowest disk, and the read performance of the fastest disk, or faster.

I'm sorry, I can't pinpoint where you're problem is.
I have seen slow perfomance from a Smart Array, only once. And reinstalling the OS (Windows) fixed it.

[Hein van den Heuvel]

This is all very normal and to be expected behaviour for a 'dumb', no-battery-backed-up-write-back-cache controller.

http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/smartarray431/questionsanswers.html

"Q14. Does the SA-431 support write-back cache?
A14. No, Compaq believes that data integrity is the most important feature of any of our array controller products. Write-back cache is vulnerable to power drops. With higher-end Compaq array controllers (e.g., the SA-5300, SA-and SA-4250 ES), write-back cache is protected by a unique removable battery backed cache daughter board. Since this is a costly feature to implement, standard with higher end array controllers, the SA-431 does not support battery backed write-back cache."

So every write has to go out there... and be waited for. And for a raid 0+1 you will be waiting for both transfers to complete, each competing for the scsi bus. the controller might even be transferrig from the OS buffer twice.

The raid-5 performance suggests that the controller failed to recognize whole-chunk writes and those had to read the old data first before it can calculate the new parity. That would be a super slow down.
1) receive new 2) read old data 3) read old parity 4) calculate new parity) 5) write new data 6) write new data 7) tell OS to send more.

You may want to look for flag/switch in the ACU to overwrite enabling write-back cache... but just 16MB will not last too long.

You may want to retry with a smaller, more application typical write blcok size (try 64K, 128K and 256K?)

You may want to focus on some, repeatable, application based benchmark.
I see many applications with 90% or more reads, and those might just be happy with the raid-5 and very happy with 0+1

hth,
Hein.

[grobelny]

It seems to me like the controller is running in Ultra2 mode (80MB/s) and not Ultra3 mode (160MB/s). I tried to measure performance of two disks at the same time and both were near 40MB/s (alone they are both over 65MB/s). None of these disks is Ultra2 so it cannot be the reason for slowdown. So what else may slow down the scsi bus as if it was Ultra2?

[Michael Steele_2]

Hi gorbelny:

First, using 'dd' may not be the way to test performance. This is because 'dd' is going to start reading from address zero and then sequentially read on up. Chances are everything is contiguous and you're going to get better performance from contiguous and sequential reading than random access, defraggmented, non-sequential reads.

Second, you're only reading and not writing.

So its better to 'cp' a large file in order to avoid both.

Third, I never heard of better writes in raid one over raid five. Raid one is used for better reads.

As for the Compaq 431, check your HBA MHz on the controller. You can get 32 and 64 MB transfer rates.

Verify your firmware for the disk array controllers.

General rule of thumb is raid 5 for database and raid 1 for filesystems, i.e., archive logs.

Finally, this is an entry level disk array for entry level servers. So don't expect much if you've been comparing it to other disk arrays.

[Michael Steele_2]

Note: 'defragmented' should read 'fragmented'.

[Hein van den Heuvel]

Michael wrote> Second, you're only reading and not writing.

Actually... the main topic suggest the test is only concerned about writing.
But thinak for reminding me

grobelny,

You are writing to a file, and involving the file system caches. Those will change teh IO buffer size you picked for dd to what the file system write 'update' or 'sync' feels best.
If you can, please repeat the test for raw device access.

Also, yourself, myself and other replies speculate about raid-5 penalty. Let's see whether the basic speed is there by reading (to of=/dev/null).
Again, best done to a raw device as you don't want the data to come from a teh cache.

gregerseni>> With 5 disks:
Read
Read
Read
Read
Modify
Write
Write
This is known as the write penalty.

I believe this to be incorrect, as I outlined in my reply which was poste pretty much at the same time.

For raid-5 to modify a block in a chunk it does NOT have to read all chuncks.
It just needs the target chunk and the partity chunk. Merge in the new data, calculate the new parity and write both.

So the Raid-5 write penalty stays the same, no mattter how many disks there are.

The Raid-5 recovery and rebuild penalty does increase with the number of disks.

Michael>> Third, I never heard of better writes in raid one over raid five. Raid one is used for better reads.

I did hear that. raid-1 only needs to do 2 writes for a simple partial stripe-update.
raid-5 needs to perform 2 read adn 2 writes, and it has to wait for the data read to be able to do the writes. Thus it is (a lot) slower.

For large sequential reads, raid-5 may outperform raid-1 because there is one extra spindle to do track read ahead.
Yes there are more spindles in the raid-1 case, but there are competing over the same job and the result may be a distortion to read aheads.

Michael>> As for the Compaq 431, check your HBA MHz on the controller. You can get 32 and 64 MB transfer rates.

There may be some confusion here.
The PCI bus can be 33 or 66 Mhz times 64-bit wide for a max peak speed of 266 MB/s
See for example:
http://en.wikipedia.org/wiki/PCI-X
And:
http://h18004.www1.hp.com/products/quickspecs/10427_div/10427_div.html

Hope this helps someone,
Regards,
Hein van den Heuvel

[Hein van den Heuvel]

>> I'd surly get better results with software RAID. Am I doing something wrong or it just works this way? Thanks in advance,

Btw... how come you mention SW raid.
Is that what you are using, or just an alternative being considered.
The replies so far have assumed you used HW assist raid as per 431 offerings.

IF you are using SW raid, then your the oberserved speeds are pretty much those to be expected!

Hein.

[grobelny]

No, I'm not using using SW RAID on this hardware. It was just an observation made on other, quite different machines. I guess that on SA431 alone I wouldn't get better results with SW RAID since the bus seems to be limited to 80 MB/s (why? I don't know). But if I distributed the disks across the other two channels of standard Ultra2 controller I would certainly get better speeds. Which is more complicated, but viable option.

[Guido W!tty]

Hola,
i have some problems with write speeds also.
Done a test with a 7 GB file.

copy from server 1 to server 2

with 100MB lan speed: 12 min. 50 sec
with 1000MB lan speed: 13 min. 42 sec

The free RAM in server 2 start at 1 GB and the server says it needs 5 minutes to copy.
After 6 minutes the free RAM drop to 20 MB!!! and the copy time ingrease to 15 minutes.

After the task was done, the free RAM slowly climed up to 900 MB.

hardware:

2 x :
ML 350 G3
2003 server std SP II (clean install)
1 GB RAM
compaq 431 array controller
5 x 18.2 GB UW 3 SCSI in RAID 5 (4 + spare)
GB switch
newest firmware + drivers

Does enyone know how to speed thinks up?
Or do i need RAID 1 ?

In the future it is gonna be a file server, so i need some speed ;-)

Regards,
Guido