MSA RAID rebuild

[Victor Matsiyevski]

RAID rebuild after disk replacement is taking 16 hours. Before it was only 4-5 hours, but HP support tells me it's normal. Nothing was changed on a system except the amount of data stored over time. Tnx.

[Sivaramakrishna_1]

The rebuilding time depend on the amount of data. As you are stating amount of data is increase it is command that taking time. I hope we can change the priority of Rebuild under certain operations so that it will complete in less time.

Siva

[Patrick Terlisten]

Hello,

this is a normal behaviour. With more data the rebuild process takes a longer time.

Regards,
Patrick

[Victor Matsiyevski]

Rebuild and expand priorities are set to high, I see that the amount of data is probably the cause ( it doubled if not tripled in recent year or so, previously RAID would rebuild in 4-5 hours). Is that true that the speed should be around 10-15 GB/min?
And what are we talking about here: 300 GB of replaced disk space or the whole RAID size which is 1.6 TB? 300/10 GB/min = 30 min, and 1.6 TB/10 GB/min = 160 min - none of this is close to 16 hours. Thank you.

[wobbe]

Also server load is a big factor. Especially when the rebuild priority is set to low or medium.

[kris rombauts]

The amount of user data stored on the diss does NOT play a role in the rebuild process, the raid controller is simply unaware of any user data at the filesystem level. Simply put, when a disk is replaced the raid controller will always rebuild using all remaining stripes of your RAID configuration available on the good disks which are part of the same raid set.

So if your 1,6 Tbyte raid array has 400 GByte user files/data or 900 GByte data, a disk rebuild will always rebuild 300 Gbyte (assuming a disk is 300 GB in size here per your example).

It is more the fact on how busy the server is with I/O during the rebuild that will play a role i.e.


I have no experience with exact rebuild performance figures on such arrays but what array model are you talking about here and what type of disks (U320 SCSI or SAS 3/6 GB)? So do we deal with MSA1000/MSA1500 or a MSA60/MSA70 with a PCI-x or PCI-e Smartarray i.e. or ??


HTH

regards, Kris

[gregersenj]

The overall answer, is that a rebuild can take as long as 15 min /GB. Or max 15 min /GB.

Rebuild time depends on varius factors.
Drive speed
number of drives in the array
RAID level
Workload
And it is possible, that the amount of data on the drives got an influence.

Unless you've got a high workload on the server, there's no need to increase rebuild priority.
The smart array will use all idle time for rebuild, no matter the setting.
It's a matter of how much of the non idle time it will use for rebuild vs Servicing the host requests.

I've got a customer where a 300 GB 15K rpm FC disk, in a 7+1 RAID 5. A reconstruct takes 16 hours.

Since the rebuild time has increased, that much.
I do recommend that you take a look at the other disks in the array.
It could be a lot of retries on one of the disks.
You can use the System Management Homepage.

BR
/jag

[Victor Matsiyevski]

This is MSA1000 with MSA30 attached, two RAID5 with hot spares (13+1spare disks each array). I don't understand: if it takes 15 min/GB max, it should take at least 75 hrs to rebuild 300 GB - this number doesn't play. The opposite, 15 GB/min, should take about 20 minutes to rebuild 300 GB, doesn't look like true either. I replaced 146 GB 15K Ultra320 SCSI with 300 GB 10 K Ultra320 SCSI.

[gregersenj · Accepted solution]

15 min/GB is the max time it may take.
That is what HP say.

It's most likely a lot faster.

In you case 13 disk RAID 5.
It's 12 read operations, 1 calculate and 1 write operation. For every block it has to reconstruct.

I have allready mentioned som numbers from a big box.
Another case some years ago.
36 GB 15K RPM, 3 disk RAID 5. - Rebuild time 12 minutes.
> 146 GB Disk, nearly 1 hour, in a 3 disk RAID 5.
You have 4 times more disks > 4 - 5 hours.

You most likely got a higher load on the box.
And if the amount of data on each disk also, count in. That will take more time.
So 8 to 16 hours, seem ok to me.

As I also mentioned.
Check the statistics, of the disks.
If all error countes is 0. Then there's no need to worry.

BR
/jag

[kris rombauts]

Victor,

did a short test on a simple RADI1 array with one logical drive in it using 2 * 300 GB disks/10 Krpm U320 , took less than 20 minutes, so is close to the mentioned 15 GB/min.


No load on that array by any server during the rebuild. So i believe, as mentioned ,that how larger the array and how more stripes that need to be read to reconstruct the data,together with host activity, how longer it takes to get your 300 GB rebuild.


Kris

[Victor Matsiyevski]

Thank you all, I got the picture. Last night it took 20 hours, backup was running. And I will check disk stats for errors tonight.

[gregersenj]

You're welcome :)

>Kris.
rebuilding a mirror is a simple copy operation.
I mention real life storage system taking 16 hours on a 300 GB 15K RPM FC disk. Recontructing a 7+1 RAID 5.
The copy back is only 4 hours.

But as you see, many paramters got influence.

BR
/jag