Proliant RAID5 failed

[Baggus]

Hi,

On a HP Proliant G6 server one of the 3 (75gb 15k SAS) disks in a RAID5 array failed.

The status of the array went to failed (wasnt given interim option). So I replaced the faulty (amber LED continously on) disk with a 150gb 15k SAS drive that was spare in another server.

Directly after the replacement of the drive I rebooted the server. There were no error messages during POST, even the Smart Array controller reported 1 logical drive and nothing unusual. Right after the POST messages when normally the OS would start, this time the screen went black. I noticed the LEDs on all the disks flashing green roughly once per second (all amber LEDs are off).

The manual clearly states the following:

Do not remove the drive. Removing a drive may terminate the
current operation and cause data loss. The drive is rebuilding, erasing, or it is part of an array that is
undergoing capacity expansion or stripe migration.

How can I know if the server is currently rebuilding or erasing (the latter one would be disasterous)?

Is is alright that all LEDs are flashing green once per second, not just the LED on the drive that I replaced?

I dont have any experience on this issue, and I am deeply worried about my data :'(

ps. I photographed the monitor before rebooting:
http://i.imgur.com/xrNFQlb.jpg
(direct link because of image size)

[Robert_Jewell]

> Is is alright that all LEDs are flashing green once per second

This means that the data rebuild in is progress.

> Directly after the replacement of the drive I rebooted the server.

I am curious as to why?  The replacement of the disk and rebuild of the volume is designed to be performed online.  It sounds like the volume is no longer available, when it should be given a RAID 5 configuration.  Likely, the only option now is allow the rebuild to complete and hope the volume goes back online. Rebooting or resetting the server would also reset the controller which would also interrupt the rebuild process.

-Bob

[Baggus]

Thanks for the reply.

How bad would it be if the rebuild process was interrupted once?
How long should a rebuild operation like this take?

If I listen closely I can hear some disk activity, but it doesn't sound like much.

What scared the **** out of me was that the manual also mentions "erasing" if the green leds flash once per sec.

[Robert_Jewell]

> How bad would it be if the rebuild process was interrupted once?

Well, hope for the best and expect the worse!

> How long should a rebuild operation like this take?

Couple of hours I would guess.

>  mentions "erasing" 

I think a better word would be initializing/formatting; though the effect is the same.

-Bob

[Baggus]

So after waiting hours for it to finish, the server still wouldnt boot up. And now my data is mangled.

I ended up installing Windows 7 on a seperate new drive. I can still see the partitions (although Windows  can't mount them anymore). Running data recovery software right now, will have to rebuild the entire server... :(

[gregersenj]

What went wrong for you?

What did really happen, and what did you do?

When the disk failed, did the OS crash, or did the server keep running, untill you powered it down?

Wich disk did you replace?

When I look at you "screen shot", it say disk in bay 2 is missing!

Dis you replace disk 1 or 3, if so you got a multible disk failure.

Did you "clear" the spare disk before you installed it. If not you may have corrupted the meta data?

Beware, that all RAID configuration and status (meta data), is stored on all disks.

BR

/jag