System Administration
cancel
Showing results for 
Search instead for 
Did you mean: 

Proper way to rebuild a mirror With HP Embedded SATA Raid?

 
Halashus
Occasional Visitor

Proper way to rebuild a mirror With HP Embedded SATA Raid?

Hello All,

Ok, so here's the situation, one of my users blew away some very important data in thier home directory, which wasn't backed up. So, we are looking to send the drive off to a data recovery service. So, I thought, since we have the drive mirrored, I would just pop one drive out, have the array rebuild itself and send one drive off for data recovery, keeping the server(Running CentOS 5) online.

Here's the procedure I followed:

1. I popped the drive out, and booted the box up
2. It complained that the array was missing a drive and wanted me to confirm by hitting enter, which I did, it then rebooted
3. I then turned it off and plugged in the new drive and booted it up and went into the Controller manager by pressing f8.
4. I added the new drive as a hotspare and told the arrary to rebuild onto it.
5. After it finished, I rebooted the box and heres where it gets interesting. It came up like I had just installed the OS! The hostname was the same, the root password was the same, but all the data was gone!

Can anyone explain to me why this happened(besides Sunspots) :)

Thanks,
-Hal
7 REPLIES 7
Halashus
Occasional Visitor

Re: Proper way to rebuild a mirror With HP Embedded SATA Raid?

anyone?
Matti_Kurkela
Honored Contributor

Re: Proper way to rebuild a mirror With HP Embedded SATA Raid?

"HP Embedded SATA Raid" is a software-based RAID solution. The hardware part is just a generic SATA controller, with perhaps some very minor hardware support for RAID-specific actions. The main part of the RAID functionality is embedded in system BIOS and the RAID driver.

At boot time, the BIOS detects a manufacturer-specific RAID configuration block on the disks (= what you've set in the Controller manager). As long as BIOS is used to access the disks, the RAID works just fine.

The bootloader uses BIOS, but once the real OS starts up, the system transitions to 32-bit mode and BIOS is put aside. At that point, maintaining the RAID becomes the job of the SATA driver. It should re-read the RAID configuration block and present an appropriate software-based RAID to the rest of the operating system.

In Linux, most SATA drivers just drive the hardware: they don't bother interpreting the manufacturer-specific RAID configuration block and establishing a software RAID at all.

https://ata.wiki.kernel.org/index.php/SATA_RAID_FAQ

You can recognize this is happening if Linux presents your "RAID 1" set as two independent disks. At that point, you have two options:
- you can use standard Linux software RAID tools (mdadm, raidstart/raidstop) to establish a native Linux software RAID on the disks
or
- you can use "dmraid": it's an alternative RAID configuration tool that can read (some) manufacturer-specific RAID configuration blocks and uses the mechanisms of Linux software RAID to implement the RAID configuration.

http://people.redhat.com/~heinzm/sw/dmraid/readme
http://people.redhat.com/~heinzm/sw/dmraid/


My guess about your situation:

Your server may have been originally installed using just one disk (let's call it disk A). After the installation, the system was rebooted to the Controller Manager and disk A was mirrored to disk B. (Or something else was done that caused the BIOS to do a full RAID re-synchronization at that point.)

Now, your system was installed and BIOS indicated you had two disks in sync... and at that point, that was true. But each time Linux started up, it kept using disk A only.

Eventually, your live system remained on disk A, and disk B held a static image of the fresh installation. Since Linux does not know or care about the RAID configuration block unless "dmraid" is used, it never updated the RAID configuration block... so as far as the BIOS knew, the RAID set was still in sync.

Now you've sent disk A to a data recovery service, and are discovering that all along, disk B remained in the "just installed the OS" state.

I'm sorry, but what you assumed was a valid and fully functional RAID 1 configuration... wasn't.

MK
MK
Halashus
Occasional Visitor

Re: Proper way to rebuild a mirror With HP Embedded SATA Raid?

Hi MK,

Thanks for the reply and attempt at theorizing why this happened. It's a good theory and I have spent the last day or so trying to see if any of it fits. However, the problems I see with it are:

a. This one up for debate, because I can't remember 100%, but I'm pretty sure I built the mirror first and then installed the OS onto it, which would obviously poke a hole in your theory.

b. Maybe there's something I don't understand here, but the way I see it, the hardware raid presents a logical drive to the OS, which is happily writing data to it. Shouldn't the hardware raid take responsibility for making sure both sides of the mirror are sync'd? otherwise what the point of buying a hardware raid card? Might as well use the built in software raid in Linux?


Does any of this make sense or am I just grasping at straws?

Thanks for the input,
-Hal
Halashus
Occasional Visitor

Re: Proper way to rebuild a mirror With HP Embedded SATA Raid?

The plot thickens...

As we have about 8 other Linux servers here build with like hardware, we are obviously a little scared right now. Since we have a few identical servers that we recently retired after closing down a remote office, I built one up as a test with the following steps:

1. Booted up into Hardware raid bios utility and created Raid 1 array.
2. Booted off install media for CentOS 5 and installed OS(note: the install only saw only the one logical drive)
3. Booted up and ran through the "first boot after install wizard" to create a new user, setup the firewall, etc and rebooted.
4. Renamed the server and copied a large file to root's home directory.
5. Shutdown the server and removed the secondary drive.
6. Booted the server up and the hardware raid bios complained that the array was in degraded state, made me hit enter to confirm.
7. After boot, just like my other production server, it had reset itself back to the point it had been right after install, presenting me with the "first boot after install wizard". Just like before, after running through the wizard I confirmed that the large file I had copied to root's home directory was not there.
8. Shutdown and removed the primary drive and re-inserted the secondary drive and booted up.
9. Again, just like before, the system booted up fine with the secondary drive, no "first boot after install wizard" and after logging in, I confirmed that the large file was present in root's home directory.


Can anyone explain why the HP Embedded hardware raid isn't doing it's job? Do I need to upgrade some firmware? Is this a known bug?


Thanks,
-Hal
Halashus
Occasional Visitor

Re: Proper way to rebuild a mirror With HP Embedded SATA Raid?

Further wierdness....

Been doing more testing and I have found that no matter which drive I fail, it always rebuilds the array from drive 0, regardless of if drive 0 was the failed drive or not.

Not very happy with HP's embedded Sata Raid at the moment. :(
RaDiOalarm
Occasional Advisor

Re: Proper way to rebuild a mirror With HP Embedded SATA Raid?

A note for the other googlers, who want to rebuild this so called "RAID" (yuck):

 

I just did a rebuild on my 2 year old DL120 G5. It was 2 hdd's in RAID1. The first HDD failed. I replaced it with another one, then according to the manual, went into RAID BIOS, added that new drive as spare, went to array and hit "rebuild". It rebuilt correctly, taking data from second HDD. The OS was pfsense 1.2.3, based on FreeBSD 7.

BTW, it took 3,5 hours for 250gb array. And yes, you cannot rebuild it online, you have to sit in bios screen.

 

My BIOS version was something from around late 2010, the raid bios probably v5.00, but I haven't seen any upgrades for it.

 

P.S. the entire server is full of suprises. for example, it didn't recognize second hdd, until I used a separate power connector. It did not want to start with the original splitted SATA connector.

Jimmy Vance
HPE Pro

Re: Proper way to rebuild a mirror With HP Embedded SATA Raid?

What server model are you dealing with?

 

The embedded SATA controllers that offer RAID mode require a special driver for the RAID functionality. If your not loading the required RAID driver even though the disks are configured as RAID, the OS will treat them as individual SATA disks.

No support by private messages. Please ask the forum! Accept or KudoI work for HPE