Array Failed - Need Help

[BenHill]

I have an ML350 with a 641 Smart Array Controller. I had all drive slots full (6 drives positions 0-5) and they were configured into 3 logical drives, each Raid-1. Then the drive in position 3 failed. That array went into interim recovery. I wanted to replace the array with two larger capacity drives. So, I shutdown the server, removed both disks of the interim recovery mode logical disk, then booted into the config and deleted the logical drive.

I then install my two new disks and all seemed well until I realized that I had backed up the wrong logical disk and didn't have a back up of my failed and now deleted logical disk.

So, not to panic, I shutdown the server, removed the two new disks, booted into the array config, removed the cooresponding logical disk. Then shutdown again and installed the failed logical disks, including both the good and failed disks. Then when I got to the array config, it didn't see anything about my previously failed logical disks. I assumed that since I never deleted the logical disk with the phyisical disks in place that I was ok. What did I do wrong (other than back up the wrong data) and can I do anything about it at this point?

Please help.

[NMory]

Well Ben...I think there is not too much that you can do. From the moment you put the new disk, and deleted the logical drive, all your information about the old drives is lost...That's why when you put the old drives in you get nothing, because to the array controller and all the other disks these disk are not part of the array, and they don't exist anymore. I think that information is stored in the disks.

I have seen things like that happened before, people always end up restoring from tape or if the disks are from the OS, then people end up reintalling the OS.

LN

[steve-o_3 · Accepted solution]

hey there Ben
well, at this point i would call tech support and see what they say. At this point i think you are pretty much up the creek except for a data recovery company, which should be able to get the data without too much trouble.
what you probably should have done was to backup the data on the still good drive from the failed logical drive set, blow the logical drive away, install the 2 newer/larger hdd, config the logical drive, and then restore the data. perhaps you could have put in a larger drive in place of the failed one, let the logical drive rebuild, fail the smaller drive and remove it, put in another bigger drive, and let it rebuild again, but that is a little cheezy and would probably lead to more problems later down the road.
the settings for the raid array should be stored on the hard drives in the array, you should be ok messing around with the settings on the controller, but when you initialize the raid array, it blows everything away

like i said, time to call hp support

steve-0

[BenHill]

Thanks for the replies. Just to document further, here's the update...

To make the situation morely clearly understood, here's where I started:

Disk 0 & 1 - 72gig RAID1 good
Disk 2 & 3 - 36gig RAID1 disk 3 bad, disk 2 interim recovery
Disk 4 & 5 - 36gig RAID1 good

Since my initial message, I inserted disk 2 & 3 only and removed all other drives. Booted, and bam, I see all three logical disks again, and it shows my 2nd RAID1 (disk 2 & 3) in interim recovery.

So, thinking I was in good shape, and having already partitioned the new 300 gig drives, and not having any other drives available, I looked for a way to transfer the data off disk 2 to a USB drive. After booting a few times to the ultimate boot CD and watching it crash, I decided to take disk 2 and 3 to my office where I have an ML350 with a 532 controller.

Initially booted with only disk 2 and 3 and it showed 'interim recovery', so it looked like I was ok. Then after more failed attempts at transferring to a USB drive. I quit getting 'interim recovery' in the controller config. I decided to take a spare 146 gig drive and put it in place of disk 3 (the failed drive) and hoped it would rebuild. It didn't. Both disk 2 and and the new disk 3 sit there with no lights once I exit the controller config. I assume it would start rebuilding immediately, even at the press F1 to boot prompt? So I assume without getting 'interim recovery', I'm not going to get a rebuild.

So, now I look in my controller config and it shows all 3 arrays as failed and when I look at the physical disks for my array in question (disk 2 (the 'good' drive) and disk 3 (the spare 146 gig)) it shows disk 2 as "REPLACEMENT" and disk 3 it doesn't even acknowledge as being there.

I suspect I turned off the server at a bad point in the config util or something, but I don't really know when I would have done that.

I did call HP and I am calling them again now, but I thought it wouldn't hurt to update you in case you could help further.

Ben

[BenHill]

At the risk of talking to myself. I did contact HP and after several calls and downloading ISO images to get a graphical ACU available, it appears that I wasn't actually screwed until later. I was able to get INTERIM RECOVERY status initially, and I should have immediately added a spare drive at that point and let it rebuild. (Something I will never forget now.) But somehow I eithe powered off the server at a bad point, or pulled the good drive out at a bad point and then the status on my 'good' drive changed from 'INTERIM RECOVERY' to 'REPLACEMENT'. Once that happened, I was screwed.

The HP Tier2 guy gave me a good feeling that data recovery would be successful, but I wish there was someway I could change the drive status from REPLACEMENT to 'OK'. Like manually edit the RIS data or something. But I haven't found anything to help in that effort. I can see in the ADU that there is a '02' where I want a '00' to be. I don't know if it is really that simple, but since the RIS is worthless to me now anyway, it would be worth a try. Anybody know anything about that?

I have attached my ADU for anyone interested.

Ben

[BenHill]

Well, no takers. I have sent the drive to Gillware for data recovery. I will update with outcome.

I got my degree from the school of hard knocks this weekend! I hate learning in this kind of situation, but I guess you have to learn where you can.

Ben