ProLiant Servers (ML,DL,SL)
1748123 Members
3473 Online
108758 Solutions
New Discussion юеВ

Mirror Rebuild Proliant ML 350 Gen3

 
Premo Mondone
Occasional Advisor

Mirror Rebuild Proliant ML 350 Gen3

I've got a client with an Proliant ML 350 Gen3 server with an HP Snart Array 641 Controller and Hot-Swap Drives. The system has two drives and was originally configured with those drives being mirrored.

Recently drive "0" failed and the system is now running on drive "1" in "Interim Recovery Mode". The system indicated that drive "0" should be replaced. I ordered a factory replacement for the drive and installed it last night.

There was no indication that I could see that the mirror was rebuilding. I ran ACU and it gave the same error message that it gave on the old drive. I went by to check the server again today, hoping that it had in fact rebuilt the mirror on the new drive, but it has not. The drive led is still orange and when I run ACU, it shows the same thing as before.

I was under the impression that the mirror should automatically rebuild itself without having to do anything. Is there some utility that I need to run to attempt to force the mirror rebuild?

If not, is my only option to back everything up and whack everything and reinstall everything from scratch?
13 REPLIES 13
Steven Clementi
Honored Contributor

Re: Mirror Rebuild Proliant ML 350 Gen3

Yo, Premo! ;o)


By default, the drive should have started the rebuild process on it's own. You can try again by removing the drive and re-inserting it again and see what happens. You can tell if a drive is rebuilding by the flashing green arrow on the drive.

You can also try rebooting the server. Remove the new drive, reboot and after the OS starts loading, insert the new drive again.

There might be an even bigger proplen if the orange/amber light did not turn off perhaps indicating that there is something wrong with the backplane or communication path from the backplane.


Steven
Steven Clementi
HP Master ASE, Storage, Servers, and Clustering
MCSE (NT 4.0, W2K, W2K3)
VCP (ESX2, Vi3, vSphere4, vSphere5, vSphere 6.x)
RHCE
NPP3 (Nutanix Platform Professional)
Premo Mondone
Occasional Advisor

Re: Mirror Rebuild Proliant ML 350 Gen3

Thanks for the reply.......

>You can also try rebooting the server. >Remove the new drive, reboot and after the >OS starts loading, insert the new drive >again.

I tried reinserting the drive and even placing it into a different slot, in case there was a problem with the backplane where the original drive was located.

However, I did NOT try to insert the drive "hot" after the OS loaded. I'll give that a try and see if I get lucky and the drive will hopefully begin to rebuild.

On another note, I don't remember the drive ever flashing green after the system had booted. I figured that would be a good sign, but when all I saw was the solid orange light, I pretty much new I was in trouble.

Thanks again for the advice.
Premo Mondone
Occasional Advisor

Re: Mirror Rebuild Proliant ML 350 Gen3

Well....no luck. I shut down the server, removed the new drive and rebooted the server.

Once the OS was loaded, I inserted the new drive and after about 30 seconds I got the orange light again. The activity led remained solid the entire time and never flickered.

I repeated the procedure, putting the drive into another Hot-Swap bay and the same thing.

Any ideas?
RaMpaNTe
Trusted Contributor

Re: Mirror Rebuild Proliant ML 350 Gen3

Premo! Here is some shots you can try. Laun ACU from windows and select "Array Settings" under rebuild priority set it as "high" then save the changes.

You can then reboot the system with the drive plugged in, during POST you'll receive a message that will say to Press F1 to continue without rebuld or to Press F2 to start the rebuild and accept data lost. You can safelly press F2, the "Data lost" mentioned is NOT the data in the drives IT IS the data in the cache module, the rebuild process should start at that point.

Another thing to try is: with the drive plugged in, run an ADU, save it and post it here so I can analyze it and tell you if the problem is with the backplane, also try updating the FW of the controller, you can do it from windows, (if you are running WK in that box)

Here is a list of the FW updates sorted by OS.

http://h18023.www1.hp.com/support/files/server/us/family/model/4678.html?prodTypeId=15351

Just select your OS and then under FW Storage Controller, select Online ROM Flash Component for OS - Smart Array 641/642

Another thing to do is, shut down the server, and reseat all components such as backplane, controller, drives, and cables.

Good Luck!!!

RaMpaNTe
You heve a question... I have an aswer!!!
Premo Mondone
Occasional Advisor

Re: Mirror Rebuild Proliant ML 350 Gen3

Thanks........I'll give that at try tomorrow morning.
Premo Mondone
Occasional Advisor

Re: Mirror Rebuild Proliant ML 350 Gen3

Well.....I went in today to give it another try. Same result. I changed the controller to "High Priority", rebooted and didn't even get the "F2" option. All I get is "F1" and some other options that have have nothing to do with rebuilding the mirror.

Prior to that I get the 1787 Error for Slot2 that tells me that I need to replace the SCSI ID-0 drive and that the SCSI ID-1 is operating in Interim Recovery Mode.

I ran the ACM and ADM and they both show the same thing as before. I would have copied the report, but the browser is temporarily "disabled" and therefore you can't cut-n-paste the report.

I guess I can try downloading the FW updates, but I'm not real hopeful. Also, I doubt seriously that it's a backplane problem. I'm not the sharpest knife in the drawer, but I find it hard to believe that the backplane is working fine for the remaining drive but that it fails on all the hot-swap drive bays. That just doesn't make sense, at least to me.

Everything tells me to replace both the old drive and the new drive. The ADU and ACU both see the old drive and new drive. If the backplane were bad, then it wouldn't even show up, right?

Is there not a way to MANUALLY issue a rebuild mirror command to the controller once the OS is loaded? The system "sees" the new drive, just like it saw the old drive. It knows it's there.
Steven Clementi
Honored Contributor

Re: Mirror Rebuild Proliant ML 350 Gen3

You can try adding the new drive as a hot spare. Not sure if that is the best course, but it might work out.


Steven
Steven Clementi
HP Master ASE, Storage, Servers, and Clustering
MCSE (NT 4.0, W2K, W2K3)
VCP (ESX2, Vi3, vSphere4, vSphere5, vSphere 6.x)
RHCE
NPP3 (Nutanix Platform Professional)
RaMpaNTe
Trusted Contributor

Re: Mirror Rebuild Proliant ML 350 Gen3

Well, based on the info and based on my expertise troubleshooting with Proliant servers I can tell you what is going on here.

When the array was built, the controller should have seen ID 0 and ID 1, that's what it supostedly had to see; but somehow the controller assigned another ID # to the drive in the bay 1, this would not affect anything as long as no drive fails.

Since the drive in the bay 1 was assigned with another ID# different than that, now that the drive is failed the controller is waiting for that drive to be replaced, and no matter how many newe drives you put in that bay, the controller wouldn't star the rebuild because it still waiting for another ID.

Solution for this: backup everything delete and create another array and then restore from back up.

RaMpaNTe
You heve a question... I have an aswer!!!
Premo Mondone
Occasional Advisor

Re: Mirror Rebuild Proliant ML 350 Gen3

Well, the OS and the mirror was factory installed so I'm not sure what was done. I afraid that I'd be doing this, but was hoping that I could avoid it as it's such a pain and this company runs some crazy shifts with their data entry people, so downtime to do this will be a pain as well.

Thanks for all the help and suggestions. I really do appreciate the ideas tossed about to try and solve the problem the "easy" way.