HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
Disk Enclosures
Showing results for 
Search instead for 
Did you mean: 

FC10 / FC60 Broken mirror rebuild help

Go to solution
Frequent Advisor

FC10 / FC60 Broken mirror rebuild help

I recently installed a FC60 with 3 FC10's. I tested my mirror building by dropping the power to one bank of drives. The test succeeded. The system remained active and I can still access the drives.
I powered on the 'downed' array, but the disks have not rebuilt in 12 hours.
In SAM the status of the FC60 controller is:

What do I need to do to have these rebuild.
a) Right now
b) automatically in the future.

Insu Kim
Honored Contributor

Re: FC10 / FC60 Broken mirror rebuild help

This rarely occurs under the normal circumstances but if it happens and the disk array is in production, I suggest that you log a call to HP to fix it.
Never try to do it on your own because you might blow out your date.

I'm writing on an assumption that the disk array is not in production.

Running "amdsp -a tells you that disks on both 1:0 and 1:1 are either in a write failed state or in a no_response state.

This is because the disk array controller doesn't notice that the disk modules are back online so we have to instruct it to notice the change.
Generally speaking, you don't need to issue this hidden command because all that you have to do is to replace a failed drive with a good one.
The command itself gives the effect of replacing the failed drive.

# amutil -d 1:0
This makes it possible to transit the failed drive to optimal.
You know that the failed drive is not really failed.
See if any changes are made to the disk array by running "amdsp -a" again.
If the state of the failed drive returned to "replaced", evenually, the disk array will perform a rebuild.
If nothing happened, try re-seating the disk, wait for 60 seconds and check the failed drive state by executing "amdsp" then issue amutil again.

If it works, do this for the failed driver on another LUN.
# amutil -d 1:1

Hope this helps,
Never say "no" first.
Frequent Advisor

Re: FC10 / FC60 Broken mirror rebuild help

I noticed another thread that talked about the drives not spinning up so I pulled and re-inserted both drives and they spun up and started rebuilding within a minute.

I am making a note of your commands for the future though.

Insu Kim
Honored Contributor

Re: FC10 / FC60 Broken mirror rebuild help

Forgot to add "array ID" to the command.
# amutil -d 1:0

Once again, should be careful when you do this.
In the case of a single drive failure, or possibly multiple drive failures in RAID1, where the LUN is degraded but not dead, it would be safe to unfail drives (amutil -d) without loss of data integrity.
That's why we used that unfail command mentioned.

On the other hand,
when one or more logical units have been marked as dead due to a second drive failure, a special process to revive the LUN should be used. Using the unfail command on more than one failed disk in a DEAD LUN can cause data loss.

For example,
2 disks in LUN 1 failed.

now, you have to determine which disk originally failed first and fail this disk again with the "amutil -D" fail command and leave the other disk that was failed as optimal using "amutil -d" unfail command.
Reseating or replacing the failed drive could be the way to transition it to a replaced state at this time, as a result, a rebuild will occur.

There should be a different way for each case so it would be safe for HP to take care of the dreadful situation where you can cause data loss.
This is why you make a support agreement with HP.

Hope this helps,
Never say "no" first.
Frequent Advisor

Re: FC10 / FC60 Broken mirror rebuild help

The reset of the drives by re-inserting them was the trick...