topic Re: FC10 / FC60 Broken mirror rebuild help in Disk Enclosures

FC10 / FC60 Broken mirror rebuild help

SteveKirby — Thu, 28 Mar 2002 18:09:45 GMT

I recently installed a FC60 with 3 FC10's. I tested my mirror building by dropping the power to one bank of drives. The test succeeded. The system remained active and I can still access the drives.
I powered on the 'downed' array, but the disks have not rebuilt in 12 hours.
In SAM the status of the FC60 controller is:
LUN 0 - DEGRADED:WAITING REPAIR
DISK 1:0 ENCLOSURE 0 SLOT 1 - NO RESPO
DISK 1:1 ENCLOSURE 0 SLOT 3 - NO RESPO

What do I need to do to have these rebuild.
a) Right now
b) automatically in the future.

Re: FC10 / FC60 Broken mirror rebuild help

Insu Kim — Fri, 29 Mar 2002 01:53:32 GMT

This rarely occurs under the normal circumstances but if it happens and the disk array is in production, I suggest that you log a call to HP to fix it.
Never try to do it on your own because you might blow out your date.

I'm writing on an assumption that the disk array is not in production.

Running "amdsp -a tells you that disks on both 1:0 and 1:1 are either in a write failed state or in a no_response state.

This is because the disk array controller doesn't notice that the disk modules are back online so we have to instruct it to notice the change.
Generally speaking, you don't need to issue this hidden command because all that you have to do is to replace a failed drive with a good one.
The command itself gives the effect of replacing the failed drive.

# amutil -d 1:0
This makes it possible to transit the failed drive to optimal.
You know that the failed drive is not really failed.
See if any changes are made to the disk array by running "amdsp -a" again.
If the state of the failed drive returned to "replaced", evenually, the disk array will perform a rebuild.
If nothing happened, try re-seating the disk, wait for 60 seconds and check the failed drive state by executing "amdsp" then issue amutil again.

If it works, do this for the failed driver on another LUN.
# amutil -d 1:1

Hope this helps,

Re: FC10 / FC60 Broken mirror rebuild help

SteveKirby — Mon, 01 Apr 2002 20:57:55 GMT

I noticed another thread that talked about the drives not spinning up so I pulled and re-inserted both drives and they spun up and started rebuilding within a minute.

I am making a note of your commands for the future though.

Thanks.

Re: FC10 / FC60 Broken mirror rebuild help

Insu Kim — Tue, 02 Apr 2002 10:41:49 GMT

Forgot to add "array ID" to the command.
# amutil -d 1:0

Once again, should be careful when you do this.
In the case of a single drive failure, or possibly multiple drive failures in RAID1, where the LUN is degraded but not dead, it would be safe to unfail drives (amutil -d) without loss of data integrity.
That's why we used that unfail command mentioned.

On the other hand,
when one or more logical units have been marked as dead due to a second drive failure, a special process to revive the LUN should be used. Using the unfail command on more than one failed disk in a DEAD LUN can cause data loss.

For example,
2 disks in LUN 1 failed.

now, you have to determine which disk originally failed first and fail this disk again with the "amutil -D" fail command and leave the other disk that was failed as optimal using "amutil -d" unfail command.
Reseating or replacing the failed drive could be the way to transition it to a replaced state at this time, as a result, a rebuild will occur.

There should be a different way for each case so it would be safe for HP to take care of the dreadful situation where you can cause data loss.
This is why you make a support agreement with HP.

Hope this helps,

Re: FC10 / FC60 Broken mirror rebuild help

SteveKirby — Sun, 22 Jan 2006 22:27:53 GMT

The reset of the drives by re-inserting them was the trick...