All Logical Volumes failed on msa1500cs for all disks in MSA30 enc

[Daniel Stith]

Background:
New install not in production yet

Schema:
One BLp30 enclosure with 9 blades (six 1cpu 3 dualP) with one blade OS build completed with BFS functionality

One MSA30 Enclosure with 14 disks (3 72G U320 11 243G U320 HP drives) and no error lights, etc. Everything looks good, but...

One MSA1500cs controller with dual ms1000's in a redundant configuration (currently one controller is 'pulled out' in order to ensure single path to fabric for 'initial server')

One used Ultra3 Compaq Enclosure with 0 disks currently (retread hardware going to be used down the road for disk-to-disk replication via Veritas)

Two HP 2/8 San Switches with only Switch A physically connected.

All the stuff is cabled right, and was working just fine...then everything was powered down to properly cable manage (no new cabling was introduced and all physical connections have been double checked by two people), and now all the Logical Volumes show as failed in the MSA1500cs. I can see the disks just fine...diagnostic runs clean. The HBA in the Blade Enc is seen by the BIOS of the server and the proper LUN selection is still present. The running FASTUtil for the QLogic I am able to see the MSA1500cs just fine from the Blade Enclosure.

It seems based upon the Volume Failed message I receive when I run the Show Units command on the MSA that the problem lies between it and the Array, but I can see the disks themselves...flash lights...etc. just fine. I've tried the 'last ditch' power everything off again and carefully bring the stuff up in order, but didn't make any difference. Should I just try to run Set Volume Rebuild and cross my fingers?

Any help would be greatly appreciated.
Dan

[Daniel Stith]

Okay, I figured out the best way to do this. Without the capability to do Volume Rebuild with the controller, and no third party software (i.e. Veritas Volume Manager) the only choice I had left was to power everything down in the correct order and then powered it back up in the correct order and hope the controller would automatically try to recreate the volumes in the RAID:

Power Down Sequence:
Blade enclosure
MSA1500cs
MSA30 drive enclosure

Power Up Sequence:
MSA30 drive enclosure (wait for stable lights)
MSA1500cs (wait for system name or system ready message in LCD, and then allow volume rebuild to complete)
Blade Enclosure

This worked like a charm and all volumes are back and healthy.