Problem with HP Cluster

[Stephen McCann]

Hi,

Our Hp Cluster failed this week for no apparent reason. We’re running Windows 2003 Enterprise with Microsoft clustering on two HP DL740’s using HP Secure Path back to a MSA1000.

According to the cluster log the quorum drive went off line and was inaccessible for approximately 9 minutes. Then several other drives on the SAN went offline as well.

Whatever happened caused the SCSI buses not to close down properly and our SQL database was corrupted.

The Quorum drive consists of 4 36.4gb 10000rpm hard drives. These 4 drives are in a RAID 1 setup and then striped. All disks are spilt over different controllers.

There is no way that this should have failed unless the entire SAN failed (unlikely). As the other drives went off line over a period of 9 minutes this doesn’t make sense either.

As all the drives, SAN and controllers appear to be ok and the cluster is now performing correctly I have no idea of how to diagnose what happened.

Any advice or comments would be apprecia

[Thomas Bianco]

the best way to fix something like that is to eliminate the impossibilities and fix whatever remains. scientific method. find ways to disprove possibilities based on the evidence.

if all the LUNS when of at once, then it indicates the problem can't be based in a single LUN. we're left with a problem with the server, fibre channel NIC(s), RAID manager, switches, or fabric.

multiple servers had the same problem? that eliminates a software or server based problem.

redundant fabrics? then it can't be a switch problem, or a lost connection.

redundant array managers? then it can't be the control board in the disk array, but it still could be the backplane.

check the SAN switches for module disconnects or power events. possibility of a "cleaning crew unplugged it" problem?

a word of caution, don't eliminate anything unless the evidence shows it isn't possible. don't eliminate that new shiny GBIC because it was just installed last week.

[Richard Perez]

Stephen
why do you have such behemoth quorum disk? Where is your data? Please post information about your disk and cluster groups.

Have you reviewed the System and Application log? Read them and find references to Securepath events at the time of the problems.

How do you know "the SCSI buses didn't close down properly" ?

Regards