https://community.hpe.com/t5/storevirtual-storage/broke-it-help/m-p/4696069#M1623 OK bit of a drastic subject line and fortunately it's an eval unit, but today we managed to do something we didn't expect.<BR /><BR />Two P4500 12 disk nodes in a single cluster connected to the same switch.<BR /><BR />We'd been pulling various cables/drives to simulate failure modes and with the FOM installed and running we were blown away at the whole "5 pings lost and it's back" thing when you pull a node completely.<BR /><BR />However, then we pulled "Node 2" (we just pulled both NICs for speed) and the group IP was no longer pingable.<BR /><BR />Upon investigation it would seem that what happened was that on "Node 1", the manager had somehow got stopped, so with us losing Node 2 deliberately all we now had left working was Node 1 with no manager, and a FOM.<BR /><BR />We just could not connect to Node 1 as (I think) you connect to a management group by the group IP, which was dead, so I was in a chicken and egg where (I think) I knew I needed to start the manager on Node 1, but I couldn't log onto the group IP to do so because two managers were down so no quorum.<BR /><BR />The solution was to reconnect the NICs to the second node so the quorum came back, and to then start the manager on the first node - but in a full on disaster we may not have that luxury.<BR /><BR />What have I missed? Wed, 06 Oct 2010 16:24:01 GMT Paul Hutchings 2010-10-06T16:24:01Z https://community.hpe.com/t5/storevirtual-storage/broke-it-help/m-p/4696069#M1623 OK bit of a drastic subject line and fortunately it's an eval unit, but today we managed to do something we didn't expect.<BR /><BR />Two P4500 12 disk nodes in a single cluster connected to the same switch.<BR /><BR />We'd been pulling various cables/drives to simulate failure modes and with the FOM installed and running we were blown away at the whole "5 pings lost and it's back" thing when you pull a node completely.<BR /><BR />However, then we pulled "Node 2" (we just pulled both NICs for speed) and the group IP was no longer pingable.<BR /><BR />Upon investigation it would seem that what happened was that on "Node 1", the manager had somehow got stopped, so with us losing Node 2 deliberately all we now had left working was Node 1 with no manager, and a FOM.<BR /><BR />We just could not connect to Node 1 as (I think) you connect to a management group by the group IP, which was dead, so I was in a chicken and egg where (I think) I knew I needed to start the manager on Node 1, but I couldn't log onto the group IP to do so because two managers were down so no quorum.<BR /><BR />The solution was to reconnect the NICs to the second node so the quorum came back, and to then start the manager on the first node - but in a full on disaster we may not have that luxury.<BR /><BR />What have I missed? Wed, 06 Oct 2010 16:24:01 GMT https://community.hpe.com/t5/storevirtual-storage/broke-it-help/m-p/4696069#M1623 Paul Hutchings 2010-10-06T16:24:01Z