Replacing a Quorum Disk

[Jack Trachtman]

I have a two node cluster with the
quorum disk defined as a virtual disk
on an EVA5000. (VMS V7.3-2)

I want to delete and recreate that disk,
but I'm not clear on the steps. (The HP
docs I've seen so far are not clear.) My
question is similar to "how would I replace
a failed Quorum disk", which has got to
be a somewhat common situation so I'm surprised that I haven't found explicit
docs on this.

Is one approach:
1) Shut down cluster
2) Delete and recreate disk on EVA
with same Unit ID
3) Boot cluster
4) Init disk from VMS side
5) Reboot cluster so cluster file
will get created on new disk

Will the above work?

Is there a simpler way?

TIA

[Garry Fruth]

The above will work. I believe the following will work:
1) Dismount/cluster qdisk; assuming it is not a pageswap/dosd/system disk or has other open files.
2) Delete and recreate on EVA
3) Reinitialize disk
4) Reboot one node

[Volker Halle]

Jack,

dismounting the quorum disk in a running cluster works (tested on V7.3-1), so you could start with DISM/CLUSTER qdsk. Access to the quorum disk will be temporarily lost but will be re-established immediately.

Assuming that your votes are set up in a way, to allow the cluster to maintain quorum, if QDSKVOTES are not present, you could then delete and re-create the quorum disk on the EVA. This will cause access to the quorum disk to be lost, but the cluster should continue if 2 nodes are up (assuming 2x VOTES=1 and QDSKVOTES=1, i.e. QUORUM = 2).

You may need to do SYSMAN> IO AUTO or IO SCSI_PATH_VERIFY after re-creating the disk unit.

Then INIT and MOUNT/SYSTEM the new quorum disk, this will allow the QUORUM.DAT file to be created by CLUSTER_SERVER and connection to the 'quorum disk' will be re-established.

Volker.

[Jan van den Ende]

Jack,

just the fact that the QDSK _IS_ a disk with a virtual hardware name makes this much easier than the case with a physical disk.

You just have to make sure of two things:
- you have to re-create the exact same-named unit
- during the period of removal though re-creation of the unit you have no "headroom" in quorunm voters, so you have to make as sure as you can that you do not loose any voters

If you think you need to change the deviceNAME of the quorum disk, yhen a cluster shutdown is the only simple way.
(I think it should be possible to do it in a rolling way as well, but that requires thorough planning, and several reboots and voting manipulations. Not for the faint of heart, nor for the unexperienced. I even doubt whether any such route will be supported)

Just first remove the old unit, and create the a new with the same name will be your best route.

Success.

Proost.

Have one on me.

jpe

[comarow]

simply replace the disk.
Conversationally boot with more then enough votes. I will build quorum.dat. Then it will use the qdisk vote.

[Wim Van den Wyngaert]

It is a pitty that analog to
set cluster/expected_votes=xx
they didn't implement
set cluster/disk_quorum=yyy

Wim

[Eberhard Wacker]

Hi Jack,
your approach would be okay but a simplier point 3: boot 1 node minimum or boot the VMS cd for the VMS init.
Any further minimum boot makes no sense because doing this a quorum.dat will not be created.

In case of making the quorum disk unavailable to the quorum disk watcher nodes a cluster state transition will occur.
So far this is no problem but Iâ m really interested if the way â let all nodes running all over the timeâ of Volker and Jan is a totally smooth one related to the activities of the connection manager. Anyone who've made it already in this way?

Che

[Volker Halle · Accepted solution]

Eberhard,

would you accept a test on a V7.3-1 single cluster node with a local SCSI quorum disk as a proof-of-concept, that you can swap the quorum disk in a running cluster - if you can provide enough votes to keep the cluster running or are willing to use the IPC interrupt (or AMDS) to recalculate quorum ?

The attached file shows a simple test on how this can be done - and it does work !

The different steps are labeled [1] to [8]:

[1] boot a single cluster node with VOTES=1, EXPECTED_VOTES=1, QDSKVOTES=1 and DISK_QUORUM=DKC500 - no quorum file does yet exist.

[2] mount the designated quorum disk (DKC500), this will cause QUORUM.DAT to be created automatically by CLUSTER_SERVER - even if you only mount that disk privately.

[3] dismount the quorum disk.

[4] unplug the quorum disk

[5] As dynamic QUORUM is 2, step [4] will cause quorum to be lost (in this simple config), but it can be easily regained using the IPC>Q interrupt.

[6] plug in the physical quorum disk into DKC400 slot and delete QUORUM.DAT (if I would have had an empty new disk, I could have used it and just do an INIT)

[7] plug the disk back into DKC500 (note: there is NO quorum.dat file anymore on that disk !)

[8] mount the 'new' quorum disk again. CLUSTER_SERVER will create QUORUM.DAT and the quorum disk will become active again.

NO REBOOTS needed at all. And even if your cluster would loose QUORUM, if the quorum disk dies, you could use IPC/DECamds and recover without any reboot.

Volker.

[Jack Trachtman]

Volker,

Thanks for doing that test! (I'm still surprised that this process isn't documented!)

BTW: I use AMDS but am not familiar with IPC. What is it?

Thanks

[Jan van den Ende]

Jack,

IPC is the Interrupt Control Program.

You enter it at the console (used to be ^P ; nowadays whatever the specific hardware requires)
then

>>> D SIRR C

deposits hex C in the SIRR register, meaning set IPL 12

IPC> Q

at IPC force Quorum recalculation

IPC> C

Continue normal operation.

--- in a cluster, this HAS to be COMPLETED within RECNXINTERVAL

It has always been around, AFAIK, although the Vax syntax was slightly different.

and, this is what AMDS can do for you, and quick, when you ask it to force quorum.


hth,

Proost.

Have on eon me.

jpe