topic Re: Replacing a Quorum Disk in Operating System - OpenVMS

Replacing a Quorum Disk

Jack Trachtman — Fri, 24 Jun 2005 19:07:03 GMT

I have a two node cluster with the
quorum disk defined as a virtual disk
on an EVA5000. (VMS V7.3-2)

I want to delete and recreate that disk,
but I'm not clear on the steps. (The HP
docs I've seen so far are not clear.) My
question is similar to "how would I replace
a failed Quorum disk", which has got to
be a somewhat common situation so I'm surprised that I haven't found explicit
docs on this.

Is one approach:
1) Shut down cluster
2) Delete and recreate disk on EVA
with same Unit ID
3) Boot cluster
4) Init disk from VMS side
5) Reboot cluster so cluster file
will get created on new disk

Will the above work?

Is there a simpler way?

TIA

Re: Replacing a Quorum Disk

Garry Fruth — Fri, 24 Jun 2005 19:51:40 GMT

The above will work. I believe the following will work:
1) Dismount/cluster qdisk; assuming it is not a pageswap/dosd/system disk or has other open files.
2) Delete and recreate on EVA
3) Reinitialize disk
4) Reboot one node

Re: Replacing a Quorum Disk

Volker Halle — Sat, 25 Jun 2005 02:54:55 GMT

Jack,

dismounting the quorum disk in a running cluster works (tested on V7.3-1), so you could start with DISM/CLUSTER qdsk. Access to the quorum disk will be temporarily lost but will be re-established immediately.

Assuming that your votes are set up in a way, to allow the cluster to maintain quorum, if QDSKVOTES are not present, you could then delete and re-create the quorum disk on the EVA. This will cause access to the quorum disk to be lost, but the cluster should continue if 2 nodes are up (assuming 2x VOTES=1 and QDSKVOTES=1, i.e. QUORUM = 2).

You may need to do SYSMAN> IO AUTO or IO SCSI_PATH_VERIFY after re-creating the disk unit.

Then INIT and MOUNT/SYSTEM the new quorum disk, this will allow the QUORUM.DAT file to be created by CLUSTER_SERVER and connection to the 'quorum disk' will be re-established.

Volker.

Re: Replacing a Quorum Disk

Jan van den Ende — Sat, 25 Jun 2005 06:40:33 GMT

Jack,

just the fact that the QDSK _IS_ a disk with a virtual hardware name makes this much easier than the case with a physical disk.

You just have to make sure of two things:
- you have to re-create the exact same-named unit
- during the period of removal though re-creation of the unit you have no "headroom" in quorunm voters, so you have to make as sure as you can that you do not loose any voters

If you think you need to change the deviceNAME of the quorum disk, yhen a cluster shutdown is the only simple way.
(I think it should be possible to do it in a rolling way as well, but that requires thorough planning, and several reboots and voting manipulations. Not for the faint of heart, nor for the unexperienced. I even doubt whether any such route will be supported)

Just first remove the old unit, and create the a new with the same name will be your best route.

Success.

Proost.

Have one on me.

jpe

Re: Replacing a Quorum Disk

comarow — Sun, 26 Jun 2005 10:25:22 GMT

simply replace the disk.
Conversationally boot with more then enough votes. I will build quorum.dat. Then it will use the qdisk vote.

Re: Replacing a Quorum Disk

Wim Van den Wyngaert — Tue, 28 Jun 2005 10:46:45 GMT

It is a pitty that analog to
set cluster/expected_votes=xx
they didn't implement
set cluster/disk_quorum=yyy

Wim

Re: Replacing a Quorum Disk

Eberhard Wacker — Tue, 28 Jun 2005 16:00:30 GMT

Hi Jack,
your approach would be okay but a simplier point 3: boot 1 node minimum or boot the VMS cd for the VMS init.
Any further minimum boot makes no sense because doing this a quorum.dat will not be created.

In case of making the quorum disk unavailable to the quorum disk watcher nodes a cluster state transition will occur.
So far this is no problem but Iâ m really interested if the way â let all nodes running all over the timeâ of Volker and Jan is a totally smooth one related to the activities of the connection manager. Anyone who've made it already in this way?

Che

Re: Replacing a Quorum Disk

Volker Halle — Wed, 29 Jun 2005 04:05:27 GMT

Eberhard,

would you accept a test on a V7.3-1 single cluster node with a local SCSI quorum disk as a proof-of-concept, that you can swap the quorum disk in a running cluster - if you can provide enough votes to keep the cluster running or are willing to use the IPC interrupt (or AMDS) to recalculate quorum ?

The attached file shows a simple test on how this can be done - and it does work !

The different steps are labeled [1] to [8]:

[1] boot a single cluster node with VOTES=1, EXPECTED_VOTES=1, QDSKVOTES=1 and DISK_QUORUM=DKC500 - no quorum file does yet exist.

[2] mount the designated quorum disk (DKC500), this will cause QUORUM.DAT to be created automatically by CLUSTER_SERVER - even if you only mount that disk privately.

[3] dismount the quorum disk.

[4] unplug the quorum disk

[5] As dynamic QUORUM is 2, step [4] will cause quorum to be lost (in this simple config), but it can be easily regained using the IPC>Q interrupt.

[6] plug in the physical quorum disk into DKC400 slot and delete QUORUM.DAT (if I would have had an empty new disk, I could have used it and just do an INIT)

[7] plug the disk back into DKC500 (note: there is NO quorum.dat file anymore on that disk !)

[8] mount the 'new' quorum disk again. CLUSTER_SERVER will create QUORUM.DAT and the quorum disk will become active again.

NO REBOOTS needed at all. And even if your cluster would loose QUORUM, if the quorum disk dies, you could use IPC/DECamds and recover without any reboot.

Volker.

Re: Replacing a Quorum Disk

Jack Trachtman — Wed, 29 Jun 2005 10:27:02 GMT

Volker,

Thanks for doing that test! (I'm still surprised that this process isn't documented!)

BTW: I use AMDS but am not familiar with IPC. What is it?

Thanks

Re: Replacing a Quorum Disk

Jan van den Ende — Wed, 29 Jun 2005 11:38:26 GMT

Jack,

IPC is the Interrupt Control Program.

You enter it at the console (used to be ^P ; nowadays whatever the specific hardware requires)
then

>>> D SIRR C

deposits hex C in the SIRR register, meaning set IPL 12

IPC> Q

at IPC force Quorum recalculation

IPC> C

Continue normal operation.

--- in a cluster, this HAS to be COMPLETED within RECNXINTERVAL

It has always been around, AFAIK, although the Vax syntax was slightly different.

and, this is what AMDS can do for you, and quick, when you ask it to force quorum.

hth,

Proost.

Have on eon me.

jpe

Re: Replacing a Quorum Disk

Volker Halle — Thu, 30 Jun 2005 00:18:40 GMT

Jan,

to exit the IPC> interrupt, you need to enter (not C, which stands for: Cancel mount-verification)

The IPC (short for IPL C interrupt, where 0xC is IPL 12.) is described in the System Managers Manual Volume 1: Essentials

Chapter: Using Interrupt Priority Level C (IPC)

Volker.

Re: Replacing a Quorum Disk

Eberhard Wacker — Thu, 30 Jun 2005 14:36:51 GMT

Hi Volker,

I wouldn't had a doubt in such a configuration but many thanx for your poc. The reason I've asked is that I cannot test at the moment the whole scenario with the recreation of a virtual quorum disk. So I can only "believe" that the same will work.

One other point raised looking at your logfile: it can be handled in a smooth way only as long as mvtimeout has not been reached, right ?

Cheers,
Eberhard

Re: Replacing a Quorum Disk

Eberhard Wacker — Thu, 30 Jun 2005 14:56:55 GMT

Hi Wim,
what I miss is the possibility to use a quorum file on a HBVShadowed disk :-)
Cheers
Eberhard

Re: Replacing a Quorum Disk

Uwe Zessin — Thu, 30 Jun 2005 15:02:52 GMT

| >>> D SIRR C
|
| deposits hex C in the SIRR register, meaning set IPL 12

Hm, SIRR is the Software Interrupt Request Register - my understanding is that it requests an IPL 12 interrupt, but does not set the IPL to 12. Imagine what happens if the processor is currently running at a higher IPL - not a good idea!

Extra points if you know what fork processes are for ;-)

The next step would be:
>>> continue

| IPC> Q
|
| at IPC force Quorum recalculation
|
| IPC> C
|
| Continue normal operation.

Volker has already commented that "C" is used to cancel a mount verification.

| --- in a cluster, this HAS to be COMPLETED within RECNXINTERVAL

Right, and there must not be a bug in the VAX-8600 ;-)

| It has always been around, AFAIK, although
| the Vax syntax was slightly different.

>>> D/I 14 C

Re: Replacing a Quorum Disk

Jan van den Ende — Thu, 30 Jun 2005 16:30:05 GMT

Re Eberhard:

what I miss is the possibility to use a quorum file on a HBVShadowed disk :-)

Why is that not supported (and never will be !!):

Suppose a cluster with two equal halves (for ease of concept, take a two-site cluster, but the principle is general).

So, each halve has n votes, and there is a quorum vote. Expected_votes 2n + 1, quorum n + 1.

If the halves lose sight of each other, one halve has the qdsk, has quorum, and can continue; the other halve looses quorum.
The cluster integrity is guarded.
Now, suppose the qdsk is shadowed.
Again, the halves loose contact.
That could well mean that the shadow members loose contact.
Now EACH halve sees its member of the qdsk shadow set as THE qdsk.
Both halves maintain quorum, and within a few handfulls of IOs (say, miliseconds?) your data is seriously corrupted..
HPUX has a good descriptive name for that situation:
A "SPLIT BRAIN CLUSTER".
REALLY unwanted.
So: _NO_ shadowed quorum disk. Period.

hth

Proost.

Have one on me.

jpe

Re: Replacing a Quorum Disk

Eberhard Wacker — Thu, 30 Jun 2005 19:16:13 GMT

Re Jan:

sure, you're absolutely right, it will never be supported but nevertheless I would wish I COULD use a (one site located) shadowset as a quorum disk volume.

Also with the current (and long existing) VMS implementation I can set up the whole cluster in a non supported way so that cluster partitioning can occur (at least at boot time)!
So I still say: I would like to HAVE the possibility to use a shadowset as a quorum volume. If I set up the configuration in a way that can lead to trouble then this is my configuration failure in each case.

Cheers (and a proost)
Eberhard

Re: Replacing a Quorum Disk

Eberhard Wacker — Thu, 30 Jun 2005 19:44:47 GMT

Re Jan II:
- sysgen parameter quorum_disk remains as it is referring to a special disk
- this disk is a member of a shadowset
This would mean: no change to the current behaviour (but probably a lot of re-writing VMS code).
Simple(?) question: am I right or totally wrong ?
Proost
Eberhard

Re: Replacing a Quorum Disk

Volker Halle — Fri, 01 Jul 2005 01:01:00 GMT

re: last few

Setting SIRR to C requests an IPL 12. interrupt, which then will issue the IPC> prompt (running at IPL 12.)

Using the IPC mechanism in a SMP system, may lead to CPUSPINWAIT or CPUSANITY crashes, so AMDS is definitely the better choice ;-)

If the quorum disk has gone dead and MVTIMEOUT has expired, you can still DISMOUNT/ABORT it, IF no (other) open files are on that disk.

Shadowsets can split as well and if either side of the cluster then continues with it's local member ?! Just use a small quorum node, put it in a safe place and forget about it.

Volker.

Re: Replacing a Quorum Disk

comarow — Sun, 03 Jul 2005 15:21:07 GMT

Since you have availability manager, you do not need the IPC utility. Just reset quorum.

We have white papers on replacing a quroum disk. I'll email it to you.

Re: Replacing a Quorum Disk

Ian Miller. — Mon, 04 Jul 2005 04:22:37 GMT

Bob, if there is documentation on this then can it be on a public place (on itrc parhaps) and can you post a reply here with the link so people can find it in future.
Thanks,
Ian.