Booting two nodes with same root system disk....

[JBR]

Hi,

What are the consequences of booting two nodes with same root system disk (os_flags 0,0)?

If I´ve one node up & running (os_flags 0,0), what happens when I boot de second one with same root disk (os_flabs 0,0)?

"Configuration: Two ES45, one up & runnig, second one standby (halt), cluster license loaded, vaxcluster=2, votes=1 , expected_votes=1, no quorum disk.."

Thanks in advanced.

[labadie_1]

As both nodes will have the same couple SCSNODE/SCSSYSTEMID, the second will not enter the Cluster and crash with Cluexit I guess.

So you will have only one node up.

[Heinz W Genhart]

Hi Macero

labadie is right. I made this mistake short time ago. The second node will not enter the Cluster and crash with Cluexit.

Regards

Geni

[Shankar Bose]

Hi Macero,

If you are after booting the two for a single system disk. Then you must have more than one system root directory. And set root no to 1. That will boot the system from SYS1.

Best Regards
Shankar

[Dean McGorrill]

Hi Macero,
I to have done that by mistake and
cluexited. you can find the roots on the
disk with a "$ dir sys$sysdevice:[0,0]sys*.dir" I'm curious as to why you would
want to do this.. or are YOU just curious? ;)

[Hoff · Accepted solution]

I repeatedly cite the need to set votes and expected_votes correctly.

Settings here are incorrect, and are potentially very unsafe.

In this case, a cluster hang is the best case. If Votes and Expected_Votes settings are incorrectly (or -- as is the case here -- set "creatively"), this configuration can trigger serious disk-level data corruptions for various configurations.

Start here: http://64.223.189.234/node/153

I've personally seen a configuration with incorrect Votes and Expected_Votes and with shared storage stomp on its disks. The corruptions that resulted were massive, and there is no change to log in. The only real available recovery was to roll the data back in from the last good BACKUP.

If this were me, I'd fix the saved values for Expected_Votes and set this box up with two nodes, and a shared quorum disk. Or with a third voting node. This configuration is cheap insurance, and it lets you use the AlphaServer ES45 nodes in parallel. Or to have one down and halted, to be able to boot it without having to deal with Votes and Expected_Votes. (It is best to make a hot-, warm- or cold-standby box as simple and fast to boot as is feasible.)

Stephen Hoffman
HoffmanLabs LLC

[JBR]

Well, actually, I have one node ES45 booting from EVA disk (no cluster license loaded, vaxcluster=0) and the second one is prepared to boot (P00>>) from same system disk, in case of the first one is broken.

I need to prevent accidentally dual boot from the same disk, beacuse this situation was produced in the past and the queue manager files and another critical files were corrupted.

My question is: If I configure one cluster with one node, without quorum disk, votes=1, expected=1, only one root sys0, and the boot_osflags for two nodes 0,0..... one one UP&Running, second one halted... is the best way to prevent corruption mentioned??????

Thanks.

[Andy Bustamante]

If you have a cluster license, consider having both nodes running. If you have both nodes running and an application that isn't cluster aware, can you start the application on the secondary node faster than rebooting? How do users access the system? If you're using TCPIP, use a FAILsafe address that migrates between nodes for your users. DECnet and LAT support a cluster address. If you're manually restoring service, you're giving up some strong features that VMS provides.

If you're stuck with a manual fail over and changing roots, having the second node running provides a "known good hardware" state. You can boot node A from root 0 and node B from root 1. When you want to fail over, make sure both nodes are down and change os_flags on BOTH nodes to avoid problems.

There many options available for commericial help if you'd like help configuring a solution. Many providers can be found here.


Andy

[Hoff]

>>>My question is: If I configure one cluster with one node, without quorum disk, votes=1, expected=1, only one root sys0, and the boot_osflags for two nodes 0,0..... one one UP&Running, second one halted... is the best way to prevent corruption mentioned??????<<<

What you are doing here is an open invitation to corrupting all shard storage, should a node be booted "incautiously."

Do not assume that a rogue node will be detected; it may not, simply because of an address collision on the network.

With the incorrect and potentially dangerous setting of EXPECTED_VOTES, the node will successfully bootstrap.

If you want blade guards, set the values to prevent the node from booting by setting EXPECTED_VOTES correctly, and establish a manual reset for the value when you want to boot the stand-by node. (I view this as a less desirable approach. The best solution is to cluster the nodes, and to ensure that VOTES and EXPECTED_VOTES are set correctly.)

There are two reasons to set EXPECTED_VOTES wrong. One is when you are deliberately and knowingly booting into a degraded condition and you have ensured there can be no corruption -- such as initially forming a cluster and setting up a quorum disk or such. The other reason to set EXPECTED_VOTES too low is because you want a disk to get corrupted.

The storage corruptions from a mis-set EXPECTED_VOTES and a partitioned cluster are impressive. OpenVMS will go out of its way to try to detect and avoid these cases, but there are cases that cannot be detected and corrected.

Stephen Hoffman
HoffmanLabs.com

[Dean McGorrill]

Macero,
yeeoowza, I would not be able to sleep
at night know there was a >>> prompt sitting
on a console ready to completely corrupt
my system! I see what you want to do. I'd
remove all the connections to the backup
cpu to remove the accident before it happens. if the first one died, I'd manually move them over from the first cpu. short of that, as asked is (are) the application(s) cluster smart? if so you could cluster and use an alias. Dean

[Robert Gezelter]

macero,

I must completely concur with Hoff. This is an engraved invitation to a nightmare.

The cost of the cluster license is FAR, FAR less than what it can cost to recover if both machines happen to be booted at the same time.

- Bob Gezelter, http://www.rlgsc.com

[Martin Hughes]

The first thing I would do is set your auto_action to HALT on your standby node. And perhaps go one step further and set your boot_flags to something spurious, such as z,0. You can change these values back when you need to boot the standby node in a controlled situation.

Then I would look at the configurations suggested earlier involving a quorum disk or 3rd node. If your application is important enough to warrant the purchase of a spare ES45 then I would assume that it is worthy of a proper cluster configuration.

Just curious, when you have a "cold" standby node, if it sits at chevron for 12 months, how do you know it will work when you need it?.

[JBR]

Thankyou to everybody for your answers. I´ve understood importance to set correctly votes.

The final configuration will be:

First ES45 Active (votes=1, os_flags 0,0)
Second ES45 Active (votes=1, os_flags 1,0)
Quorum disk, qdskvotes=1
Expected Votes=3

When I need to make a failover .... Shutdown first one, shutdown second one, boot second ES45 from sys0 and boot first from sys1.

But if I perform two simultaneous boots on two ES45 from root sys0, by error .... Will they to continue to crash with cluexit bugcheck? I suppose that the answer is YES.

[Robert Gezelter]

Macero,

With all due respect, I do not agree with the proposed procedure for recovering in the event of a failure of the primary system. Shutting down each node and rebooting the alternate node from the primary's system root means a restart time on the order of minutes, at best.

The way that OpenVMS clusters were designed to failover uses the fact that the functions can be failed over to another cluster member without the need to restart the other cluster member. This can be initiated automatically, using a program or a background batch job; or it can be done manually using a command procedure specifically tailored for that particular application. It is important to realize that "failure" is not a all or nothing proposition, it is quite possible to move one group or one application from one node in a cluster to another, without shifting all of the load to the other, or in the case of a cluster that is larger than two active members, re-distribute the load among the other members.

My recommendation would be to carefully look at the application and how it interacts. In working with clients to architect, design, and implement OpenVMS clusters since they were first announced in 1982, I have almost always found solutions that did not require the re-boot of the entire cluster.

I have also found that solutions that require manual intervention tend to be highly prone to operational error and mistakes, and are best to be avoided as much as possible.

I hope that the above is helpful. If I have been unclear, please feel free to followup in this forum or privately.

- Bob Gezelter, http://www.rlgsc.com

[JBR]

Hi Bob, thank for your quickly reply, but the application only can run in one node because it has a restriction due "Decnet Mac Address" (BaseStar + PLC´s + PL/I, etc) and, at the moment, due application design, it´s impossible to use Decnet Cluster Alias.
I can assume a lost of service by minutes but I need to assure the integrity of data. (In the past, NO CLUSTER, one node Active, other StandBy "P00>>", by error, boot two nodes simultaneous from same system disk = data corruption)

My question is simple: Having a OpenVMS cluster, Can I assure that booting two nodes simoultaneous from the same system disk & same root (sys0) will produce a cluexit bugcheck in one node and no data corruption?

Thank you

Best Regards.

[Robert Gezelter]

Macero,

With all due respect, I would want to personally check that dependency on DECnet address. I have been told of many such dependencies in the past, and have found that most were an illusion. Also, I have dealt with many applications that stated "This cannot be run in a cluster". Investigation revealed that the actual restriction was "Can only be run on one cluster node at a time." (Which I was able to implement without a problem, thus creating fast failover without human intervention).

I do not have a cluster handy at this instant that I can use to verify what you ask.

- Bob Gezelter, http://www.rlgsc.com

[Hoff]

DECnet MAC address? Do research it. That could easily be a misunderstanding about having two NICs active using DECnet Phase IV on the same LAN, and there are easy and effective ways to deal with that.

And your failover scheme is likely to be more difficult, as there tend to be small differences -- NIC hardware addresses, et al -- that can mean booting a different node from an existing and established root can be problematic.

If I had to swap DECnet addresses, I'd set up a way to swap DECnet address. This can be done without rebooting, too. Rebooting and nodes sharing roots is not a solution I would typically recommend -- and this is an inherently risky solution, in my opinion.

I'd also encourage a more general configuration review here, too. There can be other opportunities to improve uptime, and reduce overhead.

Stephen Hoffman
HoffmanLabs LLC