topic Re: Open VMS cluster issue in Operating System - OpenVMS

Open VMS cluster issue

Shehan — Fri, 29 Feb 2008 12:55:51 GMT

Hi All

I have configure two DS10 nodes cluster with one BA356 shelf.I have configured quorum dis in the storage. Cluster is seems to be working and "show cluster" command show both members. Also when I shutdown one node, it says brknode. But the problem is when I perform @sys$system:shutdown command to do a cluster shutdown, node which I issued the this command is hang while it is shutting down.

Anybody have idea about this ?

Regards
Nirukshitha

Re: Open VMS cluster issue

Peter_364 — Fri, 29 Feb 2008 13:06:46 GMT

Hi,

What do you mean, if you perform a cluster shutdown, does the other node hang or do you mean that the shutdown-node hangs? In the later case is he waiting for the other node to reach the same point where they both jump over the cliff, in the earlier case, some cluster parameters is not correctly defined which renders the other node hanging.

Re: Open VMS cluster issue

Shehan — Fri, 29 Feb 2008 13:19:06 GMT

Hi Peter

shutdown-node is hang in the process of shutting down.

Regards
Nirukshitha

Re: Open VMS cluster issue

Robert Gezelter — Fri, 29 Feb 2008 13:24:40 GMT

Nirukshitha,

Please check your VOTES-related parameters.

- Bob Gezelter, http://www.rlgsc.com

Re: Open VMS cluster issue

Volker Halle — Fri, 29 Feb 2008 13:42:35 GMT

Nirukshitha,

first of all, a CLUSTER shutdown requires, that you issue this command on ALL nodes of the cluster individually.

When you shut down the first of the 2 nodes for a 'cluster' shutdown and 'it hangs', what are the last messages on the console terminal of this node and the other node or on the terminal, on which you have invoked the shutdown procedure ?

Volker.

Re: Open VMS cluster issue

Shehan — Fri, 29 Feb 2008 13:52:29 GMT

Hi Volker

See below for last hang point

Shutdown options [NONE]: cluster

%SHUTDOWN-I-OPERATOR, this terminal is now an operator's console
%SHUTDOWN-I-DISLOGINS, interactive logins will now be disabled
%SET-I-INTSET, login interactive limit = 0, current interactive value = 1
%SHUTDOWN-I-STOPQUEUES, the queues on this node will now be stopped

%SHUTDOWN-I-SITESHUT, the site-specific shutdown procedure will now be invoked
%SHUTDOWN-I-STOPUSER, all user processes will now be stopped
%SHUTDOWN-I-STOPACMESRV, the ACME server will now be shut down
%SHUTDOWN-I-STOPAUDIT, the security auditing subsystem will now be shut down
%SHUTDOWN-I-STOPSMISRV, the SMI server will now be shut down
%SHUTDOWN-I-STOPSECSRV, the security server will now be shut down
%SHUTDOWN-I-REMOVE, all installed images will now be removed
%SHUTDOWN-I-DISMOUNT, all volumes will now be dismounted

Also Other node doesn't effect this shutdown. It seems to be sync issue.I have to reboot other node purposely. Otherwise it doesn't reboot from this cluster shutdown

Regards
Nirukshitha

Re: Open VMS cluster issue

The Brit — Fri, 29 Feb 2008 13:52:45 GMT

Nirukshitha,
If you select "Cluster Shutdown" when running shutdown.com, then this is the correct behaviour.

If you choose this option, then you "MUST" shutdown ALL nodes. The cluster nodes will all shutdown to a certain point, and then wait until the remaining nodes reach that point. Then all nodes take the last step together.

If you don't want this behaviour, then shutdown individual nodes normally, specifying "Remove Node". This will cause the remaining nodes to adjust their quorum so that they remain functional (assuming that VOTES and EXPECTED_VOTES are correctly set)

Dave.

Re: Open VMS cluster issue

Robert Gezelter — Fri, 29 Feb 2008 13:53:26 GMT

Nirukshitha,

Please allow me to clarify my previous posting.

Please check all VOTES-related parameters. This includes VOTES, QUORUM, and the parameters relating to the location of the quorum disk.

Incorrect settings of these parameters can lead to situations similar to those that are described in this post.

- Bob Gezelter, http://www.rlgsc.com

Re: Open VMS cluster issue

Volker Halle — Fri, 29 Feb 2008 13:57:55 GMT

Nirukshitha,

Dave is right. You need to issue the 'CLUSTER' shutdown command on ALL nodes of a cluster.

If there would be a problem with VOTES etc., the OTHER node would hang, if you shut down this one AND you'll see approriate messages on the console terminal.

Volker.

Re: Open VMS cluster issue

Peter_364 — Fri, 29 Feb 2008 14:09:14 GMT

Yes, it's what I meant by claiming that the node is waiting on the other node to reach the same point.
Perform a cluster shutdown on both nodes, and the first node that reach end-point wait for the other to reach the same point and they will together enter the console.

Re: Open VMS cluster issue

Hakan Zanderau ( Anders — Fri, 29 Feb 2008 14:10:49 GMT

The "Cluster Shutdown" option is a cool cat.

I have never used it working as sysman, but when I teach The OpenVMS Cluster course I show how it works.......suddenly the whole classroom silences.

Back to Nirukshithas problem.

As Robert said.."Check the parameters", just be be sure everything is correctly set.

$ MCR SYSMAN
$ SET ENV/CLU
$ PARAMETER SHOW VOTES
$ PARAMETERS SHOW EXPECTED_VOTES
$ PARAMETERS SHOW DISK_QUORUM

It should read "Votes = 1"
"Expected_votes = 3"
"Disk_Quorum = "diskname""

The CLUSTER SHUTDOWN is ONLY used when shuting down all members. It will only affect the node execting shutdown. It will not leave the cluster until ALL members have reached the same level, wich can be thought of as "hanging".

In a two-node cluster ( with quorumdisk ) you don't have to use REMOVE NODE either.
Quorum will be 2 anyway.

However if you want to shutdown both node at the same time, use SYSMAN...

$ MCR SYSMAN
$ SET ENV/CLU
$ SHUTDOWN /CLUSTER

This will shutdown all nodes in the cluster.

regards,

Hakan Zanderau
HA-solutions

Re: Open VMS cluster issue

Jess Goodman — Fri, 29 Feb 2008 20:50:58 GMT

A minor correction to Hakan's post...

To shutdown the entire cluster use:

$ MCR SYSMAN
SET ENVIR/CLUSTER
SHUTDOWN NODE /CLUSTER

Re: Open VMS cluster issue

Shehan — Mon, 03 Mar 2008 06:05:30 GMT

Hi Dave

The Problem is other node doesn't come to certain point. It remains without shutting down.I think shutdown doesn't sense to other node.

Regards
Nirukshitha

Re: Open VMS cluster issue

Willem Grooters — Mon, 03 Mar 2008 06:40:08 GMT

You have 2 nodes, 1 shelf containing the quorum disk. The question is: how is this shelf connected to the cluster? What are your settings of VOTES and EXPECTED_VOTES for each node and the quorum disk?

Just an idea.
In case the shelf is connected to one system, and shutting down that one, will cause the other node to hang because quorum is lost once the quorum watcher is down.

WG

Re: Open VMS cluster issue

Volker Halle — Mon, 03 Mar 2008 07:34:38 GMT

Nirukshitha,

you do issue the @SYS$SYSTEM:SHUTDOWN ... CLUSTER shutdown on both nodes, do you ?

What happens, if you shut down the 'other' node first ? There could be a problem in SYSHUTDWN.COM preventing/hanging the shutdown of the other node.

What are the last messages shown on the console of BOTH systems ?

Volker.