topic Re: Inform nodes on shutdown in Operating System - OpenVMS

Inform nodes on shutdown

Peter Zeiszler — Thu, 01 Mar 2007 16:25:36 GMT

I know there is a logical for inform nodes during shutdown. I would still like system to be notified whenever ANY node shutsdown. Besides modifying sys$system:shutdown.com is there any other logical that would alerts SYSTEM that another node is shutting down?

The reason I want this is that I have multiple nodes in a cluster. Some of these are hard partitions and I do not have a console manager connected to them with an active output all of the time. If someone shuts down one of my systems I want that message displayed on ALL nodes in the cluster at least captured on the console.

Any ideas?

Re: Inform nodes on shutdown

EdgarZamora_1 — Thu, 01 Mar 2007 16:57:59 GMT

Not sure if I understand you fully. You want the SYSTEM account alerted when someone shuts down a node. Via an email? An OPCOM message?

I understand you don't want to modify SHUTDOWN.COM. Why not include something in SYSHUTDWN.COM to maybe send SYSTEM an email? Not foolproof (won't work if system crashes or if the person shutting down the system chooses not to invoke the site specific shutdown procedure). And in case you forward SYSTEM VMS mail to Exchange, you might put a little bit of a delay in the SYSHUTDWN to allow your message to get out.

Hope that helps.

Re: Inform nodes on shutdown

Dale A. Marcy — Thu, 01 Mar 2007 17:02:31 GMT

I think if you define SHUTDOWN$INFORM_NODES in Sylogicals to be the nodes you want notified, it should work like what you are asking if the system is shutdown normally. That is what we do to limit the shutdown notices only to the node involved in the shutdown on our cluster, because users frequently don't see the node name and panic that we are taking the system out from under them.

Re: Inform nodes on shutdown

Hoff — Thu, 01 Mar 2007 17:24:56 GMT

The sys$setcluevt system service seem potentially applicable, if you are willing to write a little code.

In earlier times, I've used DECnet task-to-task for this same purpose, as well, and waited around for the NETSHUT event to arrive. You have to be slightly careful to distinguish if this is a network shutdown or a cluster node shutdown, but you'll certainly get notified.

Others have pointed to SYSHUTDWN.COM and that will work fine, though that approach won't capture crashes, OPCCRASH or other such exits.

The other approach is to monitor the cluster connection manager messages, though that would be comparatively ugly.

And the brute-force approach is to use the lock manager, and roll your own cluster monitor. At its simplest, that's a couple of days of code and debug work, and probably less.

Stephen Hoffman
HoffmanLabs

Re: Inform nodes on shutdown

atul sardana — Thu, 01 Mar 2007 18:00:12 GMT

Dear Peter,

I think you just want to see,which node is going to shut or out of cluster from a cluster node because you have only one console for monitoring all cluster nodes.
If i am right then you can give a command on system prompt

$>>show cluster/continue

Through this command you can see in the output of all nodes of a cluster like this

SYSTEM_LILAC>>sho cluster/continue

View of Cluster from system ID 10261 node: LILAC 2-MAR-2007 04:16:50
***********************************
* SYSTEMS * MEMBERS *
***********************************
* NODE * SOFTWARE * STATUS *
***********************************
* LILAC * VMS V7.3-2 * MEMBER *
* LOTUS * VMS V7.3-2 * MEMBER *
* TULIP * VMS V7.3-2 * MEMBER *
* JASMIN * VMS V7.3-2 * MEMBER *
* ORCHID * VMS V7.3-2 * MEMBER *
***********************************
and after given command to shut for any node then that node removed from this list. This output is refreshed after seconds and you can screen refresh rate by given command after press on this prompt

Command>SET INTERVAL = seconds
command>exit

Thanks,
Atul Sardana.

Re: Inform nodes on shutdown

atul sardana — Thu, 01 Mar 2007 18:10:01 GMT

Dear Peter,

In any cluster node you can also connect only one dump terminal through serial console for montoring and gives a simple command for all ....

$>>Monitor Cluster

and you can see output of all cluster nodes
CPU busy and Memory utilization also.

Thanks,
Atul sardana.

Re: Inform nodes on shutdown

Peter Zeiszler — Thu, 01 Mar 2007 19:22:49 GMT

The ideas of show cluster continous is what I currently do when I have a planned reboot.

What I wanted was the consoles - which are always monitored and logged in as system - get an alert on the nodes that aren't going down. I don't want a show cluster/continous on those consoles because it keeps that data logged.

I can't do the mail send thing unless I put in a delay because the queue manager usually kills the information before it can send it out.

I guess I will setup syshutdown to do a reply/user=system type message.

Re: Inform nodes on shutdown

John Gillings — Thu, 01 Mar 2007 23:54:26 GMT

Peter,

For any program that paints a page like SHOW CLUSTER/CONTINUOUS, you can use PIPE to send the output stream into code to monitor the output for changes, and do whatever you like when one is detected.

I've attached a simple example monitoring SHOW CLUSTER/CONTINUOUS. When a node goes down, the status will change.

You could run this in batch, or on the console (maybe even as a subprocess). All the output of the same lines that would have been logged are simply swallowed by the monitoring thread. This could be run in batch, possibly on multiple nodes.

Re: Inform nodes on shutdown

labadie_1 — Fri, 02 Mar 2007 01:50:20 GMT

Hello John

Thanks for the program, it is fun.

A little dcl loop using f$csid should also give me the info that a node is no longer in the Cluster ?

Re: Inform nodes on shutdown

John Gillings — Sun, 04 Mar 2007 18:17:59 GMT

Re: Gerard

>A little dcl loop using f$csid should also
>give me the info that a node is no longer
>in the Cluster ?

True, and it would be cheaper because it would avoid the pipeline processing. On the other hand, the logic is a bit more complex as you potentially have to look for something missing from an F$CSID loop. In this case we can exploit the logic in SHOW CLUSTER. Note that the code I posted is self configuring - it will work, as is, on any cluster - no need to code node names or any other site specific details.

The technique used in my example is very general. It can be used to monitor the output of virtually any process, looking for changes in an updated display. I've used it in places where I want to avoid large volumes of log files repeating the same output, and where I want to get immediate notification of an event which can be seen in a monitor program, but not easily available from elsewhere (in my case it's RTR link state changes). All that is required is the logic to parse the display.

Re: Inform nodes on shutdown

Mrityunjoy Kundu — Mon, 05 Mar 2007 05:09:40 GMT

You can define the logical name SHUTDOWN$INFORM_NODES to be a list of OpenVMS Cluster nodes that
are notified when the system is shut down. You must define SHUTDOWN$INFORM_NODES before
executing SYS$SYSTEM:SHUTDOWN.COM.
To define SHUTDOWN$INFORM_NODES, enter a command in the following format:
DEFINE SHUTDOWN$INFORM_NODES "node-list"
where node-list specifies the list of nodes to be informed. For example:
$ DEFINE SHUTDOWN$INFORM_NODES "NODE1,NODE2,NODE3"
If you plan to inform the same nodes every time you shut down the system, add the command to the
site-specific startup command procedure SYLOGICALS.COM.
If you define SHUTDOWN$INFORM_NODES, all member nodes included in the list are notified when you
execute SHUTDOWN.COM. Users on the node that is being shut down are always notified, regardless of
whether you define SHUTDOWN$INFORM_NODES. If you omit the name of the node that is being shut
down from the list specified in the DEFINE command, SHUTDOWN.COM automatically adds the name to
the list.