Inform nodes on shutdown

[Peter Zeiszler]

I know there is a logical for inform nodes during shutdown. I would still like system to be notified whenever ANY node shutsdown. Besides modifying sys$system:shutdown.com is there any other logical that would alerts SYSTEM that another node is shutting down?

The reason I want this is that I have multiple nodes in a cluster. Some of these are hard partitions and I do not have a console manager connected to them with an active output all of the time. If someone shuts down one of my systems I want that message displayed on ALL nodes in the cluster at least captured on the console.

Any ideas?

[EdgarZamora_1 · Accepted solution]

Not sure if I understand you fully. You want the SYSTEM account alerted when someone shuts down a node. Via an email? An OPCOM message?

I understand you don't want to modify SHUTDOWN.COM. Why not include something in SYSHUTDWN.COM to maybe send SYSTEM an email? Not foolproof (won't work if system crashes or if the person shutting down the system chooses not to invoke the site specific shutdown procedure). And in case you forward SYSTEM VMS mail to Exchange, you might put a little bit of a delay in the SYSHUTDWN to allow your message to get out.

Hope that helps.

[Dale A. Marcy]

I think if you define SHUTDOWN$INFORM_NODES in Sylogicals to be the nodes you want notified, it should work like what you are asking if the system is shutdown normally. That is what we do to limit the shutdown notices only to the node involved in the shutdown on our cluster, because users frequently don't see the node name and panic that we are taking the system out from under them.

[Hoff]

The sys$setcluevt system service seem potentially applicable, if you are willing to write a little code.

In earlier times, I've used DECnet task-to-task for this same purpose, as well, and waited around for the NETSHUT event to arrive. You have to be slightly careful to distinguish if this is a network shutdown or a cluster node shutdown, but you'll certainly get notified.

Others have pointed to SYSHUTDWN.COM and that will work fine, though that approach won't capture crashes, OPCCRASH or other such exits.

The other approach is to monitor the cluster connection manager messages, though that would be comparatively ugly.

And the brute-force approach is to use the lock manager, and roll your own cluster monitor. At its simplest, that's a couple of days of code and debug work, and probably less.

Stephen Hoffman
HoffmanLabs

[atul sardana]

Dear Peter,

I think you just want to see,which node is going to shut or out of cluster from a cluster node because you have only one console for monitoring all cluster nodes.
If i am right then you can give a command on system prompt

$>>show cluster/continue

Through this command you can see in the output of all nodes of a cluster like this

SYSTEM_LILAC>>sho cluster/continue

View of Cluster from system ID 10261 node: LILAC 2-MAR-2007 04:16:50
***********************************
* SYSTEMS * MEMBERS *
***********************************
* NODE * SOFTWARE * STATUS *
***********************************
* LILAC * VMS V7.3-2 * MEMBER *
* LOTUS * VMS V7.3-2 * MEMBER *
* TULIP * VMS V7.3-2 * MEMBER *
* JASMIN * VMS V7.3-2 * MEMBER *
* ORCHID * VMS V7.3-2 * MEMBER *
***********************************
and after given command to shut for any node then that node removed from this list. This output is refreshed after seconds and you can screen refresh rate by given command after press on this prompt

Command>SET INTERVAL = seconds
command>exit

Thanks,
Atul Sardana.

[atul sardana]

Dear Peter,

In any cluster node you can also connect only one dump terminal through serial console for montoring and gives a simple command for all ....

$>>Monitor Cluster

and you can see output of all cluster nodes
CPU busy and Memory utilization also.


Thanks,
Atul sardana.

[Peter Zeiszler]

The ideas of show cluster continous is what I currently do when I have a planned reboot.

What I wanted was the consoles - which are always monitored and logged in as system - get an alert on the nodes that aren't going down. I don't want a show cluster/continous on those consoles because it keeps that data logged.

I can't do the mail send thing unless I put in a delay because the queue manager usually kills the information before it can send it out.

I guess I will setup syshutdown to do a reply/user=system type message.

[John Gillings]

Peter,

For any program that paints a page like SHOW CLUSTER/CONTINUOUS, you can use PIPE to send the output stream into code to monitor the output for changes, and do whatever you like when one is detected.

I've attached a simple example monitoring SHOW CLUSTER/CONTINUOUS. When a node goes down, the status will change.

You could run this in batch, or on the console (maybe even as a subprocess). All the output of the same lines that would have been logged are simply swallowed by the monitoring thread. This could be run in batch, possibly on multiple nodes.

[labadie_1]

Hello John

Thanks for the program, it is fun.

A little dcl loop using f$csid should also give me the info that a node is no longer in the Cluster ?

[John Gillings]

Re: Gerard

>A little dcl loop using f$csid should also
>give me the info that a node is no longer
>in the Cluster ?

True, and it would be cheaper because it would avoid the pipeline processing. On the other hand, the logic is a bit more complex as you potentially have to look for something missing from an F$CSID loop. In this case we can exploit the logic in SHOW CLUSTER. Note that the code I posted is self configuring - it will work, as is, on any cluster - no need to code node names or any other site specific details.

The technique used in my example is very general. It can be used to monitor the output of virtually any process, looking for changes in an updated display. I've used it in places where I want to avoid large volumes of log files repeating the same output, and where I want to get immediate notification of an event which can be seen in a monitor program, but not easily available from elsewhere (in my case it's RTR link state changes). All that is required is the logic to parse the display.