topic Re: Disabling node TOC? in Operating System - HP-UX

Disabling node TOC?

Sarah Nordstrom — Fri, 27 Mar 2009 13:12:24 GMT

Is there any (supported) way to completely disable Serviceguard rebooting a node entirely? We have a number of machines that run more than one service (some not using MC/SG), and when SG reboots a node, all the other services on the node are also killed.

Re: Disabling node TOC?

Matti_Kurkela — Fri, 27 Mar 2009 13:52:12 GMT

If your packages have the "service_fail_fast_enabled" flag set, this forces the node to reboot whenever a failure is detected. That would be a case of "you asked for it, you got it."

If you don't use the "service_fail_fast_enabled" setting, the only reason for SG to trigger a reboot is if all the heartbeat connections are lost and the node failed to get the cluster lock. If this happens often, you should work on improving the reliability and/or redundancy of your heartbeat connections.

In a robust cluster, a SG-triggered reboot should be a very rare event - and when it happens, the rebooting node will usually be isolated by network faults so it could not provide any services to anyone at that time anyway.

Avoiding the reboots completely would require developing a new strategy for avoiding the split-brain situations. This strategy must be absolutely bulletproof: a wrong decision here means either not providing a service when you could, or corrupting your package disks.

MK

Re: Disabling node TOC?

Sarah Nordstrom — Fri, 27 Mar 2009 13:56:37 GMT

It is indeed a rare event, but when it happens (e.g. during a maintenance event) it can be catastrophic. It seems like in the situation where it would normally reboot (all HB links down and no cluster lock) it could just shut *itself* down, but that doesn't seem to be an option...

Re: Disabling node TOC?

Matti_Kurkela — Sun, 29 Mar 2009 16:42:46 GMT

The solution in that case is to tell ServiceGuard in advance that a node is going to go away, using the "cmhaltnode" command.

It causes all running packages on that node to shut down and failover to other nodes (if enabled and possible), *and* it prepares the rest of the cluster to accept that the halting node may go off-line after the halt operation is completed.

The "cmhaltnode" command is *not* in any way similar to the "shutdown" command: the OS of the node and any non-packaged services on the node will keep running, only ServiceGuard operations on the node are shut down.

If you have a two-node cluster and you use cmhaltnode to stop one node, check the syslog of the other node: you will see a message about shutting down the "safety time protection" (= the "deadman" panic reboot module) as the cluster transitions into single-node operation. After that, it's safe to do whatever you want with the halted node.

If this is not applicable to your situation, please explain in more detail. If we can find out exactly why the reboot is triggered in your situation, perhaps we can find a way to avoid it.

MK