topic Re: Ignore heartbeat failure in Operating System - HP-UX

Ignore heartbeat failure

Kris Knigga — Wed, 14 Oct 2009 19:36:51 GMT

I have a three-node and a two-node ServiceGuard cluster. My network folks are going to be doing a massive switch maintenance and I'm practically guaranteed to lose my heartbeat. They tell me they can't promise my redundant switches won't be rebooted at the same time.

Is there a way to tell ServiceGuard to hold tight and not TOC anything? Would disabling package switching on all packages be enough?

Re: Ignore heartbeat failure

Steven E. Protter — Wed, 14 Oct 2009 19:57:27 GMT

Shalom,

Yes there is a way.

Halt the cluster before this badness happens.

I would propose something else though.

Sit there on console and watch this happen.

This is a great failover test.

If I were not able to watch this process, I would cmhaltnode the nodes or cmhalt the entire cluster.

SEP

Re: Ignore heartbeat failure

Kris Knigga — Wed, 14 Oct 2009 20:08:12 GMT

Wouldn't halting the cluster shut down all packages?

Re: Ignore heartbeat failure

Raj D. — Wed, 14 Oct 2009 21:13:12 GMT

Krism

Depneding on your SG hertbeat configuration you can decide if the cluster can stay up during maintenance, if you have more lan and HEARTBEAT_IP configured , during loss HB lan cause the HB to failover to the other lan. Check HEARTBEAT_IP configuration in cluster.ascii file.
If do not have more HEARTBEAT_IP & if Heartbeat is not there the node will assume saftytimer expired and may cause TOC.

>Wouldn't halting the cluster shut down all packages?

- yes halting cluster it will shutdown all the packages and nodes as well.
- However manually doing it is better and there will be a visibilty what is happening to what packages,an are they shutting down proerly or not, & if the vgs are getting deactivated properly or not.

Cheers,
Raj.

Re: Ignore heartbeat failure

Michael Steele_2 — Wed, 14 Oct 2009 22:54:41 GMT

Hi

a) switch to a one node cluster
b) disable the failover procedure

Do you need the commands? See cmhaltnode, cmviewcl, and some other related commands first. Paste in what you find and I'll verify it.

Re: Ignore heartbeat failure

Kapil Jha — Thu, 15 Oct 2009 00:55:37 GMT

Any which ways if you network team is not sure if they would be able to keep network alive, so it does not mean if ur cluster is running or not if network goes down.

So better be you take downtime from your side.

You can move all the packages to one node and halt the other mode so no need for heartbeat :)

BR,
Kapil+

Re: Ignore heartbeat failure

Torsten. — Thu, 15 Oct 2009 04:32:10 GMT

Hi Kris,

I agree with Michael Steele.

To prevent any case of a split brain situation, stop the other nodes and run your package on a single node cluster only during this maintenance.

Re: Ignore heartbeat failure

Kris Knigga — Thu, 15 Oct 2009 16:31:17 GMT

Thank you all. I think the best course of action will be just to shut down all of the packages and then the cluster.