topic Re: SG - Node reboot upon network cables disconnections in Operating System - HP-UX

SG - Node reboot upon network cables disconnections

Farid Abizeid — Fri, 01 Jul 2005 07:49:01 GMT

Hi,

I would like to have your feedback regarding this issue:
2 Node SG, HP-UX 11i v1 cluster using 2 RP3440 and an EVA storage.
1 NIC for HB, two remaining NICS used for data (1 active, 1 Stand by)

Upon removing all 3 network connections from the stand by node, the active node reboots by itself, furthermore the cluster will not start but manually.

Your response is highly appreciated

Regards,
Farid

Re: SG - Node reboot upon network cables disconnections

David Child_1 — Fri, 01 Jul 2005 08:10:14 GMT

Farid,

Refer to this document. It details a lot of senarios dealing with this;

UXSGLVKBAN00000010

(http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062686681)

David

Re: SG - Node reboot upon network cables disconnections

Farid Abizeid — Fri, 01 Jul 2005 08:15:21 GMT

I am sorry David, I cannot find the link.
can you please verify

Thank you for your fast response

Farid

Re: SG - Node reboot upon network cables disconnections

Farid Abizeid — Fri, 01 Jul 2005 08:27:46 GMT

It is OK, I found it, the HB loss seems to be our case, will check that
I wonder if there are newer updates on this issue

Thank you David

Farid

Re: SG - Node reboot upon network cables disconnections

John Poff — Fri, 01 Jul 2005 08:29:15 GMT

Hi,

This is normal behavior. MC/SG will TOC a node if it cannot communicate with a majority of other cluster member. It does this to protect data integrity.

Try this link at the HP Docs site. It is a section in the Managing MC/ServiceGuard manual, which explains what you are seeing:

http://docs.hp.com/en/B3936-90073/ch03s07.html#d0e4517

JP

Re: SG - Node reboot upon network cables disconnections

Farid Abizeid — Fri, 01 Jul 2005 09:11:46 GMT

Thank you Guys for your great support

Farid

Re: SG - Node reboot upon network cables disconnections

Farid Abizeid — Thu, 28 Jul 2005 06:09:36 GMT

Going back to this subject
When we disconnect all network connections from node1, the TOC occurs on node2
which makes the system unavailable.

One would expect the TOC to happen on node1
leaving node2 operational.

Your opinions are highly appreciated.

Regards,
Farid

Re: SG - Node reboot upon network cables disconnections

Kent Ostby — Thu, 28 Jul 2005 06:30:04 GMT

Farid --

Here is how it works.

When there is a network disconnect between the nodes, the nodes attempt to "reform" the cluster.

Since neither node can communicate with each other they cannot form a cluster of a majority of the nodes (i.e. they cannot form a cluster of 2 of the 2 nodes).

Serviceguard, therefore, uses a tie-breaking system to ensure that both nodes don't try to access the data (and hence corrupt it).

The tie-breaker is known as the lock disk (some systems use a quorem server).

Once the nodes realize that they cannot talk to each other, they "race" to the lock disk to try to get it.

In your case, the node with the "failed" network gets to the lock disk first.

When node 2 gets to the lock disk, it sees that it's already owned by node 1.

To ensure that there is no data corruption, node 2 kills itself with the TOC.

This is standard Serviceguard behavior.

Best regards,

Kent M. Ostby

Re: SG - Node reboot upon network cables disconnections

Farid Abizeid — Thu, 28 Jul 2005 07:12:39 GMT

Thank you Kent for your reply
So this is by design, can you think of a workaround ? or you believe that loosing all network connections at the same time is not a real life scenario.. would appreciate further comments.

On the other hand if you disconnect all FC connections to the SAN storage system of Node1, Node2 still works but the package does not failover to node2
Do you have an interpretation for this ?

Best regards,
Farid

Re: SG - Node reboot upon network cables disconnections

melvyn burnard — Thu, 28 Jul 2005 07:25:08 GMT

>So this is by design, can you think of a workaround ?
Yes, you coul duse the serial heartbeat for a 2 node cluster, but there are issues with this.

>or you believe that loosing all network connections at the same time is not a real life scenario..
Well this is essentially a Multiple Points of Failure, which Serviceguard is generally not designed to cater for.
The suggested option here would be to have all lans set to be HEARTBEAT_IP.

>On the other hand if you disconnect all FC connections to the SAN storage system of Node1, Node2 still works but the package does not failover to node2. Do you have an interpretation for this ?
Yes, the network manager does not monitor FC disk interfaces. To monitor this you need to look at using EMS monitors.
But again, this is an MPOF....

Re: SG - Node reboot upon network cables disconnections

Farid Abizeid — Fri, 29 Jul 2005 01:52:13 GMT

Thank you Melvyn for your answers
Do you have some official MC/SG test failure scenarios - this would be great then.

We will follow your suggestions to use EMS and all Network as Heartbeat_IP

Again thank you very much and best regards,
Farid

Re: SG - Node reboot upon network cables disconnections

melvyn burnard — Fri, 29 Jul 2005 02:36:54 GMT

Take a look at pages 320-323 of http://docs.hp.com/en/B3936-90079/B3936-90079.pdf

There are some suggested tests, and also a short discussion regarding using EMS monitoring.

Re: SG - Node reboot upon network cables disconnections

Richard Perez — Wed, 03 Aug 2005 12:13:53 GMT

Farid
"or you believe that loosing all network connections at the same time is not a real life scenario.. would appreciate further comments."

I would recommend looking at Quorum Server to decide which node remain up. With QS you will be sure that the node with IP connectivity will be up.