topic Re: lan failure simulation in Operating System - HP-UX

lan failure simulation

christian_2 — Thu, 24 Jan 2002 11:44:27 GMT

Hi guys,

I'm doing lan link tests and I got an unexpect result. When I disconected all cables from one
machine, the other, that has all cables conected rebooted and the one that has no cables got the lock and still active. the
command cmviewcl show that lan interfaces wasnt
down.

Have someone pass throught this problem ?
Now, I'm just using core-io lan, HP-UX 11 and
MCSG 11.07

thanks in advace!

Re: lan failure simulation

BFA6 — Thu, 24 Jan 2002 11:57:27 GMT

Hi,

Have you checked syslog.log for error messages from cmcld ?

Hilary

Re: lan failure simulation

Paula J Frazer-Campbell — Thu, 24 Jan 2002 11:59:43 GMT

Christian

I do not run service guard but it looks like a heartbeat setting or a patch related to heartbeat.

Paula

Re: lan failure simulation

BFA6 — Thu, 24 Jan 2002 12:09:40 GMT

Christian,

Another thought.
If node A is running package & has cluster lock disk, and you pull lan cables from A (both data & heartbeat), both machines will think the other has died & race for the cluster lock disk. A already has it, B won't get the lock disk, so will TOC.

Hilary

Re: lan failure simulation

David Navarro — Thu, 24 Jan 2002 12:13:10 GMT

Hi, this sounds like a patch problem. Be sure you have installed last patches for SG and LAN, etc..

David.

Re: lan failure simulation

David Navarro — Thu, 24 Jan 2002 12:20:50 GMT

Hi, again, I have read Hilary answer,and I agree with their opinion. If you disconnect all cables, communications betwen nodes are lossed, then cluster must be reformed, then first machine that get cluster lock. will form the new cluster.

A good test, can be disconnect just one network cable, look if traffic and addresses are conmuted to the other one. Then disconnect this other, I think package will be transferred to alternate node in this situation.

David.

Re: lan failure simulation

Bill McNAMARA_1 — Thu, 24 Jan 2002 12:38:07 GMT

thats normal.. service guard check this..

rather than ifconfig lan down
use lanadmin to reset the lan.
That should simulate it.

Or pull cable or wet the card!
I prefer lanadmin!

Later,
Bill

Re: lan failure simulation

melvyn burnard — Thu, 24 Jan 2002 12:56:43 GMT

What you are seeing is normal behaviour, based on the fact that you have Multiple Points of failure.
As previously said, if the node that stayed up already had access to the cluster lock disc, because ALL communiactions were lost, that node managed to grab the cluster lock diosc BEFORE the other node, and hence stayed up, forcing the other node to TOC.

You should definitely check you ar patched to the lates possible level, bearing in mind that SG patches are NOT included in the Patch Bundle CD's!

Re: lan failure simulation

Sanjay_6 — Thu, 24 Jan 2002 13:55:02 GMT

Hi,

Take a look at this thread from the SG FAQ, which tries to explain the scenario you are having,

http://docs.hp.com/hpux/onlinedocs/ha/haFAQindex2.html#All%20Networks%20fail,%20which%20node%20wins?

Here is the FAQ,

http://docs.hp.com/hpux/onlinedocs/ha/haFAQindex2.html

Hope this helps.

Regds

Re: lan failure simulation

christian_2 — Sat, 26 Jan 2002 14:02:31 GMT

thank you guys, but this
is not an correct behave.
there was a GOOD node on
cluster and this should
be up, get the lock and
take the ownership of
all packages running on node
that has all cables disconected. this one, that
have cables disconect should not get the lock because his
networks were down. if someone
have tested this on both nodes, please notify me

thanks again

Re: lan failure simulation

Duncan Edmonstone — Sat, 26 Jan 2002 22:05:52 GMT

No this *is* good behaviour when you consider what ServiceGuard is designed to do:

- Protect against single points of failure
- Under no circumstances corrupt your data

What you have simulated is a situation with multiple points of failure, consider what could be going on in this scenario:

Both nodes cannot talk to each other except via cluster lock - what is very clearly a network failure on one machine to you does not necessarily appear to be from the other machine - consider that from the point of view of the other machine it can still see the network but doesn't know what state the other node is - it could be that the multiple point of failures is networking equipment between the two nodes - now both nodes still have active networks (and presumably some clients can connect to each) but they can't talk to each other. If the nodes behaved as you have suggested they should then they would BOTH beleive they are the good node, and both try to run your package - not good for the integrity of your data.

This is an immutable rule of ServiceGuard in a two node cluster - when either node doesn't know what the other is doing, it's cluster lock time, and it won't always be the genuinely good node that wins.

HTH

Duncan

Re: lan failure simulation

christian_2 — Mon, 28 Jan 2002 12:18:25 GMT

thanks duncan, but
I still belive that is not
correct. If the node has
all networks down, it should
not stay running and the
other that has all cables ok
call the TOC. Imagine this
situation in a production
environmet. An node, that
is all right with it, call
the TOC unexpect. I agree
with this situation if BOTH
have at least one network up.
I will open an case in HP call
center. Thank you all

Re: lan failure simulation

melvyn burnard — Mon, 28 Jan 2002 12:52:36 GMT

You are incorrect, in that this is normal SG behaviour. The fact is you are talking about having MULTIPLE points of failure, not a Single one.

If you wish to try to prevent this, then you could consider setting up a serial heartbeat link, as this is designed to do exactly what you are looking for, i.e. ensure the node with the good network connections stays up.
There are some not good things about having a serial heartbeat as well.
The bottom line is that you are pullling ALL network connections, and if your network redundancy design is such that this could happen, I would redesign the networking area.

Opening a call with HP should give you the same answer.