topic Re: One cluster, two subnets - help! in Operating System - HP-UX

One cluster, two subnets - help!

NSMD BT UNIX — Wed, 08 Oct 2003 07:48:35 GMT

[Sorry for the previous empty message - hit return too early]

Hi SG gurus,

We want to configure a cluster with 10 nodes, of which half are in one subnet (subnet 1) and the other half in another subnet (subnet 2). The two subnets are for client data only, a separate network is used for SG heartbeat traffic.

I was able to successfully run cmquerycl to scan all nodes and generate a config file. Now I run cmcheckconf to verify the configuration and this is what I am getting (the command is launched from a node in subnet 1):

Error: Unable to establish communication to node node_s2: 19

where node_s2 is a node is subnet 2. If I launch the command from a node in subnet 2, I get the same error on a node in subnet 1. If I comment out in the config file all nodes in subnet 2 and launch cmcheckconf from subnet 1, it works fine (and inversely, when the command is launched from subnet 2 with the nodes in subnet 1 commented out).

My question is thus: is it possible to have an SG cluster that spans multiple subnets for client data?

The SG version is A.11.15 and the nodes are Superdome vPars. I have a case open with HP support on this. Any help is appreciated.

Thanks in advance,
Dmitri

Re: One cluster, two subnets - help!

Michael Steele_2 — Wed, 08 Oct 2003 07:52:04 GMT

General rule of thumb for building ServiceGuard: Stay within one subnet.

Heartbeats go out through every lan and are tested during 'cmquerycl' with 'linkloop MAC'.

MAC's are going to travel through a router well.

Re: One cluster, two subnets - help!

Michael Steele_2 — Wed, 08 Oct 2003 07:53:04 GMT

MAC's ARE NOT going to travel through a router well.

Sorry, still working on my first cup.

Re: One cluster, two subnets - help!

Geoff Wild — Wed, 08 Oct 2003 09:02:24 GMT

You will have to have lan cards configured on each server into each subnet to make this work....so, you will need at a minimum for each server:

2 Lan cards for Subnet 1
2 Lan cards for Subnet 2
1 Lan Card for Heartbeat (Subnet 3)

I mention the heartbeat as you will need a network that all servers see at the same time...

Rgds...Geoff

Re: One cluster, two subnets - help!

Sanjay_6 — Wed, 08 Oct 2003 09:13:05 GMT

I would suggest you get atleast three cards on each system. 1 for subnet 1, other for subnet 2 and let the third be failover. Let both the cards (subnet 1 & 2) carry the heartbeat, so you have multiple heartbeats.

You can configure the floating ips for the packages in different subnets.

Hope this helps.

Regds

Re: One cluster, two subnets - help!

John Poff — Wed, 08 Oct 2003 09:13:42 GMT

Hi,

We have a cluster built with this configuration. Geoff has it exactly right. You'll need multiple LAN cards that can see each subnet. It will work. As always with MC/SG, get the LAN configuration correct and SG will be happy.

JP

Re: One cluster, two subnets - help!

NSMD BT UNIX — Wed, 08 Oct 2003 10:01:54 GMT

Thanks everyone for your replies.

One thing I forgot to mention: the packages are configured in such a way that for each primary node there is one failover node *in the same subnet* (never in another subnet).

Referring to what Geoff said, my goal is to have:
- for nodes in subnet 1: 2 lan cards in subnet 1 (no cards in subnet 2)
- for nodes in subnet 2: 2 lan cards in subnet 2 (no cards in subnet 1)
- for all nodes: 1 lan card on subnet 3 for the heartbeat.

Can this configuration work given that there is never a failover between subnet 1 and subnet 2 (and inversely)?

Re: One cluster, two subnets - help!

Rita C Workman — Wed, 08 Oct 2003 11:50:04 GMT

Or you try using the Continental Cluster software for MC/SG. Since the failover to a different subnet is what it does.

Just a thought,
Rita

Re: One cluster, two subnets - help!

James Randall — Fri, 10 Oct 2003 12:49:19 GMT

You might want to rethink your plan and create two seperate clusters.

This would eliminate the cross-subnet issue you are talking about.

If you will not be trying to cross a package from subnet1 to subnet2 (or vice versa).....Why does it need to be one cluster?

Re: One cluster, two subnets - help!

Marcel Kuiper — Thu, 23 Oct 2003 10:59:55 GMT

The reason we are implementing one cluster instead of two or more is licensing costs (we are using MetroCluster with EMC/SRDF and the license is on a per-cluster basis).

At the time of the original posts, I had multiple problems in my configuration:

1) Not all network interfaces were declared in the cmclnodelist file. The service (client data) interfaces were declared but the heartbeat interfaces were missing.
2) Linkloop problems between some service interfaces and their standby interfaces.

As soon as I resolved these issues, everything worked fine. I was able to compile and run the cluster and all the packages.

So, to resume, it *is* possible: two subnets inside of one cluster, provided that packages are not crossed between the two subnets.

Thanks everyone for your replies :-)

Re: One cluster, two subnets - help!

Brian Markus — Fri, 24 Oct 2003 00:11:38 GMT

We use APA (Auto Port Aggregation) I have lan100, etc.. two cards meshed into one for redundancy on subnet 1, and the same on subnet 2. Total of 5 network connections per machine. 2 = subnet1 2=subnet2 1=heartbeat. It's been working great for 2 years now. It's kind of a pain to setup at first, but once it's done it's rock solid.

Hope this helps.

-Brian.

Re: One cluster, two subnets - help!

NSMD BT UNIX — Fri, 24 Oct 2003 04:09:48 GMT

Brian, does this mean that you do not have any standby interfaces configured in the cluster?

Seems to me like a good alternative.

Re: One cluster, two subnets - help!

Brian Markus — Sat, 25 Oct 2003 01:19:29 GMT

On that machine I felt confident in APA to cover redundancy. We have had a card fail and have absolutely no down time.

Here is part of my cmclconf.ascii file.

# Link Aggregate lan100 contains the following port(s): lan3,lan1
# Possible standby Network Interfaces for lan100: lan0.

I personally have been saved several times by APA thus far, and build all my boxes with it when I can.

Good luck

-Brian.

Re: One cluster, two subnets - help!

Pramod_4 — Mon, 27 Oct 2003 11:30:09 GMT

Dmitri,

I was wondering whether you can post your cluster.ascii file here.

There is no reason why you can not have a cluster with two subnets and a dedicated heartbeat lan.

Thanks,
Pramod