Multiple default gateways, but network unreachable...

[Eric M. Johnston]

So, I've got a somewhat unique networking setup, which I cannot change. For simplicity's sake, let's say there are:

LAN A: 10.100.0.0/24
LAN B: 10.100.1.0/24
LAN C: 10.100.2.0/24

All three LANs are connected to a common router with the addresses 10.100.0.1, 10.100.1.1, and 10.100.2.1. My HP-UX box (B.11.23 U ia64) is connected to both LANs A & B. It has the addresses 10.100.0.10 and 10.100.1.10.

There's a third machine (also HP-UX, but in this context, I don't think it really matters) on LAN C with, say, the address 10.100.2.20.

The dual LAN A+B setup is to accomodate a critical application's [archaic] notion of network redundancy. It cannot change. The router has IP spoofing protection enabled.

Were I to choose, say, the LAN A interface as the default gateway on the LAN A+B machine, responses to a ping of the LAN B address from the LAN C machine are blocked by the router's spoof protection. Fair enough.

So, I have created two default gateways, one for each interface. It looks like this:

> netstat -rnv
Routing tables
Dest/Netmask Gateway Flags Refs Interface Pmtu
127.0.0.1/255.255.255.255 127.0.0.1 UH 0 lo0 4136
10.100.0.10/255.255.255.255 10.100.0.10 UH 0 lan0 4136
10.100.1.10/255.255.255.255 10.100.1.10 UH 0 lan1 4136
10.100.0.0/255.255.255.0 10.100.0.10 U 2 lan0 1500
10.100.1.0/255.255.255.0 10.100.1.10 U 2 lan1 1500
127.0.0.0/255.0.0.0 127.0.0.1 U 0 lo0 0
default/0.0.0.0 10.100.0.1 UG 0 lan0 0
default/0.0.0.0 10.100.1.1 UG 0 lan1 0

I also set ip_strong_es_model to 0.

Happily, this fixes my pinging from LAN C problem: I can ping either the LAN A or B addresses from LAN C without running afoul of the router's spoofing protection.

BUT, now I can't initiate connections from the LAN A+B box to LAN C. It's as if the OS doesn't know how to choose which gateway to use. If I open a connection from LAN C to the LAN A+B box first, then immediately initiate a connection to LAN C (presumably before a routing table gets flushed or whatever), things work, But, after a couple of minutes of inactivity, the LAN A+B box can no longer reach LAN C. E.g.,

> ssh 10.100.2.20
ssh: connect to host 10.100.2.20 port 22: Network is unreachable

I've tried this with ip_forwarding set to both 0 and 2, with the same result. I've tried running gated, with the same result (though I don't think the router is advertising routes).

Am I correct in assuming that the OS is paralyzed with indecision given the multiple gateways? Is there a way to tell the OS to prefer a default gateway in the absence of route info? (I tried setting the counts of the default routes to different values > 0, but no love.) How should the OS normally behave with multiple default gateways?

Thanks in advance for your guidance and wisdom...

Eric

[Eric M. Johnston]

In the above, I meant to say that I've set ip_strong_es_model to 1.

After further fiddling, I'm still unable to get HP-UX to use one of the default gateways for traffic originating from the LAN A+B machine.

Surely this is possible? Wouldn't this be the same setup as a host with redundant connections to the Internet?

Do you think it would help for me to have the router advertise its routes and use gated? (Not sure if that's possible, though.)

I should also mention that ip_ire_gw_probe is set to 0, as the router doesn't respond to ICMP echo requests. Could that be part of the problem?

TIA,

Eric

[Steven E. Protter]

Shalom Eric,

This is complex. It doesn't make me feel wise at all.

Note: default gateway. There can only be one. Look up the word default, that is how it is applied here.

Yes your router configuration could be contributing to this problem.

You can have multiple gateways but not multiple default gateways. The goal seems to be the problem.

SEP

[Eric M. Johnston]

Hi Steven,

Thanks for the response!

I understand what you're staying, and largely agree. However, my understanding is that one should be able to define more than one "default" gateway. How the OS chooses which gateway to use can depend on a number of things, such as a metric, link speed, etc.

I don't really care which gateway traffic originating from the box uses, but I do need responses to incoming requests to go back out on the same interface they came in on (hence ip_strong_es_model = 1).

I have discovered something interesting, though: if I define two static routes to LAN C, things (tentatively) seem to work. I.e.,

# route add net 10.100.2.0 netmask 255.255.255.0 10.100.0.1 1
add net 10.100.2.0: gateway 10.100.0.1
# route add net 10.100.2.0 netmask 255.255.255.0 10.100.1.1 2
add net 10.100.2.0: gateway 10.100.1.1

Somehow, our good friend HP-UX is choosing one of these routes. But, it doesn't seem capable of doing the same with two default routes. (!) What would the difference be?

[Eric M. Johnston]

Well, I've spent too much time on this problem already. So, I'm going with my static route workaround. The pain of it is that I have more than just the LAN C in my example above -- I need to get to 14 other LANs. So, that's 28 static routes on 5 HP-UX machines connected to LANs A+B.

Why on earth this should be necessary is beyond me. Unless HP-UX is ignoring the default gateways for some silly reason (gateway probing turned off, whatever), I'm inclined to think this is a bug.

(I hate to bring it up, but the Windows XP boxen connected to both LANs A & B do not exhibit this behavior. They work fine with two default gateways, one through LAN A, one through B.)

Thanks for reading,

Eric

[Christopher Caldwell]

IIRC, you can use more than one default route on HP-UX, but HP-UX only uses the last default route entered. Rick Jones might chime in to remind us if things are still that way.

Things are different on route gear. For instance, you can have more than one default route on a Cisco router, and it will load balance.

Default route basically says if you don't have a more specific route, then go to the default gateway destination.

I recommend that your default gateway be the most well connected network device you have on the same IP network as your host (i.e. a router, not an HP-UX host).

When you address LAN cards on HP-UX, you should see static routes built for your directly connected networks.

So, if you draw the thing up, then run netstat -rn, all of the possible destinations should appear in the routing table. If an explicit destination isn't in the routing table, it'll use the default route (last one entered on HP-UX)- and if that route doesn't lead you to where you want to go, things won't work.

In your config, 10.100.2.20 doesn't appear in the routing table, so we'll assume the default route gets used.

Does the router on 10.100.1.1 or 10.100.0.1 have a route to 10.100.2.20? (It should if it has an inteface defined on the 10.100.2 network).

Are you doing something funny with uRPF, so that the router barks at packets that don't seem to be sourced from the right interface?

For troubleshooting purposes, I'd kill one of the default routes (or at least try one, then try the other, but not both together).

[Volker Borowski]

Hi,
so when adding static, you provide a diffrent cost (hopcount 1 and 2), but when adding default you give the same cost (hopcount)(0) ...

Try adding the second default with higher cost, although it will not been chosen until LAN A goes down I think ?

Give it a try and monitor the hop count in "netstat -nr".

Volker

[Eric M. Johnston]

Christopher,

Thanks for the info! My router is passing along traffic between the networks just fine. The problem is, I want to reach both the LAN A and LAN B interfaces on my multihomed machine from LAN C without triggering the router's spoof denial. (I believe that's the uRPF of which you speak? Firewall guys won't let me turn it off -- "enterprise policy".)

This means I must set ip_strong_es_model to 1, to force packets to exit on the same interface they come in on.

However, if I have just one route (whether by the default gateway or by a static route) to LAN C through, say, LAN A's gateway, connections to the LAN B interface fail since responses from the LAN B interface don't know how to get to LAN C.

It doesn't appear to me that HP-UX uses the last default route entered -- it's not using either one. My original setup was with a single default route through one of the interfaces, which is what was not working.

Volker,

I've tried playing with the count, or cost. Ultimately, I saw no difference when giving either the default or the static routes different counts, and have settled on setting them all to 1. (They can't be zero, I found -- that count means the address is on the local machine.)

Thanks!

Eric

[Mark Kirkpatrick]

When we use the strong_es_model we also use the "source" route option on the route command to ensure that that the "default" route chosen is the one assigned to the interface the address is associated with.

We add addresses and routes allways together- to help minimize or eliminate the asymetrical routing problem.