Three NICs, two subnets, only default route works

[Charles Soto]

Okay, we've got a HP-9000 (rp5450) running 11.11. It is configured with two 4-port 10/100 NICs, which are functioning. Right now, I've got one port on each up, in addition to the CoreIO port. These are lan0, lan1 and lan5. Two of these are assigned IPs on the same subnet (lan0 and lan5). The remainder (lan1) is assigned an IP in a different subnet. The default route is set correctly for lan0 and lan5. I updated /etc/rc.config.d/netconf to supply a static route for the second subnet, but it will not work. This IP is unreachable from outside the default subnet. I must be doing something wrong.

Here are the relevant settings from /etc/rc.config.d/netconf (in the order they apper in this file):

INTERFACE_NAME[0]=lan0
IP_ADDRESS[0]=129.116.76.220
SUBNET_MASK[0]=255.255.255.192
BROADCAST_ADDRESS[0]=""
INTERFACE_STATE[0]=""
DHCP_ENABLE[0]=0

ROUTE_DESTINATION[0]=default
ROUTE_MASK[0]=""
ROUTE_GATEWAY[0]=129.116.76.193
ROUTE_COUNT[0]=1
ROUTE_ARGS[0]=""

IP_ADDRESS[1]=129.116.74.37
SUBNET_MASK[1]=255.255.255.224
INTERFACE_NAME[1]=lan1
BROADCAST_ADDRESS[1]=""
INTERFACE_STATE[1]=up
DHCP_ENABLE[1]=0

ROUTE_DESTINATION[1]="net 129.116.74.32"
ROUTE_MASK[1]=255.255.255.224
ROUTE_GATEWAY[1]=129.116.74.33
ROUTE_COUNT[1]=1
ROUTE_ARGS[1]=""

IP_ADDRESS[5]=129.116.76.221
SUBNET_MASK[5]=255.255.255.192
INTERFACE_NAME[5]=lan5
BROADCAST_ADDRESS[5]=""
INTERFACE_STATE[5]=up
DHCP_ENABLE[5]=0

This should work, right? Here's output from netstat -rn:

colossus 24: netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 4136
129.116.74.37 129.116.74.37 UH 0 lan1 4136
129.116.76.220 129.116.76.220 UH 0 lan0 4136
129.116.76.221 129.116.76.221 UH 0 lan5 4136
129.116.74.32 129.116.74.37 U 2 lan1 1500
129.116.76.192 129.116.76.221 U 2 lan5 1500
129.116.76.192 129.116.76.220 U 2 lan0 1500
129.116.74.32 129.116.74.33 UG 0 lan1 0
127.0.0.0 127.0.0.1 U 0 lo0 0
default 129.116.76.193 UG 0 lan5 0

It appears that subnet 129.116.74.32 lists the proper gateway. I even tried using "host" routing for this specific host. No change.

Any ideas will be rewarded with points!

Thanks,
Charles

[Sridhar Bhaskarla]

Hi Charles,

IP_ADDRESS[1]=129.116.74.37
SUBNET_MASK[1]=255.255.255.224
INTERFACE_NAME[1]=lan1

The above shows that this IP is already in the subnet of 74.32 and you don't have to add that route statement.

If you want to reach an IP ex., 74.10 which is not in the same subnet as of your system, then you would need to add the route if 74.33 is a router in which case your configuration would be

ROUTE_DESTINATION[1]="net 129.116.74.0"
ROUTE_MASK[1]=255.255.255.224
ROUTE_GATEWAY[1]=129.116.74.33
...

If you want to talk to a node say 74.76, then there will be another route as follows

ROUTE_DESTINATION[1]="net 129.116.74.64"
ROUTE_MASK[1]=255.255.255.224
ROUTE_GATEWAY[1]=129.116.74.33
..

See if it works. Basically you should be able to ping to 74.33 as it is in the same subnet as of your system.

-Sri

[John Palmer]

Hi Charles,

A couple of observations...

1. Two NIC's in the same subnet is not a supported configuration. It does work but sometime you get 'funnies'. The alternatives are to use Auto Port Aggregation (requires compatible switch) or to configure more than one IP address on the same NIC.
2. It's not recommended to leave gaps in the indexing of your shell array variable in netconf. Your IP_ADDRESS[5] etc should really be [2] as that's the next available number. In your case lan5 does appear to have been configured correctly though.

I presume that your problem is that remote devices can't talk to your lan1 card? This is because you haven't created any remote routes which use that NIC. The route you have declared 129.116.74.32 is the local subnet. All outgoing traffic to remote subnets is therefore going to go via your default gateway.

Regards,
John

[Charles Soto]

Okay. I have moved each NIC to a separate subnet. Now, I just need to have each subnet respond to it's particular gateway, in order to respond to traffic from outside its respective subnet.

Here's what I need to do:

NIC__IP_____________SM______________GW
lan0_129.116.76.220_255.255.255.192_default(129.116.76.193)
lan1_129.116.74.37__255.255.255.224_129.116.74.33
lan5_129.116.75.133_255.255.255.128_129.116.75.129

In each case, the gateway is an external router.

What's the proper configuration to accomplish this?

Thanks,
Charles

[Charles Soto]

Okay, guys! Thanks to all your "hints," I was able to figure it out. I had not set ip_strong_es_model to 1 (default is 0). This is why all of my route add testing wasn't working.

To make this permanent, I modified /etc/rc.config.d/nddconf by adding the following:

TRANSPORT_NAME[2]=ip
NDD_NAME[2]=ip_strong_es_model
NDD_VALUE[2]=1

(Use ndd -c to load these immediately).

With this set, I set the following parts of /etc/rc.config.d/netconf back to:

INTERFACE_NAME[0]=lan0
IP_ADDRESS[0]=129.116.76.220
SUBNET_MASK[0]=255.255.255.192
BROADCAST_ADDRESS[0]=""
INTERFACE_STATE[0]=""
DHCP_ENABLE[0]=0

ROUTE_DESTINATION[0]=default
ROUTE_MASK[0]=""
ROUTE_GATEWAY[0]=129.116.76.193
ROUTE_COUNT[0]=1
ROUTE_ARGS[0]=""

IP_ADDRESS[1]=129.116.74.37
SUBNET_MASK[1]=255.255.255.224
INTERFACE_NAME[1]=lan1
BROADCAST_ADDRESS[1]=""
INTERFACE_STATE[1]=up
DHCP_ENABLE[1]=0

ROUTE_DESTINATION[1]=default
ROUTE_MASK[1]=""
ROUTE_GATEWAY[1]=129.116.74.33
ROUTE_COUNT[1]=1
ROUTE_ARGS[1]=""

IP_ADDRESS[5]=129.116.75.133
SUBNET_MASK[5]=255.255.255.128
INTERFACE_NAME[5]=lan5
BROADCAST_ADDRESS[5]=129.116.75.255
INTERFACE_STATE[5]=up
DHCP_ENABLE[5]=0

ROUTE_DESTINATION[5]=default
ROUTE_MASK[5]=""
ROUTE_GATEWAY[5]=129.116.75.129
ROUTE_COUNT[5]=1
ROUTE_ARTS[5]=""

Here's the output of netstat -rn:

colossus 13: netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 4136
129.116.74.37 129.116.74.37 UH 0 lan1 4136
129.116.75.133 129.116.75.133 UH 0 lan5 4136
129.116.76.220 129.116.76.220 UH 0 lan0 4136
129.116.74.32 129.116.74.37 U 2 lan1 1500
129.116.76.192 129.116.76.220 U 2 lan0 1500
129.116.75.128 129.116.75.133 U 2 lan5 1500
127.0.0.0 127.0.0.1 U 0 lo0 0
default 129.116.75.129 UG 0 lan5 0
default 129.116.74.33 UG 0 lan1 0
default 129.116.76.193 UG 0 lan0 0

Now all interfaces are happily responding to traffic from outside their respective subnets. Nothing seems to be broke :)

Let me know if I'm doing something stupid, or if this looks kosher to you.

Thanks again!
Charles

[Ron Kinner]

Normally you only want one default route. The default route is the route of last resort. The default route is the route to try when you have no idea how to get somewhere. There is no way for the box to tell which of the three should be used so normally what it does is just use the first one which shows up in the netstat -r. (netstat -rv should show you which route is actually being used (by the number of uses or packets) (Actually it could sense to have more than one possible default route if the box were doing bad gateway detection and each default gateway knew how to get to the same networks but then you would want to give them each a different metric "count" so it can order them and not use 1 for each. Haven't tried it but I assume it would choose the lowest metric when picking a static route. )

The rule of thumb is the default route points to the best route to the internet. If there is no route to the internet then the default route should point to the gateway which knows how to get to the most different subnets. Then you add individual route statements for all other remote networks that you need to reach.

Ron