Strange IP routing configuration

[Mihails Nikitins]

Hi,

I have an application system with some strange IP routing configuration. Since vendor's docs provide only one example (without explanation), I'd appreciate any suggestions.

The system consists of two HP servers, each of them has two interfaces. (host1: 10.11.12.21/25, 10.11.12.141/25 and host2: 10.11.12.22/25, 10.11.12.142/25). Static routing routes outside these 2 networks perfectly but I don't understand how it can work.

Here is netconf example from one host

/etc/rc.config.d/netconf

HOSTNAME="host2"
OPERATING_SYSTEM=HP-UX
LOOPBACK_ADDRESS=127.0.0.1

INTERFACE_NAME[0]=lan0
IP_ADDRESS[0]=10.11.12.22
SUBNET_MASK[0]=255.255.255.128
BROADCAST_ADDRESS[0]=""
LANCONFIG_ARGS[0]="ether"
DHCP_ENABLE[0]=0

INTERFACE_NAME[1]=lan1
IP_ADDRESS[1]=10.11.12.142
SUBNET_MASK[1]=255.255.255.128
BROADCAST_ADDRESS[1]=""
LANCONFIG_ARGS[1]="ether"
DHCP_ENABLE[1]=0

ROUTE_DESTINATION[0]=default
ROUTE_MASK[0]=""
ROUTE_GATEWAY[0]=10.11.12.22
ROUTE_COUNT[0]=0
ROUTE_ARGS[0]=""

ROUTE_DESTINATION[1]=10.11.12.22
ROUTE_MASK[1]=""
ROUTE_GATEWAY[1]=localhost
ROUTE_COUNT[1]=0
ROUTE_ARGS[1]=""

ROUTE_DESTINATION[2]=10.11.12.142
ROUTE_MASK[2]=""
ROUTE_GATEWAY[2]=localhost
ROUTE_COUNT[2]=0
ROUTE_ARGS[2]=""

GATED=0
GATED_ARGS=""
RDPD=0
RARPD=0

Some essential lines from /etc/hosts (there are no naming services configured)

127.0.0.1 localhost loopback
10.11.12.21 host1 host1
10.11.12.141 host1 host1_LAN2
10.11.12.22 host2 host2
10.11.12.142 host2 host2_LAN2


netstat -r output

host2:/etc/rc.config.d#netstat -r
Routing tables
Dest/Netmask Gateway Flags Refs Use Interface Pmtu
localhost localhost UH 0 1261318 lo0 4136
host2 host2 UH 0 89177 lan0 4136
host2 host2 UH 0 0 lan1 4136
10.11.12.0 host2 U 2 0 lan0 1500
10.11.12.128 host2 U 2 0 lan1 1500
127.0.0.0 localhost U 0 0 lo0 4136
default host2 U 0 0 lan0 1500
*********************************************************

So, routing rules point to host itself. I guess that routing is working via something like ARP proxy, because I see IP addresses from other networks in ARP cache as having MAC address of our Cisco router (10.11.12.126, 0:10:7b:0:6d:c8)! I'm quite sure that our Cisco is not configured as ARP proxy.

Some lines from arp -a output

host2:/etc/rc.config.d#arp -a
outside1 (10.44.44.5) at 0:10:7b:0:6d:c8 ether
outside2 (10.45.45.4) at 0:10:7b:0:6d:c8 ether
host1 (10.11.12.21) at 0:10:83:f5:19:56 ether
host1 (10.11.12.141) at 0:10:83:f5:19:57 ether


By the way, /etc/hosts also looks strange .

10.11.12.22 host2 host2
10.11.12.142 host2 host2_LAN2

Usually it is required that one name can correspond only to one IP. When making lookup for host2, the first match will be returned. So, host2 cannot be resolved as 10.11.12.142. On the other hand, both IP addresses are being substituted by the same name in some output (e.g., netstat) that is not convenient.

Thank you in advance for any ideas!

[Wodisch]

Hello Mihails,

who says "one name can only correspond to one IP"?
There are to schools on that, but both are more custom than rule!
HP and the readers of the famous O'Reilly book "DNS&BIND" strongly believe that *ALL* IP-addresses of one station should resolve to the same name!
But usually you then have at least two names for each such interface:

1.2.3.4 name1 name1a curly
2.3.4.5 name1 name1b moe
3.4.5.6 name1 name1c larry

As an example of the advantage of that:
- think you have a system with multiple ip-addresses
- think some users login over those different ip-addresses
- think some users connect to somewhere else over those different ip-addresses

Now:
- all users on the local machine *see* the same name connected to *their* interface
- all users on these other stations see those connections coming from that single system

Your assumptions on "proxy ARPs" for routes with the metric 0 are ok.

And your "/etc/hosts" looks like this, I believe:

10.11.12.22 host2 host2_LAN1
10.11.12.142 host2 host2_LAN2

Hence, if you want a special interface, uses its name (the alias name):

ping host2_LAN1
or
ping host2_LAN2

HTH,
Wodisc

[Sanjay_6]

Hi Mihails,

Take a look at this thread. It might help you in understanding your route setup,

http://us-support.external.hp.com/cki/bin/doc.pl/sid=7681201206dbdd63fe/screen=ckiDisplayDocument?docId=200000053723989

Hope this helps.

Regds

[Tim D Fulford]

Mihals

I'm not 100% sure, so feel free to disagree but:

These boxes can only possibly see IP's on 10.11.12.0/24 subnet? This is not much different to the default without routing! With the exception that everything goes through lan0 interface (which is probably a bad idea.

A second thing is ip_forwarding turned on?
10.20
# nettune -l ip_forwarding
&
# nettune -s 1 ip_forwarding
11.x
# ndd -get /dev/ip ip_forwarding
&
# ndd -set /dev/ip ip_forwarding 1

Tim

[Mihails Nikitins]

Thanks for your comments!

Wodish wrote:
***
HP and the readers of the famous O'Reilly book DNS&BIND" strongly believe that *ALL* IP-addresses of one station should resolve to the same name! <...>
Your assumptions on "proxy ARPs" for routes with the metric 0 are ok.
***

OK, now I understand this idea. It looked strange for me, because when doing name->address resolving, the answer is always the first entry.

Plese comment your words about proxy ARPs. My assumption is 'It looks using proxy ARPs, but I don't understand where it's configured!'


To Sanjaj:
Thank you, I know these basic principles.

Tim Fulford wrote:
***
These boxes can only possibly see IP's on 10.11.12.0/24 subnet? This is not much different to the default without routing! With the exception that everything goes through lan0 interface (which is probably a bad idea.
***

In fact there are two subnets 10.11.12.0/25 and 10.11.12.128/25. Both systems works as cluster can use both subnets. Both subnets can be reached from other _outside_ networks. However, traffic goes outside through the first interfaces.

Imagine I want to send a packet to an outside network.
Routing rules should work

default->10.11.12.22->localhost

So the outgoing packet seems to arrive on the system's own loopback interface!!! What happens next? I see entries in ARP table. How the the system knows it should send packet to the router's ARP then? Nothing is told about gateway 10.11.12.126 in netconf file! As I see, no routing daemons are running.

ndd produces an error, is it normally?

#ndd -get /dev/ip ip_forwarding
operation failed, Invalid argument

Thanks in advance for more comments!

[Tim D Fulford]

Mihalis

My comment about the subnets was a bit too subtle. Sorry.

What I was trying to say was the configuration will let you see addresses in the range 10.11.12.1 to 10.11.12.254. (or network 10.11.12.0/24)

You currently have two subnets with the ranges
10.11.12.1 - 10.11.12.127 (10.11.12.0/25)
10.11.12.129 - 10.11.12.254 (10.11.12.128/25)

With NO ROUTING set you can see all addresses in BOTH of the above subnets.

With the routing you currently lave set (this is what I'm not 100% sure about). All the packets with no rule go via lan0 or 10.11.12.0/25 subnet, as this is the default route. You also have host routes bak to itself via lo0 (127.0.0.1, localhost or loopback).

As far as I can see you cannot see hosts outside the 10.11.12.0/24 subnet with your routing set! This is what I infere from the netstat -r. Could you give a netstat -rn as this command does not resolve hostnames & you get a better idea of what packets go where.

Tim

[Mihails Nikitins]

Yes, 'netstat -rn' gives better view...

#netstat -rn

Routing tables
Dest/Netmask Gateway Flags Refs Use Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 201604 lo0 4136
10.11.12.12 10.11.12.12 UH 0 810252290 lan1 4136
10.11.12.132 10.11.12.132 UH 0 324 lan2 4136
10.11.12.128 10.11.12.132 U 2 0 lan2 1500
10.11.12.0 10.11.12.12 U 2 0 lan1 1500
127.0.0.0 127.0.0.1 U 0 0 lo0 4136
default 10.11.12.12 U 0 0 lan1 1500

This routing should not work, but it works. It looks like something is listening to
loopback interface.

Unfortunately, I cannot find free utility 'lsof' compiled for 64-bit HP-UX 11. It may provide additional information.

[Ron Kinner]

You asked:

"So, routing rules point to host itself. I guess that routing is working via something like ARP proxy, because I see IP addresses from other networks in ARP cache as having MAC address of our Cisco router (10.11.12.126, 0:10:7b:0:6d:c8)! I'm quite sure that our Cisco is not configured as ARP proxy."

Actually this is normal behavior for a router. It doesn't know the real MAC address of the host (It just knows how to get to it) so it substitutes its own MAC for that of any host which is not on the local network. Makes it a lot easier for the router to do its job. It only has to listen for its own MAC and some broadcast stuff. Then it strips off the Ethernet header and reads the IP header to see where it goes. If it somehow knew the correct MAC (which would cause a big increase in traffic over the WAN) it would have to listen for hundreds or thousands of different MACs.

This is the only part of your routing that looks normal. How the thing is able to talk to any non-local network without a real default gateway is magic. I suspect that the HP talks to itself via the Ethernet and sends an ARP for the distant address and the router which is listening on the same Ethernet is responding. Not a really reliable routing system but it appears to work for you.

I presume a traceroute would show the same router involved each time.

Ron

[Mihails Nikitins]

Rusty,

I'm afraid I disagree with your ideas about MACs. In normal situation the host's ARP table (seen by 'arp -a') should contain MACs of recently contacted hosts in the same network including the MAC of the default gateway. That's all.


IMHO, if the host doesn't have any IP for default routing, the router should never accept any ethernet frames from it. On the second level the host asks by ARP: 'Hey, if someone has address 200.200.200.200, please tell me you MAC!". The router should be specially configured to reply with its own MAC: 'Give me your frame, I'll take care about it.'

Please note that other HP hosts on the same network do not have the router's MAC for concrete external addresses
in their ARP cache (see my previous postings).

[Ron Kinner · Accepted solution]

I guess I'm the "Rusty" you are talking to. (It's actually "Ron").

You don't say who makes your router but all Cisco routers with IOS 10.0 or higher will respond automatically with their own MAC address to an ARP for a host in a network that they know about. (Not sure what happens if they don't know how to get to a network.)

Per Cisco:
"The Cisco IOS software uses proxy ARP (as defined in RFC 1027) to help hosts with no knowledge of routing determine the media addresses of hosts on other networks or subnets. For example, if the router receives an ARP request for a host that is not on the same interface as the ARP request sender, and if the router has all of its routes to that host through other interfaces, then it generates a proxy ARP reply packet giving its own local data-link address. The host that sent the ARP request then sends its packets to the router, which forwards them to the intended host. Proxy ARP is enabled by default."
Ref:
http://www.cisco.com/univercd/cc/td/doc/product/software/ios122/122cgcr/fipr_c/ipcprt1/1cfipadr.htm#xtocid11

You can turn this behavior off with the interface command "no ip proxy-arp" but unless you do that's what you will get. Suspect it's the same for most other manufacturers.

On a Cisco, do a "show ip interface" command and it will tell you (on about the 10th line of the response)that proxy arp is enabled on each interface.

The only way you are going to get out of your local network without a gateway is by using an ARP to get the MAC to send it to. Because that is what you must be doing, your arp table associates the offnetwork hosts with the MAC address of the router. If you had a default gateway your ARP table would only show local stuff since your machine would know to send stuff offnet to the router and wouldn't waste time ARPing for them.

This setup is pretty much ancient history these days which is why I called it magic in my first post. I had to look it up to see exactly how it worked even tho I've been doing Cisco routers for almost 8 years. While it does make you immune to changes in the router's ip address it takes an extra couple of seconds to get anywhere that is not in the arp table so most people prefer to either set a gateway or run a dynamic protocol like RIP or OSPF.

Ron

[Mihails Nikitins]

Ron,

Sorry for misspelling your name (probably, I looked at some other page) and thank you very much for the last posting!

It was a great suprise for me to find that Cisco routers work as ARP proxies _by_default_. Since I always specify default gateways and there is no line 'ip proxy-arp' in Cisco configuration, it was hard to understand. Now it looks pretty simple. :-) I'm afraid that this is not a good idea to switch on ARP proxy by default on routers.

However, I still do not understand how the routing rules are executed on the mentioned HP-UX hosts. As I see in netconf, outgoing packets are rerouted to loopback interface. Why HP-UX decides to issue ARP request at this point and does not try to apply routing rules as endless loop?

BR,
Mihail