Dynamic Routing Hell

[Dermot Beirne]

I have a problem that has caused me grief on several occasions. I have a HP9000 running HPUX 10.20. This machine is our DNS server amongst other things. The routing table on the server is massive, when doing a netstat -r, it literly takes hours to go through them.
Whenever a minor change occurs on the network, this server has to have a ton of route add entries run to get connectivity back to certain remote devices that are trying to connect to it. It seems to pick up every route on the network and apply it to it's table itself. If a leased line to a remote site goes down and the ISDN (on a different router) kicks in, the server updates it routes to the new router, but when the line comes back up, the server does not divert its route back, we have to do route adds again. This is causing me a lot of problems, so I have some questions:

1. How do I list the servers routes for a particular network, and not have to list the whole table.

2. How can I delete all routes in the table for any IP address starting with a certain octet, regardless of destination router.
e.g.
Say I have 3 routes as follows:
1. 140.xx.1.1 gw 10.xx.xx.5
2. 140.xx.2.0 gw 10.xx.xx.120
3. 140.xx.52.33 gw 10.xx.xx.35

I want to remove all routes for any 140 address regardless of the different gateways they are using. Can I do this?

3. Is there something I can do to stop the server dynamically updating it's routing table, or stop listening to the routers broadcasts which are causing this. I would rather have full control of the table manually, and simply add a route add net for my particular subnets, and know that it will not change if a router update specifes another one. I want the server to use it's default gateway only, and nothing else unless it tell it manually.


I know this is long, but I'd really appreciate any assistance.

[Uday_S_Ankolekar]

Why do you want to have multiple router entry anyway?

Your DNS server should talk to only one router in your LAN and that router should handle all the routings for the additional lans.
Configure your DNS server to talk to a default gateway that is the 'gateway' router in your lan.

What is your default gateway in DNS server?

/etc/rc.config.d/netconf is the file that contains all the network related configuration.

Post your netconf file.

Good luck,
-USA..

[Patrick Wallek]

It sounds as if you may have the router discovery protocol turned on in your /etc/rc.config.d/netconf. Check this file the the 'RDPD=??' line. If it is set to 1, then you might consider setting it to 0 to turn it off. You could also do a 'ps -ef | grep rdp' to see if the rdpd is running.

If you turn off rdpd then you shouldn't get the automatic route updates that you are seeing now.

Another possibility is that you have gated running. Check your /etc/rc.config.d/netconf file and see if you have a GATED=1 line. If so, you could try turning this off as well (set GATED=0). If you need to run gated, have a look at your /etc/gated.conf file. There may be some changes you can make there to help yourself. 'man gated.conf' for more information.

I know of no way to delete all entries in the route table for a particular subnet. Unfortunately.

[Jeff Schussele]

Hi,

I agree with USA.
This really should be a function of your network team.
You should only have to define a default gateway for the "main" I/F. Then any static routes for traffic that comes to this system on different I/Fs.
Seems to me that you have to maually do what should be automatic if the network were designed properly.
I'd engage your network team to clean this situation up as it will only probably get worse.

To the matter at hand. What you'll need to look at for this - on your end - would be the ndd command. And specifically the parameters that deal with dead gateways - they are:
ip_ire_gw_probe
ip_ire_gw_probe_interval
ip_ire_redirect_interval

To deal with routing look at:
ip_ire_hash
ip_ire_status
ip_ire_cleanup_interval
ip_ire_flush_interval

HTH,
Jeff

[Dermot Beirne]

I don't want multiple routes, that the problem. I have set a default gateway, but the server simply adds a route to it's table for different routers on the same LAN for the networks that they connect to. I want it to ignore any redirect or route update messages that are running on the network, (i cannot disable them on the routers, they are necessary between the routers) and i want it to always use it's default gateway for everything.
Dermot.

[Dermot Beirne]

Patrick,
Both RDPD and Gated are set to 0 in netconf.

I will do some research into the parameters that Jeff mentioned.

Thank you.

By the way, I found the following command on a previously asked, very similar question in the forums.

netstat -rn | awk '$3=/UGHD/ {system ("route delete "$1" "$2" ")}' > /dev/null

It removes all dynamically learned routes from the table.
Dermot.

[Ron Kinner · Accepted solution]

Appears that you have gated or routed running. It's best to let your friendly default router take care of the routing. That's what it is there for.

In the file
/etc/rc.config.d/netconf
you need to change
GATED=1
to GATED=0
or
ROUTED=1
to ROUTED=0

While you are in the file make sure you have a default route configured:

ROUTE_DESTINATION[0]="default"
ROUTE_MASK[0]=""
ROUTE_GATEWAY[0]="A.B.C.D"
ROUTE_COUNT[0]="1"
ROUTE_ARGS[0]=""

Save the file and then do inetd -c. That causes it to reread the netconf file but it may not stop the gated/routed program. You may have to kill it manually. So your best bet may be to reboot if you can as that will also clean out your manually and dynamically added routes.

You can flush the routing table with
route -f
This gets rid of all of the routes and if you are using gated or routed they will be relearned. I'm not sure what it does to the default route from netconf so use with care if you turn routed/gated off. You might want to use
route -f add default defaultrouterIP 1
instead. That will flush the route table and reinstall the default route at the same time. (defaultrouterIP is the IP of the default router)

It's best when looking at a large route table to do
netstat -rn
or even better
netstat -rn |grep 140.

The long delay is caused by netstat looking up the hostname for every route destination. The -n option tells it to just use the IP address so it will work faster. The other delay is talking to the terminal so if you can redirect the input to grep it will go faster since it only has to "print" the stuff you want to see.

route does not seem to have an easy way to kill of groups of routes at a time. I think you would need a script which did a quick netstat -rn | grep 140.*.*.* (hope that expression works - best to check it before you let it do anything automatically) then run route delete commands for each of the routes that shows up. This is something I suppose SED would be good for. Best to post another request for that sort of script. Maybe in hpux/general or sysadmin.

You may also want to look at your arp table with arp -a. Entries here are supposed to expire after a while but until they do they can cause problems. arp -d hostname will remove an entry.

Ron

[Uday_S_Ankolekar]

It may be due to IGRP turned on in Router ,You can turn off this in router if possible.

Or you can run a route -f followed by route add to deafult gateway in a script and run it periodically.

Just a thought..
-USA..

[Ron Kinner]

I guess I need to learn to type faster. When I started typing my previous post there was only the one reply.

If gated is off then I guess you are getting a lot of redirects. I would suspect that your default router is not the gateway to the internet but the gateway is on the same LAN so that everytime you go to a new site your default router says you can get there better by going to the gateway. This will create an entry in your routing table each time. Would be better to point at the gateway router since then you would only get redirects to local stuff which is finite in number and could be fixed with a few static routes. I'm not sure if you can tune how long these stay or if you can ignore the redirects on a 10.20. 11.0's ndd gives you a lot more options than 10.20's nettune. Do:
nettune -l
and see if there is anything about icmp redirects.


Ron

[Dermot Beirne]

The -n in the netstat was what made the biggest difference! Thanks for that! The long delay was that the table was trying to resolve every IP address to a hostname. Now I can list the table at great speed and find my offending routes quickly and fix them. As regards stopping the dynamic routing updates, I'll have to enquire further, try writing a script that removes dynamically learned routes periodically.
Thanks to you all for your assistance.

Dermot.

[Jim Keeble]

I noticed you are running 10.20, so the ndd tunable won't apply.

For now, add the network routes for each of the subnets that the re-directed host routes show up for. If they are the result of ICMP re-directs, there should be a "D" included in the FLAGS column of netstat -rn.

In the long run, the layout of your network could probably be simplified by replacing some of the backbone with switches an a single router with interfaces on each subnet. I know, all it takes is money.