topic Re: route apperas in routing table after system reboot in Operating System - OpenVMS

route apperas in routing table after system reboot

Martin Sedlak — Fri, 11 Sep 2009 08:29:26 GMT

Hi the route 10.0.0.0/8 comes to the routing table when OpenVMS starts up.
$tcpip sh route
DYNAMIC
Type Destination Gateway
AN 10.0.0.0/24 10.0.0.1
AN 10.0.0.0/8 10.37.150.1

This is strange, because before the system was rebooted, we made sure, the route is removed from routing tables:
$ tcpip route delete -net 10.0.0.0 10.37.150.1
$ tcpip set noroute 10.0.0.0 /gateway=10.37.150.1
$ tcpip set noroute 10.0.0.0 /gateway=10.37.150.1 /permanent

Dynamic routing protocols are not configured.

There are following interfaces configured:
$ tcpip sh int
Interface IP_Addr Network mask Receive Send MTU
IE0 10.0.1.1 255.255.255.0 0 1 1500
IE1 10.37.150.29 255.255.254.0 387 6 1500
IE5 192.168.80.211 255.255.254.0 9 1 1500
LE1 10.37.153.132 255.255.255.224 6 6 1500
LO0 127.0.0.1 255.0.0.0 42 42 4096
WE0 10.0.0.1 255.255.255.0 0 1 1500

As the route has UGS flags:
$ netstat -rn
Destination Gateway Flags Refs Use Interface
default 10.37.153.129 UGS 1 4 LE1
10.0.0/24 10.0.0.1 U 1 0 WE0
10/8 10.37.150.1 UGS 0 0 IE1

I assume that it must be created by an route command. When monitoring each phase of SYSTARTUP_VMS.COM, we found out that the route comes into the dynamic routing table (tcpip sh route) somewhere in the @SYS$STARTUP:TCPIP$STARTUP.COM phase.

Now I'm searching the TCPIP startup phase, however I'm not too much succesfull.

Has anyone faced similar issue? Have you got an idea/clue, where to have a look?

TCPIP> sh ver
HP TCP/IP Services for OpenVMS Alpha Version V5.4 - ECO 7
on a AlphaServer DS25 running OpenVMS V7.3-2

Thank you very much.

Re: route apperas in routing table after system reboot

marsh_1 — Fri, 11 Sep 2009 10:01:33 GMT

hi,

it will probably get that routing info back from that switch (10.37.150.1) when TCPIP has established contact with it, if you do a $tcpip show r/p it won't be there.

hth

Re: route apperas in routing table after system reboot

Jon Pinkley — Fri, 11 Sep 2009 12:05:56 GMT

Martin,

Dynamic routes can be the result of receiving an ICMP redirect from a router, in this case, I would assume the redirects are coming from the default router at 10.37.153.129

See this thread

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1096481

See response dated Feb 9, 2007 18:39:50 GMT

Jon

Re: route apperas in routing table after system reboot

Martin Sedlak — Mon, 14 Sep 2009 06:41:26 GMT

mark: thx. I'm getting the config already
John: thx. I'll follow your idea with ICMP redirect

I will get back here with the results soon :)

Re: route apperas in routing table after system reboot

Martin Sedlak — Wed, 14 Oct 2009 11:20:03 GMT

Hello, it took quite long time, but finally, the ICMP redirects were disabled. However, when the system was rebooted, the route came up again :)

It looks like the problem is not related to icmp redirect but it resides elsewher.

Unfortunately, the environment did not allowed us to dump the interfaces, therefore we do not know if to track the problem outside the VMS or inside.

Should you have any advice or comment, do not heshitate to reply here. Thank you.

Re: route apperas in routing table after system reboot

John Gillings — Wed, 14 Oct 2009 20:10:04 GMT

Martin,

>However, when the system was rebooted, the
>route came up again :)

I haven't seen a description of any actual problem these routes are causing, other than the apparent expectation that they "shouldn't be there".

Is the route really incorrect? Have you checked that it isn't doing you a favour by bypassing one or more routing hops?

Please post any error messages, or error symptoms that you believe are a result of the unexpected routes.

If everything is working correctly, the best advice I can give for the stated problem is "don't look".

That said...

A long time ago, on even older versions of OpenVMS and TCPIP services, and possibly aided and abetted by incompatible router and switch protocols, there were some serious problems that *incorrect* ICMP redirects could cause, the worst of which was a redirect of all addresses back to the node itself, effectively cuting the node off the network. I had a node that suffered this problem, so I wrote a procedure that ran every hour to delete any bad redirects from the routing table. SMOP a dozen or so lines in DCL. If the route offends you, pluck it out!

I doubt OpenVMS engineering would be terribly interested in this unless you could demonstrate the issue (along with a bona fide negative symptom) on more recent versions of OpenVMS and TCPIP.

Re: route apperas in routing table after system reboot

Martin Sedlak — Thu, 15 Oct 2009 06:28:53 GMT

Hi John, thank you for the reply. The 10/8 route is causing wrong routing. There are a few 10.37.x.x/24 segments connected to the OpenVMS system.

Unfortunately, I have not seen any error message pointing to this.

Re: route apperas in routing table after system reboot

Jon Pinkley — Thu, 15 Oct 2009 10:25:11 GMT

Martin,

First, after a closer read of your original problem statement, it appears that I may have sent you off on the wrong route.

Can you give us the output of the following?

$ ifconfig -a ! this gives more info than $ tcpip show interface

From your initial post:

------------------------------------------------
There are following interfaces configured:
$ tcpip sh int
Interface IP_Addr Network mask Receive Send MTU
IE0 10.0.1.1 255.255.255.0 0 1 1500
IE1 10.37.150.29 255.255.254.0 387 6 1500
IE5 192.168.80.211 255.255.254.0 9 1 1500
LE1 10.37.153.132 255.255.255.224 6 6 1500
LO0 127.0.0.1 255.0.0.0 42 42 4096
WE0 10.0.0.1 255.255.255.0 0 1 1500

As the route has UGS flags:
$ netstat -rn
Destination Gateway Flags Refs Use Interface
default 10.37.153.129 UGS 1 4 LE1
10.0.0/24 10.0.0.1 U 1 0 WE0
10/8 10.37.150.1 UGS 0 0 IE1
------------------------------------------------

I assume the routing table you were expecting and wanting is this:

$ netstat -rn
Destination Gateway Flags Refs Use Interface
default 10.37.153.129 UGS 1 4 LE1
10.0.0/24 10.0.0.1 U 1 0 WE0
10/23 10.37.150.1 UGS 0 0 IE1

The difference being the length of the network mask on the IE1 interface route.

It appears to me that TCPIP services is creating classfull routes for "connected" networks, when it should be creating CIDR (variable length mask based routes) based on the mask specified by the netmask specified on the interface.

In other words, the interface mask for IE1 is 255.255.254.0, which corresponds to /23, but the route being created by TCPIP services is a class A /8 (which is "correct" only in the pre-CIDR internet, where masks were based on the internet address prefix, into three classes, A /8, B /16, and C /24).

If you don't have GATED enabled (you said you don't have any dynamic routing enabled), it is still possible that it is receiving IRDP (ICMP Router Discovery Protocol, RFC 1256) advertisements and acting on them, although the TCPIP documentation doesn't suggest this. For more information about IRDP, Google for RFC 1256.

Your last post said: "There are a few 10.37.x.x/24 segments connected to the OpenVMS system." If "a few" means more than 10.37.150/24 and 10.37.151/24 (or 10.37.150/23), then you will need a router and the appropriate static route (for example, if all 10.37/17 addresses (10.37.0.0 - 10.37.127.255)should be reached via the IE1 interface, then you could use one static route to forward all 10.37/17 addresses to the next hop router (10.37.150.1?) and it will be responsible to forward to the correct network. Alternatively, you could put multiple specific /24 routes if you have sparse routing.

As to a work around: My first thing to try would be to add an explicit route for 10.37.x.x/m with net hop address of the correct router. And if the shorter (less specific 10/8) route is there, delete it. I am not sure if that will keep it from returning or not. You will need to try it.

Jon

Re: route apperas in routing table after system reboot

Jim_McKinney — Thu, 15 Oct 2009 10:27:36 GMT

This may not be convenient, but, if you continue to be unable to determine whether this is a learned route or one that results from your chosen config you could try unplugging your node from the network and booting - if it is immediately present in your routing table after the boot you'll know that is a direct result of your software config.

Re: route apperas in routing table after system reboot

The Brit — Thu, 15 Oct 2009 10:45:11 GMT

This is just a question, and probably has no bearing on the problem, I'm just curious).

Is interface LE1 a "logical" device (i.e. failover set), and if so, which physical devices make up the set?

Dave.

Re: route apperas in routing table after system reboot

Jon Pinkley — Thu, 15 Oct 2009 12:13:35 GMT

Martin,

Did you give us a partial list of routes?

$ netstat -rn
Destination Gateway Flags Refs Use Interface
default 10.37.153.129 UGS 1 4 LE1
10.0.0/24 10.0.0.1 U 1 0 WE0
10/8 10.37.150.1 UGS 0 0 IE1

I would have expected more routes, like

192.168.80/23 192.168.80.211 U x x IE5
192.168.80.211 192.168.80.211 UHL x x IE5

in other words, for each interface address, I would expect to see routes.

So can you pleas provide unedited output of the following commands?

$ ifconfig -a
$ netstat -rn

See attachment for some testing I did using secondary addresses on a test machine with tcpip configured on a single network interface.

Re: route apperas in routing table after system reboot

Martin Sedlak — Thu, 22 Oct 2009 12:59:36 GMT

Thanks to all of you that are posting here.

Jim: I'll try to folow your idea... We might also stop/start tcp/ip services throught console when running the tcpdump in paralel

Jon: There is almost latest patch level installed, therefore I suppose there should not be issue with 'pre-CIDR' masks.
AA51:SMSC> product sh hist
------------------------------------ ----------- ----------- --- -----------
PRODUCT KIT TYPE OPERATION VAL DATE
------------------------------------ ----------- ----------- --- -----------
DEC AXPVMS VMS732_UPDATE V16.0 Patch Install 08-JUL-2009
DEC AXPVMS VMS732_PCSI V5.0 Patch Install 08-JUL-2009
DEC AXPVMS TCPIP V5.4-15ECO7 Full LP Install 28-MAR-2008
DEC AXPVMS VMS732_PCSI V4.0 Patch Install 28-MAR-2008
DEC AXPVMS VMS732_SYS V15.0 Patch Install 28-MAR-2008
DEC AXPVMS VMS732_UPDATE V15.0 Patch Install 28-MAR-2008

There is just one IP address configured per interface. The routes in netstat -rn follows the interfaces. I do not see any problem here.
AA51:SMSC> ifconfig -a
IE0: flags=c43
*inet 10.0.1.1 netmask ffffff00 broadcast 10.0.1.255 ipmtu 1500

IE1: flags=c43
*inet 10.37.150.29 netmask fffffe00 broadcast 10.37.151.255 ipmtu 1500

IE5: flags=c43
*inet 192.168.80.211 netmask fffffe00 broadcast 192.168.81.255 ipmtu 1500

LE1: flags=c43
*inet 10.37.153.132 netmask ffffffe0 broadcast 192.168.22.255 ipmtu 1500

LO0: flags=100c89
inet 127.0.0.1 netmask ff000000 ipmtu 4096

TN0: flags=80

TN1: flags=80

WE0: flags=c43
*inet 10.0.0.1 netmask ffffff00 broadcast 10.0.0.255 ipmtu 1500

AA51:SMSC> netstat -rn
Routing tables
Destination Gateway Flags Refs Use Interface

Route Tree for Protocol Family 26:

Route Tree for Protocol Family 2:
default 10.37.153.129 UGS 86 165772956 LE1
10.0.0.1 10.0.0.1 UHL 0 0 WE0
10.0.1/24 10.0.1.1 U 1 0 IE0
10.0.1.1 10.0.1.1 UHL 0 0 IE0
10.3.56/21 10.37.150.1 UGS 0 0 IE1
10.3.64/21 10.37.150.1 UGS 0 0 IE1
10.37.150/23 10.37.150.29 U 1 1444 IE1
10.37.150.29 10.37.150.29 UHL 0 0 IE1
10.37.153.128/27 10.37.153.132 U 1 108 LE1
10.37.153.132 10.37.153.132 UHL 0 0 LE1
10.86.160/19 10.37.150.1 UGS 0 0 IE1
10.139.79/24 10.37.150.1 UGS 0 0 IE1
10.140.138/24 10.37.150.1 UGS 0 0 IE1
10.162.3/24 10.37.150.1 UGS 0 0 IE1
10.162.32/24 10.37.150.1 UGS 0 0 IE1
10.162.34/24 10.37.150.1 UGS 0 0 IE1
10.174.62/24 10.37.150.1 UGS 0 0 IE1
10.181.5/24 10.37.150.1 UGS 0 0 IE1
10.181.195/24 10.37.150.1 UGS 0 0 IE1
127.0.0.1 127.0.0.1 UHL 11 20951202 LO0
135.157/16 10.37.150.1 UGS 0 82104 IE1
135.209.64/18 10.37.150.1 UGS 0 427 IE1
135.214.139/24 10.37.150.1 UGS 0 0 IE1
141.204/16 10.37.150.1 UGS 0 0 IE1
155.174/16 10.37.150.1 UGS 0 124 IE1
172.27.1.20 10.37.150.1 UGHS 0 0 IE1
172.27.17.16 10.37.150.1 UGHS 0 0 IE1
192.168.80/23 192.168.80.211 U 1 10157223 IE5
192.168.80.211 192.168.80.211 UHL 0 124 IE5

Re: route apperas in routing table after system reboot

Jon Pinkley — Thu, 22 Oct 2009 21:51:30 GMT

Martin Sedlak Oct 22, 2009 13:59:36 GMT N/A: Question Author

--------------------------------------------------------------------------------
Thanks to all of you that are posting here.

Jim: I'll try to folow your idea... We might also stop/start tcp/ip services throught console when running the tcpdump in paralel

Jon: There is almost latest patch level installed, therefore I suppose there should not be issue with 'pre-CIDR' masks.
AA51:SMSC> product sh hist
------------------------------------ ----------- ----------- --- -----------
PRODUCT KIT TYPE OPERATION VAL DATE
------------------------------------ ----------- ----------- --- -----------
DEC AXPVMS VMS732_UPDATE V16.0 Patch Install 08-JUL-2009
DEC AXPVMS VMS732_PCSI V5.0 Patch Install 08-JUL-2009
DEC AXPVMS TCPIP V5.4-15ECO7 Full LP Install 28-MAR-2008
DEC AXPVMS VMS732_PCSI V4.0 Patch Install 28-MAR-2008
DEC AXPVMS VMS732_SYS V15.0 Patch Install 28-MAR-2008
DEC AXPVMS VMS732_UPDATE V15.0 Patch Install 28-MAR-2008

There is just one IP address configured per interface. The routes in netstat -rn follows the interfaces. I do not see any problem here.
AA51:SMSC> ifconfig -a
IE0: flags=c43
*inet 10.0.1.1 netmask ffffff00 broadcast 10.0.1.255 ipmtu 1500

IE1: flags=c43
*inet 10.37.150.29 netmask fffffe00 broadcast 10.37.151.255 ipmtu 1500

IE5: flags=c43
*inet 192.168.80.211 netmask fffffe00 broadcast 192.168.81.255 ipmtu 1500

LE1: flags=c43
*inet 10.37.153.132 netmask ffffffe0 broadcast 192.168.22.255 ipmtu 1500

LO0: flags=100c89
inet 127.0.0.1 netmask ff000000 ipmtu 4096

TN0: flags=80

TN1: flags=80

WE0: flags=c43
*inet 10.0.0.1 netmask ffffff00 broadcast 10.0.0.255 ipmtu 1500

AA51:SMSC> netstat -rn
Routing tables
Destination Gateway Flags Refs Use Interface

Route Tree for Protocol Family 26:

Route Tree for Protocol Family 2:
default 10.37.153.129 UGS 86 165772956 LE1
10.0.0.1 10.0.0.1 UHL 0 0 WE0
10.0.1/24 10.0.1.1 U 1 0 IE0
10.0.1.1 10.0.1.1 UHL 0 0 IE0
10.3.56/21 10.37.150.1 UGS 0 0 IE1
10.3.64/21 10.37.150.1 UGS 0 0 IE1
10.37.150/23 10.37.150.29 U 1 1444 IE1
10.37.150.29 10.37.150.29 UHL 0 0 IE1
10.37.153.128/27 10.37.153.132 U 1 108 LE1
10.37.153.132 10.37.153.132 UHL 0 0 LE1
10.86.160/19 10.37.150.1 UGS 0 0 IE1
10.139.79/24 10.37.150.1 UGS 0 0 IE1
10.140.138/24 10.37.150.1 UGS 0 0 IE1
10.162.3/24 10.37.150.1 UGS 0 0 IE1
10.162.32/24 10.37.150.1 UGS 0 0 IE1
10.162.34/24 10.37.150.1 UGS 0 0 IE1
10.174.62/24 10.37.150.1 UGS 0 0 IE1
10.181.5/24 10.37.150.1 UGS 0 0 IE1
10.181.195/24 10.37.150.1 UGS 0 0 IE1
127.0.0.1 127.0.0.1 UHL 11 20951202 LO0
135.157/16 10.37.150.1 UGS 0 82104 IE1
135.209.64/18 10.37.150.1 UGS 0 427 IE1
135.214.139/24 10.37.150.1 UGS 0 0 IE1
141.204/16 10.37.150.1 UGS 0 0 IE1
155.174/16 10.37.150.1 UGS 0 124 IE1
172.27.1.20 10.37.150.1 UGHS 0 0 IE1
172.27.17.16 10.37.150.1 UGHS 0 0 IE1
192.168.80/23 192.168.80.211 U 1 10157223 IE5
192.168.80.211 192.168.80.211 UHL 0 124 IE5

Martin,

It appears that tcpip is creating the correct routing entries for the IE1 interface itself.

10.37.150/23 10.37.150.29 U 1 1444 IE1
10.37.150.29 10.37.150.29 UHL 0 0 IE1

I don't see the "bad route" in the output you provided. Did you manually delete the 10.0.0.0/8 route?

If you can delete the route, and it does not reappear until you reboot, then it is almost certainly something that is being done in your startup sequence that is adding the routes.

What is creating all these static routes?

10.3.56/21 10.37.150.1 UGS 0 0 IE1
10.3.64/21 10.37.150.1 UGS 0 0 IE1
10.86.160/19 10.37.150.1 UGS 0 0 IE1
10.139.79/24 10.37.150.1 UGS 0 0 IE1
10.140.138/24 10.37.150.1 UGS 0 0 IE1
10.162.3/24 10.37.150.1 UGS 0 0 IE1
10.162.32/24 10.37.150.1 UGS 0 0 IE1
10.162.34/24 10.37.150.1 UGS 0 0 IE1
10.174.62/24 10.37.150.1 UGS 0 0 IE1
10.181.5/24 10.37.150.1 UGS 0 0 IE1
10.181.195/24 10.37.150.1 UGS 0 0 IE1
135.157/16 10.37.150.1 UGS 0 82104 IE1
135.209.64/18 10.37.150.1 UGS 0 427 IE1
135.214.139/24 10.37.150.1 UGS 0 0 IE1
141.204/16 10.37.150.1 UGS 0 0 IE1
155.174/16 10.37.150.1 UGS 0 124 IE1
172.27.1.20 10.37.150.1 UGHS 0 0 IE1
172.27.17.16 10.37.150.1 UGHS 0 0 IE1

I would guess that the 10/8 route is being defined at the same time as these others.

If these routes don't appear in output from

$ tcpip show route /perm

then something is adding them.

Have you looked at sys$manager:tcpip$systartup.com ?

Jon

Re: route apperas in routing table after system reboot

Jon Pinkley — Thu, 22 Oct 2009 21:58:33 GMT

Sorry, I didn't mean to include your whole comment in my reply.

Re: route apperas in routing table after system reboot

Martin Sedlak — Fri, 23 Oct 2009 06:43:35 GMT

Hi Jon, thank you for the response. The 'bad route' is not in the list of current routes, because it was removed manualy after openvms starup.

We can remove the route manualy, that is not a problem. However, when the system is restarted next time, it comes back again :)

You are right, the routes that do not come with interfaces are permanent routes ($ tcpip show route /perm) stored in the SYS$COMMON:[SYSEXE]TCPIP$ROUTE.DAT

Yes, I went throught all atarup scripts, searching something, that might add the route, however I was not succesfull. It looks like noone has added something to the starup scripts.

I'll try to follow idea with tcpdump while tcpip services are restarted. I'll keep you informed, however this will take some time as I'm bussy with another issues :/

Thx., M.

Re: route apperas in routing table after system reboot

Jon Pinkley — Fri, 23 Oct 2009 17:08:53 GMT

Martin,

Before trying tcpdump, I would try Jim McKinney's suggestion of shutting down, removing network connections, and rebooting. If the 10/8 route reappears, then it isn't something on the network that is adding the route. As you noted in your original post, the S flag is present (in USG), indicating a static route. I would not expect these to be the result of packets arriving on your interfaces.

At the same time, you may want to employ this "big gun" approach to determining what is used by your startup sequence.

Set SYSGEN parameter STARTUP_P2 to "VDC", which will enable verification and logging of the startup to sys$system:startup.log

Also, to catch some things that may not be logged, you can turn on file retention with a really short period so you can see what files are accessed during the boot sequence.

First verify that no files on your system disk have expiration dates in the future, and that file retention is not currently in effect. Then set the system disk to have a very short volume retention. Then reboot, and see what files have been accessed. This will give you a list of possible suspects.

First verify that volume retention isn't set

$ pipe show device/full sys$sysdevice: | search/nowin sys$pipe "ret. period"
%SEARCH-I-NOMATCHES, no strings matched

No strings matched means no volume retention set

Verify that no files have expiration dates set in the future. If you have DFU the command is:

$ define DFU$NOSMG T ! avoid SMG
$ now = f$time()
$ dfu search /exp=since="''now'"/sort sys$sysdevice /out=sys$scratch:files_expired_in_future.lis

If there are files with dates in the future, then these won't track access, since the expiration dates are never decreased by the retention mechanism.

If you don't have DFU directory can be used but it will take a much longer time and possibly list duplicates (due to alias SYS$COMMONS)

$ dir/expire/since="''now'"/date=(cre,mod,exp) sys$sysdevice:[000000...]

$ set volume/ret=(0,0-0:0:0.01) sys$sysdevice ! turn on file access recording in expiration date
$ pipe show device/full sys$sysdevice: | search/nowin sys$pipe "ret. period"
Min ret. period 0-00:00:00.00 Max ret. period 0-00:00:00.01

Then reboot.

Immediately after the system reboot is complete, find all the files that have been accessed since boot.

$ set volume/retention=0 ! turn off file expiration updating (or return it to previous value)
$ define DFU$NOSMG T ! avoid SMG
$ dfu search /exp=since=boot/sort sys$sysdevice /out=sys$scratch:files_accessed_during_boot.lis ! or use directory of you don't have

Now you will have a much better chance of finding what is adding the route, assuming that it is being done by something in the boot sequence.

Jon