topic Re: network routing problems in Operating System - HP-UX

network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 10:19:02 GMT

I have 2 HP-UX 11.11 servers that as far as I can tell have similar configuration... Both were working fine (and no config changes have been made in over a month) until Friday night when one started having issues sending email... Saturday night about half the computers on our network could no longer telnet to this server... I talked to our network person and she was able to help me figure out that the subnet mask was wrong (even though it had been working fine for over a month like that) so I fixed that and then the rest of the computers on the network were able to telnet to this server. Mail still wasn't working though and now I have one user that cannot telnet to this system again...

I posted over in the sys admin section about the sendmail problem and someone had me try traceroute from both systems which gave very different results... I am now thinking this is a network config problem rather than a mail config problem so I am hoping someone over here might have some suggestions... Here are the results of a traceroute to the address sendmail uses:

working server:
traceroute healthplus.com.s8a1.psmtp.com
traceroute to healthplus.com.s8a1.psmtp.com (64.18.7.10), 30 hops max, 40 byte p
ackets
1 ASTARO.flintdns (126.1.3.229) 0.293 ms 0.208 ms 0.150 ms
2 10.0.0.1 (10.0.0.1) 0.915 ms * *
3 209-254-57-73.ip.mcleodusa.net (209.254.57.73) 6.905 ms * 8.911 ms
4 * * FSHRINFCH02JP01-SO0-0-0-0.mcleodusa.net (64.198.100.37) 24.427 ms
5 * STLSMOGZH00JC01-SO0-2-0-0.mcleodusa.net (64.198.101.26) 35.455 ms *
6 * * KSCAMO54H00JP01-SO2-2-0-0.mcleodusa.net (64.198.100.170) 33.927 ms
7 * DNVTCOUZH00JC01-SO0-3-0-0.mcleodusa.net (64.198.101.90) 48.157 ms *
8 SNJUCACLH25JC01-SO0-1-1-0.mcleodusa.net (64.198.100.78) 75.986 ms * 76.061
ms
9 * cr1-eqix-peer.sje007.internap.net (206.223.116.134) 76.379 ms 76.454 ms
10 * core4.sje.inappnet-28.cr1.sje007.internap.net (66.79.148.130) 70.111 ms
69.419 ms
11 * border1.pc1-0-bbnet1.sje.pnap.net (66.151.144.4) 69.952 ms *
12 * * *
13 * * *
14 * * *
15 * * *
(I stopped it at this point)

other server:
traceroute healthplus.com.s8a1.psmtp.c>
traceroute to healthplus.com.s8a1.psmtp.com (64.18.7.10), 30 hops max, 40 byte p
ackets
1 DEVUX.flintdns (126.1.3.17) 0.159 ms !N 0.044 ms !N 0.034 ms !N

The first place the working server heads is our firewall server, the non-working server just seems to be looking at itself... How can I get the non-working server to head to the firewall server? netstat -rn lists the firewall server as default on both servers...

Re: network routing problems

Steven E. Protter — Tue, 23 Oct 2007 10:45:09 GMT

Shalom,

I'd suggest making the default gateway on the non-working server the same as the working one.

/etc/rc.config.d/netconf

It should if I understand the question point to the firewall.

SEP

Re: network routing problems

Wouter Jagers — Tue, 23 Oct 2007 10:45:57 GMT

See http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1171222

It's a good idea to keep this problem in your initial post: the more background info available, the better people will be able to help.

Cheers,
Wout

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 10:51:35 GMT

default gateway is already the same on both servers...

Re: network routing problems

A. Clay Stephenson — Tue, 23 Oct 2007 10:53:31 GMT

The most common configuration is that the default router is responsible for routing to the firewall. Since netstat -rn only lists the default router on the "bad" box then that suggests a few things to try --- and I am assuming that you are able to actually ping the default router from this bad box.
1) Is the default router on the "bad" box the same as for the "good" boxes? 2) Is the ROUTE_COUNT[n] value specified in /etc/rc.config.d/netconf for the default route set at 1?

It would help if you posted the output of netstat -rn on the "bad" box.

By the way, the subnet problem you listed earlier that still "works" is not all that uncommon. In fact, it's one of the things I do to torment admins in training. There is nothing like a subnet mask off by 1 bit to drive a jr. admin (or a sr. one that doesn't bother to verify the subnet mask as a matter of course) nuts. Some hosts work fine and others don't and they don't usually see the pattern until the subnet mask and the IP addresses are expressed in binary and then the mental light bulbs illuminate.

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 10:56:26 GMT

Working server:
netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 4136
192.168.0.1 192.168.0.1 UH 0 lan1 4136
126.1.3.225 126.1.3.225 UH 0 lan0 4136
192.168.0.0 192.168.0.1 U 2 lan1 1500
126.1.0.0 126.1.3.225 U 2 lan0 1500
127.0.0.0 127.0.0.1 U 0 lo0 0
default 126.1.3.229 UG 0 lan0 0

non-working server:
netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 4136
126.1.3.17 126.1.3.17 UH 0 lan1 4136
126.1.0.0 126.1.3.17 U 2 lan1 1500
127.0.0.0 127.0.0.1 U 0 lo0 0
default 126.1.3.229 UG 0 lan1 0

Re: network routing problems

Sandman! — Tue, 23 Oct 2007 12:36:12 GMT

Does the non-working server have anything to do with DNS at all? Owing to its hostname being >DEVUX.flintdns<. It is possible that the DNS record for this host got corrupted and created a circular link to itself (may explain why it keeps looking at itself and never moves on to the next hop). Your best bet would be to ask the DNS admin at your site if any changes were made to the DNS system.

~hope it helps

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 12:41:55 GMT

The .flintdns just gets added to the end of everything I think, this server doesn't have anything to do with DNS...

Re: network routing problems

Sandman! — Tue, 23 Oct 2007 12:55:07 GMT

But I have not seen a traceroute where the first hop was the actually the local interface. And that suggests that it could be a DNS issue. You might want to check with your site DNS admin.

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 12:57:16 GMT

Is there anything in particular they should be looking for? No other systems on the network are having any issues so she keeps telling me it a setting on my server...

Re: network routing problems

Sandman! — Tue, 23 Oct 2007 13:13:50 GMT

Show them the output of traceroute on the non-working server. The very first hop is the IP of a local interface (lan1)...is it not? And that is incorrect. The first hop should be to a remote host. Note the first hop of the working server is ASTARO.flintdns which is a remote host.

traceroute to healthplus.com.s8a1.psmtp.com (64.18.7.10), 30 hops max, 40 byte packets
>> 1 DEVUX.flintdns (126.1.3.17) 0.159 ms !N 0.044 ms !N 0.034 ms !N <<

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 13:24:11 GMT

I see what the problem is, I just don't know what is causing it... If I show the network admin the traceroute results her response is going to be "well just configure server B like server A"... From her standpoint the 500 PC's and 50 other servers on the network are all working just fine including the other HP-UX server so it must be a problem with this server, not the network...

I need help trying to figure out what the difference is between the two HP-UX boxes that is causing one to work and the other not to work..

Re: network routing problems

Sandman! — Tue, 23 Oct 2007 13:40:04 GMT

Try pinging ASTARO.flintdns from the non-working server and post the results here. Also post the output of the arp cache on both the good and bad servers.

# ping ASTARO
# arp -an

Re: network routing problems

Patrick Wallek — Tue, 23 Oct 2007 13:56:14 GMT

You posted:

non-working server:
netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 4136
126.1.3.17 126.1.3.17 UH 0 lan1 4136
126.1.0.0 126.1.3.17 U 2 lan1 1500
127.0.0.0 127.0.0.1 U 0 lo0 0
default 126.1.3.229 UG 0 lan1 0

The one line that I see that looks strange is:
126.1.0.0 126.1.3.17 U 2 lan1 1500

That is saying that anything going to the 126.1.*.* subnet goes through 126.1.3.17, which is the local host.

I think you need to remove that route. If you look on your working machine, the thing that resembles that line the closest is:

126.1.0.0 126.1.3.225 U 2 lan0 1500

I think this probably stems from your incorrect subnet mask issue.

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 14:05:06 GMT

The ASTARO (our firewall) server does not allow you to ping it so pinging fails from both servers...

If you look at the netstat -rn for both servers and compare the entries for lan0 on the working server and lan1 on the non-working server they are both set up the same:

netstat -rn
Routing tables
Destination Gateway Flags Refs Interface Pmtu
127.0.0.1 127.0.0.1 UH 0 lo0 4136
126.1.3. 126.1.3. UH 0 lan# 4136
126.1.0.0 126.1.3. U 2 lan# 1500
127.0.0.0 127.0.0.1 U 0 lo0 0
default 126.1.3.229 UG 0 lan# 0

Re: network routing problems

Sandman! — Tue, 23 Oct 2007 14:57:16 GMT

What about the arp cache?

# arp -an

post above output here

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 15:12:41 GMT

There are over a hundred results for each server, is there a particular address you are looking for? They both show the same results for the firewall server...

Re: network routing problems

Sandman! — Tue, 23 Oct 2007 15:57:22 GMT

Yes that's all I was interested in the MAC address of the ASTARO. Anyway try pinging the good server from the bad one and vice-versa and post the results here.

Re: network routing problems

Jenni Wolgast — Tue, 23 Oct 2007 16:00:07 GMT

The servers can ping each other just fine but they are both in each other's host files...

Re: network routing problems

Tim Nelson — Tue, 23 Oct 2007 16:17:29 GMT

Just a note as you mentioned that you cannot ping the router/firewall from this bad host.

I ran into this in a DMZ where the firewall was managed by Intel folk and they turned off ping allowance on the local net to my server.

The default route requires that you have ping access to the default router unless you specifically disable it with an ndd command and make it part of bootup.

Remove the default route and re-add it. If it works for a couple seconds and then you loose access again and traceroutes die with N! then this might be the issue.