ping fails, nslookup works (udp port unreachable)

[ES UNIX Team]

All,
I am running HP-UX 11i. Ping, traceroute and other applications fail occasionally with "unknown host". nslookup always works and if you ping a host after nslookup is run against the host it will work for at least a few seconds. resolv.conf and nsswitch.conf are correct. I have a lot of experience with DNS and don't see a problem there. resolv.conf lists the domain and the two nameservers, nsswitch.conf lists files then DNS for host lookups.

I was able to determine the following behaviour on failures with tcpdump.

The local server opens a random port > 1024 to the DNS server port 53. The DNS server responds to the randomly opened port. The local server dumps the response (sends another message to the DNS server saying "udp port xxxxx unreachable for dns.server.com".

I have several 11i servers that work and several that don't. The ones that don't all display the same behavior. Patch levels are the same on all (working and non-working). chatr reports that all utilities (ping, nslookup, etc.) are using libc.2 which is what seems to be correct.

Any thoughts?

[Geoff Wild]

Have you checked your network statistics for errors?

netstat -i

Do you have a network group with a scope? can they monitor the network traffic at the same time as when you do you pings?

Does ping fail completely - or do you just miss some packets? That is when it fails, is it always 100% packet loss?

I've seen things like this happen when your lan cards are set to auto-negotiate - better to hard code the speed/duplex and the switch ports.

Rgds...Geoff

[doug mielke]

Since it looks like a failure to the dns server I'll assume that pings to hosts listed in /etc/hosts never fail. If so, I'd try to look at the network gear in between the dns and local server for errors, like the speed/duplex mis match, and for blocked broadcast/multicast settings on that gear.

If the hosts or dns are on the same segment as host you're pinging from, there should be an arp -a entry, then you can confirm mac address checking for an ip confilct.

A weird one from years ago for me was someone making a static route entry on another system with low metric that told all systems that " the shourtest route to any system is through me"
Condition was systems that the offending 'router' knew about always worked, and other systems would connect only if that 'router' was busy.

[ES UNIX Team]

Hey guys. Thanks for the responses, but none of these relate to our problem. Any more ideas? Ping is always 100% failure and the entries in the hosts file always work. It doesn't seem to be a problem with the network gear though because tcpdump shows that the response from the DNS server is received, but our server seems to discard it (reports that udp port xxxxx is not open for server dns.server.com). Its like our server closes the port before it gets a response.

[Geoff Wild]

On a server where ping fails - does it also fail to the ip address?

Another tool you can use:

linkloop

linkloop -i 1 0x00306E21A7BF
Link connectivity to LAN station: 0x00306E21A7BF
-- OK


-i = interface to go out on

Rgds...Geoff

[Marvin Strong]

The last time, I had a problem similar to this, it was a routing issue.

is the info from netstat -rvn correct?

[ES UNIX Team]

Pings to the IP address work. linkloop works. There are no connectivity issues. I am getting the DNS response from the DNS server. Here is the relevant tcpdump output to further explain the issue:

localserver.company.org.50117 > dnsserver.company.org.domain: [udp sum ok] 52272+ A? remoteserver. (25) (DF) (ttl 64, id 59686, len 53)

dnsserver.company.org.domain > localserver.company.org.50117: 52272 NXDomain q: A? remoteserver. 0/1/0 ns: . (100) (DF) (ttl 62, id 26512, len 128)

localserver.company.org > dnsserver.company.org: icmp: localserver.company.org udp port 50117 unreachable for dnsserver.company.org > localserver.company.org [|udp] (DF) (ttl 62, id 26512, len 128) (DF) (ttl 255, id 59688, len 112)

As you can see the DNS server responds and the response makes it to our lan card. All connectivity is good. When it is working the first two packets are exactly the same as above and the third packet doesn't exist. When it fails we see the third packet. It seems like our server is closing the udp port that it expects to get the response on.

All applications seem to cause this behavior except nslookup so since nslookup works that further indicates that connectivity should be OK.

More thoughts?

[Mel Burslan]

any firewalls in between ?
under today's extremely dire situations, most firewall admins block UDP connections to prevent a DOS attack. They may let a few packets pass to test connectivity and after the number of packets reach a certain threshold they may start blocking them with the assumption of the connectivity between the hosts has been verified and passage of UDP packets is no longer necessary.
especially the interfaces you are pinging are facing the internet and you are not the firewall admin, this is a plausible scenario.

Just a thought.

[ES UNIX Team]

There are no firewalls between the two nodes. Since the packet is coming back to the local server there shouldn't be any issues with connectivity. The main point here is that the response from the DNS server does in fact make it back to our server and our server discards it. There is no firewall/IDS software running on our server. Sometimes this will actually work, but some times (say 20%) it fails with "udp port xxxx unreachable for dnsserver.company.org". It just seems like our server is closing the port before it receives the response.

[Geoff Wild]

Still sounds like a resolver issue of sorts...

Try nsquery

nsquery hosts servernameinhostsfile

nsquery hosts servernamenotinhostsfile

post output please.


Rgds...Geoff

[ES UNIX Team]

Here is nsquery output:
nsquery hosts matrix
using "files dns" for the hosts policy
searching /etc/hosts for matrix
hostname: matrix
address: x.x.x.x
switch configuration: terminate search

nsquery hosts liberty
using "files dns" for the hosts policy
searching /etc/hosts for liberty
liberty was NOTFOUND
switch configuration: allows fallback
searching dns for liberty
hostname: liberty.company.org
address: x.x.x.x
switch configuration: terminates search

This always works fine. That is why I don't think there is a connectivity/resolver issue. nslookup/nsquery always work. But ping/telnet/etc. do not. And when the do fail we can clearly see that our server is discarding the DNS response packet. It closes the port before the DNS response to ping/telnet/etc. but not for ns commands.

What would make ping fail even though nslookup always works?

[Geoff Wild]

Difference between ping and nslookup - nslookup uses it own resolver code, to accomplish direct access to some features.

From the results of nsquery - I'd say you are bang on - no DNS issues.

What about the possibility that another device on your network has the same ip as the server you are pinging from?

What about arp cache?

I've had issues pinging printers when my package fails over - sometimes the printers don't update the arp cache and don't know the way to the new mac address that the ip is now bound to - usualy requires a reboot of the printer...

Rgds...Geoff

[ES UNIX Team]

Seems like ping would work if I were able to resolve the IP address. Even when ping fails with "unknown host" it works to the IP address. Tcpdump clearly shows one difference between the "working" and "non-working" situation every time. Shouldn't my focus be on that? When it doesn't work tcpdump shows that our server seems to be refusing the DNS server's response to our query. The response got back to our NIC, but the next thing out of our NIC is a message back to the DNS server saying the port (the random port our server opened to talk to the DNS server) is not open. Shouldn't the focus be on why this port is closing? If it didn't happen so fast I'd loop a netstat command to check ports but its too fast to pick up anything. I'm pretty sure tcpdump is indicating that the port is being closed by our server.

[Ashwani Kashyap]

Try this patch , PHNE_29887

[KapilRaj]

Does reverse-lookup work for your host from where u r trying the ping ?. If not, set it up for your host snd then try ...

Kaps

[Mobeen_1]

Have any IP addresses changed recently? Also i would think there would be no harm if you flush the cache and see if there is any change in behaviour.

regards
Mobeen

[ES UNIX Team]

Thanks guys for all your responses. As it turned out we needed to bump up the DNS timeout value. We didn't like the DNS timeout values (too long) before in 11.0 so we set RES_RETRY and RES_RETRANS to 1 and 5 respectively. On 11.0 it never dips below a certain value (I believe 15 seconds) regardless of what you set these to. But on 11i it does. We bumped them to 2 and 10 respectively and things are better.

So it really was that our server was closing the UDP port before it got the DNS server's response. It had nothing to do with the DNS server or the network equipment.

[Luqman Achmat]

Hi there

I have the same problem on my linux machine. I haven't found a solution yet, but when I stop the firewall service (ipchains), all works okay. When I start up my firewall services, tcpdump shows the outgoing udp packet as well as the incoming udp packet. However, on the 3rd response, I get a udp port xxxx unreachable.

Any suggestions?

Luq A