Weird DNS resolution pattern

[David P Lavoie]

Hi,

I've got a weird issue and I wanted to poke the brain of experts here... My organization is migrating DNS from Windows 2003 server to Windows 2008 R2. When this was completed, I changed the /etc/resolv.conf file to point to the new servers and I began to see a strange behaviour on the HP UX servers only. Windows servers are not affected by this change.

When I execute a traceroute to the DNS server using short hostname, sometimes it takes up until 5 seconds to resolve the IP address. When it arrives at destinestion, 1 probe out 2 is timing out. Here's an example:

traceroute -q 6 devpwdc01                                                                                                        
traceroute to devpwdc01 (10.3.99.10), 30 hops max, 40 byte packets
 1  10.3.15.250 (10.3.15.250)  0.108 ms  0.081 ms  0.082 ms  0.080 ms  0.083 ms  0.095 ms
 2  10.3.77.250 (10.3.77.250)  0.406 ms  0.376 ms  0.409 ms  0.395 ms  0.401 ms  0.395 ms
 3  10.3.77.251 (10.3.77.251)  5.097 ms  0.628 ms  0.639 ms  0.645 ms  1.167 ms  0.618 ms
 4  devpwdc01.pptc.gc.ca (10.3.99.10)  0.270 ms *  0.297 ms *  0.291 ms *

I don't have the same issue when I go back to the Windows 2003 DNS server. Linux servers does not have the same slowdown on name resolution but when running traceroute only the 1st probe gets through, all the others are timing out. Slow name resolution problem is not consistent either. Sometime it responds pretty fast, somethime it does not. dig, nslookup and host commands respond normally.

HP UX servers are not using NIS or LDAP, user accounts are local. /etc/hosts file contains only the local hostname. HP UX servers are running version 11.31 and NICs are configured in HPLM. Windows 2008 servers are using HP's NIC teaming equivalent for windows. IPfilter is not running on HP UX and the firewall appliance is already allowing DNS communication between the servers.

/etc/resolv.conf

cat /etc/resolv.conf
domain pptc.gc.ca
search pptc.gc.ca
nameserver 10.3.99.10
#nameserver 10.3.98.10
nameserver 10.3.55.10

/etc/nsswitch.conf

passwd:       files
group:        files
hosts:        files dns
networks:     files
protocols:    files
rpc:          files
publickey:    files
netgroup:     files
automount:    files
aliases:      files
services:     files

Let me know what you think or if you need more information (which you probably will be).

[Ken Grabowski]

What does your routing tables look like?  I've seen this most often with multipe routing paths, one good and one bad.

[David P Lavoie]

Hi,

Thank you for your response. Routing table is pretty basic.

 netstat -rn
Routing tables
Destination           Gateway            Flags Refs Interface  Pmtu
127.0.0.1             127.0.0.1          UH    0    lo0       32808
10.3.8.67             10.3.8.67          UH    0    lan900    32808
10.3.8.0              10.3.8.67          U     2    lan900     1500
127.0.0.0             127.0.0.1          U     0    lo0       32808
default               10.3.15.250        UG    0    lan900     1500

[Ken Grabowski]

It does look pretty basic. Everything else appears to be properly configured and since your problem doesn’t happen on the old DNS server, but does on the new, I think your problem is down stream in the DNS servers or network routers. Possibly the new DNS servers are not working with the same information as the original DNS servers.

[David P Lavoie]

Thank you Ken for your answer. Yeah, that was my guess too. I'm trying to get a Network engineer involved on this problem.