What can be causing DNS timeouts from internal to external?

[Geoff Wild]

All our internal DNS (bind 9.2.3) servers forward to an ISP's DNS servers for External (internet) name resolution.

That said, even if I bypass them and go direct to root servers, we get the same behaviour.

On unix machines - there is a delay and sometimes no response.
On windows, we almost always get:

> www.disney.com
Server: myserver.myinternaldomain.net
Address: 192.168.162.6

DNS request timed out.
timeout was 2 seconds.
Non-authoritative answer:
Name: disney.com
Address: 199.181.132.250
Aliases: www.disney.com


Or

> labtec.com
Server: myserver.myinternaldomain.net
Address: 192.168.162.6

DNS request timed out.
timeout was 2 seconds.
DNS request timed out.
timeout was 2 seconds.
*** Request to myserver.myinternaldomain.net timed-out


Again, this is only happening for external resolution - internally the servers resolve almost instaneously with the correct answer...

I can't find any errors in log files on DNS servers...

Here's part of the named.conf file for a slave:

options {
directory "/etc/namedb";
forwarders {
207.181.101.4; 207.181.101.5;
};
forward only;

};

logging {

channel all_channel {
file "/var/log/named/named.log" versions 5 size 5M; // keep 5 versions max
5 MB in size
print-category yes;
print-severity yes;
print-time yes;
};
channel update_channel {
file "/var/log/named/update.log" versions 5 size 5M; // keep 5 versions ma
x 5 MB in size
print-category yes;
print-severity yes;
print-time yes;
};
channel query_channel {
file "/var/log/named/query.log" versions 5 size 5M; // keep 5 versions max
5 MB in size
print-category yes;
print-severity yes;
print-time yes;
};
//category queries { null; };
category queries { query_channel; };
category update { update_channel; };
category default { all_channel; };

};

zone "." in {
type hint;
file "named.cache";
};


Rgds...Geoff

[Sundar_7]

Geoff,

Are you able to directly query these two servers successfully ? 207.181.101.4; 207.181.101.5;

It will not hurt to have your network folks monitor the 53 UDP & TCP (mostly UDP) traffic from your internal DNS server to the external servers.

Sundar.

[Florian Heigl (new acc)]

Geoff,

usually I just see this when using a windows dns forwarder somewhere in the line; with that (obviously) out of the question it would seem like a network issue; I'd suggest asking bind itself, bind9 has the best debug output of all applications I ever ran. Restart the nameserver with a debuglevel above 4 and it will automatically dump a log somewhere (i think the toplevel of it's chroot directory) that contains *really* *useful* *data*.

Or you could try enabled query statistics, but I think that's not really helping much, as it will only verify a problem exists, but not show its cause.

[Geoff Wild]

Sundar, from internally, we still get timeouts - but if I'm on an external server - and query them - then DNS is fine

Rgds...Geoff

[Sundar_7]

Then, I would seriously look in to the firewall rules for the 53/UDP from the internal server to the external DNS servers.

[Geoff Wild]

But if it was a firewall issue - then wouldn't it fail all the time?

Rgds...Geoff

[Florian Heigl (new acc)]

i.e. nat table exhaustion on the firewall(s) could cause infrequent failures, though that would need some HEAVY load.
filesharing comes to mind there... hehe
Also routing congestion somewhere in your upstream, DNS being udp based can't recover from packet loss, except by retrying till a packet gets through.

[Patrick Wallek]

Hi Geoff,

Can you do a traceroute to those servers? How long does it take? Is it an extraordinarily long route to those servers?

I know it's a real long shot, but you never know.

[TwoProc]

Geoff, it's a long shot, b/c you've already probably got this configured - but I figured I'd send this on a "just in case" mission.
You've probably already got this - BUT...

Review /etc/resolv.conf and make sure that "domain" is defined. I seem to remember hitting this and it acting somewhat similar a long time ago...

[Geoff Wild]

Traceroute not available through firewall...

domain is set in resolv.conf

Thanks...Geoff

[Wouter Jagers]

My two cents (or less, maybe):

When traceroute is disallowed and the network admins have also covered traceroute -I, I usually tend to do my 'network probing' using hping2, which is an excellent tool for forging your own packets with relative ease.

Who knows it could help you finding out possible high latency between you and the remote dns sockets..

Cheers,
Wout

[Florian Heigl (new acc)]

Geoff, can You ask your firewall guys to stop breaking teh internet?
at least temporarily they should support your debugging. :)

for traceroute you need:
proto icmp icmp-type 0
proto icmp icmp-type 8
proto icmp icmp-type 11

how about the named debug mode?

florian

[Geoff Wild]

Debug doesn't seem to show much - the queries aren't there:

# cat named.run
14-Mar-2007 09:53:10.955 starting BIND 9.3.2-P1 -d4
14-Mar-2007 09:53:10.963 found 2 CPUs, using 2 worker threads
14-Mar-2007 09:53:10.972 loading configuration from '/etc/named.conf'
14-Mar-2007 09:53:10.986 set maximum stack size to -3: success
14-Mar-2007 09:53:10.986 set maximum data size to -3: success
14-Mar-2007 09:53:10.986 set maximum core size to -3: success
14-Mar-2007 09:53:10.986 set maximum open files to 65536: success
14-Mar-2007 09:53:10.987 listening on IPv4 interface lo0, 127.0.0.1#53
14-Mar-2007 09:53:10.988 clientmgr @282428: create
14-Mar-2007 09:53:10.988 clientmgr @282428: createclients
14-Mar-2007 09:53:10.989 clientmgr @282428: create new
14-Mar-2007 09:53:10.989 client @2a3ea8: create
14-Mar-2007 09:53:10.989 clientmgr @282428: create new
14-Mar-2007 09:53:10.989 client @2a5bc8: create
14-Mar-2007 09:53:10.989 clientmgr @282428: createclients
14-Mar-2007 09:53:10.989 clientmgr @282428: create new
14-Mar-2007 09:53:10.989 client @2a7a48: create
14-Mar-2007 09:53:10.989 listening on IPv4 interface bge0, 10.8.60.45#53
14-Mar-2007 09:53:10.989 clientmgr @2a9810: create
14-Mar-2007 09:53:10.989 clientmgr @2a9810: createclients
14-Mar-2007 09:53:10.989 clientmgr @2a9810: create new
14-Mar-2007 09:53:10.990 client @2a9c68: create
14-Mar-2007 09:53:10.990 clientmgr @2a9810: create new
14-Mar-2007 09:53:10.990 client @2ab988: create
14-Mar-2007 09:53:10.990 clientmgr @2a9810: createclients
14-Mar-2007 09:53:10.990 clientmgr @2a9810: create new
14-Mar-2007 09:53:10.990 client @2ad808: create
14-Mar-2007 09:53:10.990 listening on IPv4 interface bge0:1, 10.8.60.44#53
14-Mar-2007 09:53:10.990 clientmgr @2af5d8: create
14-Mar-2007 09:53:10.990 clientmgr @2af5d8: createclients
14-Mar-2007 09:53:10.990 clientmgr @2af5d8: create new
14-Mar-2007 09:53:10.990 client @2afa28: create
14-Mar-2007 09:53:10.990 clientmgr @2af5d8: create new
14-Mar-2007 09:53:10.990 client @2b1748: create
14-Mar-2007 09:53:10.990 clientmgr @2af5d8: createclients
14-Mar-2007 09:53:10.990 clientmgr @2af5d8: create new
14-Mar-2007 09:53:10.991 client @2b35c8: create
14-Mar-2007 09:53:10.991 listening on IPv4 interface bge1, 10.8.60.46#53
14-Mar-2007 09:53:10.991 clientmgr @2b5390: create
14-Mar-2007 09:53:10.991 clientmgr @2b5390: createclients
14-Mar-2007 09:53:10.991 clientmgr @2b5390: create new
14-Mar-2007 09:53:10.991 client @2b57e0: create
14-Mar-2007 09:53:10.991 clientmgr @2b5390: create new
14-Mar-2007 09:53:10.991 client @2b7500: create
14-Mar-2007 09:53:10.991 clientmgr @2b5390: createclients
14-Mar-2007 09:53:10.991 clientmgr @2b5390: create new
14-Mar-2007 09:53:10.991 client @2b9380: create
14-Mar-2007 09:53:10.998 res 2c4d18: create
14-Mar-2007 09:53:10.999 dns_requestmgr_create
14-Mar-2007 09:53:10.999 dns_requestmgr_create: 2dc490
14-Mar-2007 09:53:10.999 dns_requestmgr_whenshutdown
14-Mar-2007 09:53:11.000 res 2e17b8: create
14-Mar-2007 09:53:11.000 dns_requestmgr_create
14-Mar-2007 09:53:11.000 dns_requestmgr_create: 2f9700
14-Mar-2007 09:53:11.000 dns_requestmgr_whenshutdown
14-Mar-2007 09:53:11.001 command channel listening on 127.0.0.1#953


Rgds...Geoff

[Florian Heigl (new acc)]

In that case please incrementally raise the debug level till the queries show up.
No bad things will happen.

[Geoff Wild]

This is looking like more of a Windows issue then Unix now - as we don't appear to have this issue at all Unix to Unix - only from Windows to Unix.

What happened to Windows during the time change? Seems to be a coincidence - or is it?

Rgds...Geoff

[Steven E. Protter]

Shalom Geoff from Jerusalem(finally),

See that there is no microsoft DNS servers between you and the objective. Microsoft's Windows 2000 server implementation does not follows standards and causes this kind of delay in getting answers.

Consider implementing limitations as to what machines are permitted to access the DNS servers. There could issues making these systems too busy.

See if the logging customization is causing the problem by temporarily cutting if off.

See that the network infrastructure and firewalls are not interfering with the timely tranmission.

SEP

[Florian Heigl (new acc)]

if you can identify windows dns servers (most probably AD DCs) in the chain, this would be almost "typical" behaviour.

it might help to verify if the DNS Client service is enabled on those boxes, it's similar to nscd on some unix platforms and just as prone to generate annoying errors.

for testing:

once a query seems to be failing, ask the
windows admin to do an
ipconfig /flushdns
(restarting the DNS Client service doesn't do that)
and retry the query.

The result might be interesting.

Microsoft also has a white paper on their DNS implementation that addresses most of the issues one sees daily. (it's by the people that coded the dns, they were aware of it's flaws, but noone cared)

[Geoff Wild]

Well, here I am, back from vaction, hoping this would have "gone away", but alas - no.

Seems to fail on first try.

What I did was, take my workstation, not use forwarders at all, and look what happens:

C:\dig>nslookup
Default Server: myserver.myinternaldomain.net
Address: 192.168.162.6

> server sha1
Default Server: sha1.myinternaldomain.net
Address: 192.168.162.125

> www.keyboards.com
Server: sha1.pcacorp.net
Address: 192.168.162.125

DNS request timed out.
timeout was 2 seconds.
*** Request to sha1.myinternaldomain.net timed-out
> www.keyboards.com
Server: sha1.myinternaldomain.net
Address: 192.168.162.125

Non-authoritative answer:
Name: www.keyboards.com
Address: 216.57.210.200

Argh...

Rgds...Geoff