new bind 9.9 and root NS take 2

[Doug O'Leary]

Hi;

A short summary of the previous post: I have a client who's migrating from two old DNS physical servers to two new virtual ones running bind 9.9.  I did their migration and, long story short, we're having problems getting the two new systems to talk to the root name servers.  My two initial theories (new NS has to be registered to talk to root NS and issue w/DNSSEC) both proved to be incorrect which leaves something on the network.

The core problem is that we cannot reach the root name servers via udp.  We *can*, however, reach google's name servers via udp.    We can also reach the root name servers via tcp...

# dig +novc @f.root-servers.net

; <<>> DiG 9.9.1-P1-RedHat-9.9.1-2.P1.fc17 <<>> +novc @f.root-servers.net
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

# dig +noanswer +noquestion +novc @8.8.8.8

; <<>> DiG 9.9.1-P1-RedHat-9.9.1-2.P1.fc17 <<>> +noanswer +noquestion +novc @8.8.8.8
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11665
;; flags: qr rd ra; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; Query time: 13 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Thu Aug 2 08:52:09 2012
;; MSG SIZE rcvd: 239

and 

# dig +noanswer +noadditional +noquestion +vc @f.root-servers.net

; <<>> DiG 9.9.1-P1-RedHat-9.9.1-2.P1.fc17 <<>> +noanswer +noadditional +noquestion +vc @f.root-servers.net
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 60360
;; flags: qr aa rd; QUERY: 1, ANSWER: 13, AUTHORITY: 0, ADDITIONAL: 23
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; Query time: 77 msec
;; SERVER: 192.5.5.241#53(192.5.5.241)
;; WHEN: Thu Aug 2 08:55:19 2012
;; MSG SIZE rcvd: 699

So, short version: the new dns systems can send outbound udp packets; but, something is blocking those packets going to the root name servers.  

Has anyone seen anything like this and/or know what might be causing it?  Failing that, does anyone know of a way to force recursions to use tcp vs udp?

This one's just plain weird... appreciate any hints/tips/suggestions.

Doug O'Leary