Telnet and reverse lookup

[Luis Toro]

We keep experiencing a problem every time we ignite several HPUX servers at our DR site (its a contracted site, ie., Sunguard). Telnet hangs for a minute or so upon initial connection. Below are the details:

resolv.conf point to a WINS server.
nsswitch has "files [ notfound continue ] dns".
neither host files have the other server's ip/hostname information.

Telneting from ServerA to ServerB is fine.
Telneting from ServerB to ServerA yields a long delay.

nslookups on both servers (forward and reverse) yield an instant response, so there is no reason to point to the WINS/DNS setup as being the cause (so I'm being told). HOWEVER, the problem goes away if I update the local hosts file on ServerB with an entry for ServerA.

At this point, everyone is pointing fingers (though it seems each group has a vaild reason to point elsewhere).

[Steven Schweda]

> nslookups on both servers (forward and
> reverse) yield an instant response [...]

> HOWEVER, the problem goes away if I update
> the local hosts file on ServerB with an
> entry for ServerA.

The HOWEVER clause seems decisive. That
which fixes the problem, fixes the problem.

So, how to explain the DNS seeming to work?
Any chance that Telnet is doing a reverse
look-up on some address different from what
you're feeding into nslookup? I don't have a
good suggestion for how to verify this easily.

[Luis Toro]

Thanks Steven. Unfortunately, the DR test is over, and there is no way to recreate the environment, until the next test. I'm hoping for someone to reply with details on how telnet works with DNS. We have some screen prints of a "sniff", but I can't really decipher it.

[Sandman!]

What about a traceroute from server B to server A, how long does that take ineither direction? And what about static routing in both directions? Is that configured correctly?

[Bill Hassell · Accepted solution]

The design of telnet includes a minimal 'security' check to see if the incoming IP address trying to connect is known to a known authority, typically a DNS, NIS or LDAP server. It does this by doing a reverse lookup, that is, look up the IP address to verify the name. A properly configured DNS server will have both records (forward and reverse) as well as MX (mail exchanger) records. This is very seldom done for Windows based DNS servers because Windows doesn't need that stuff (it accepts everything without verification). So there is a timeout because the server doesn't do anything for the request and HP-UX sits waiting for a response. The timout for the first server is about 20-30 seconds. Repeat for the second server in resolv.conf and again for the 3rd server (if present). So a delay of about 30, 60 or 90 seconds is normal for an unknown or IP that cannot be verified.

When you put the IP address in /etc/hosts, the resolver sees this first (based on your nsswitch.conf) and immediately connects. If you can't get the windows servers fixed, just put all your production IP addresses in /etc/hosts (thousands of addresses are just fine if needed).

[skt_skt]

I would also suggest to make use of /etc/hosts rather than /etc/resolv.conf.

Also try the following after the name server entries on /etc/resolv.conf

retrans 1000
retry 1

[Luis Toro]

Thanks for your replies. Bill's reply was very helpful, but the DNS group is still hanging their hat on the lack of a delay on the nslookups (forward and reverse). Here's another tidbit of info:
The DNS server that we're instructed to configure, resolves names of the type:
servername.winnet.company.com (windows servers, which get assigned addresses via DHCP). The HPUX servers are all HPservername.company.com. So the winnet server "passes up" to some other server those hostname resolutions (resulting in a "non-authoritative answer").

[Steven Schweda]

> The DNS server that we're instructed to
> configure, resolves names [...]

Of course, the real question is who resolves
the _numbers_ when the Telnet server tries to
do the _reverse_ look-up on the connecting
client. Adding an entry to /etc/hosts covers
both directions, but on a DNS server, the two
functions are distinct. (It's easy to
configure a DNS server to do only, say,
name-to-number look-ups, leaving you on your
own for the other direction.)

[Sandman!]

Are ServerA and ServerB in the same subnet? If DNS is the problem then why would telnet be okay going from ServerA to ServerB and not vice-versa. Could you post the output of a traceroute from A to B and vice-versa. Make sure that it is not a routing problem.

[Luis Toro]

Different subnets. All 3 pieces (serverA, serverB and DNS server) are on 3 different Vlans. Can't provide a traceroute since this only occurs during a disaster recovery exercise (at a contracted DR site). Its happened during our last 2 tests, and now we're getting more pressure to resolve.

[Bill Hassell]

Actually, it doesn't matter how the DNS servers are supposed to work -- you test reverse lookup by using nslookup and nsquery:

nslookup 12.34.56.78
nsquery hosts 12.34.56.78

If you get no answer back, then telnet login will be delayed by the number of servers listed in /etc/resolv.conf. As you have seen, 'fixing' the problem using /etc/hosts proves that the DNS servers aren't configured correctly. nsquery is particularly useful as it shows the steps taken by any program looking for IP or hostname based on the nsswitch.conf file.

[Tor-Arne Nostdal]

You mention that this happens during DR test.

ARP?
****
Is it only the initial connect, or does the delay happen "always" as long as you're running on the failover?
Could it be that arp cache should be cleared...

Routing
*******
Is there different routing taking place in your network? (ref. traceroute, netstat -r, default gw)
In one of our systems we saw that even though we addressed the secondary interface on a server, we always ended up on the primary (default gw) when the 2 servers where in different vlans (redundancy 'all over' is nice, but not always giving you what you want). We had to create some static routes for the virtual addresses.

DNS (again)
***********
In my opinion all "critical hosts" for the system (tightly integrated hosts, like via NFS) should be defined locally in /etc/hosts.
This require some more managing, but you could have a central managment between these servers and use f.ex. rdist to help out keeping track of them. You might also want to use a file locking while editing the master and a history (for easier recovery).

Others might use DNS... and if it is an Windoze server... it's ehh some challenges.

* Check that both forward and reverse lookup is defined for the failover node.
* If it is Windows Administrators managing the DNS, you might also want to check if they have heard about case-sensitive issues.
Some applications actually treat ServerA differently from servera or SERVERA or ServerA...

The funny thing on some Windoze servers is that they need to key in the Forward lookup one place and the reverse another place. If you delete a Forward entry, the reverse is still there...
So - if you're using DNS, make sure it's right!

/Tor-Arne

[Luis Toro]

Thanks for all the info.
Bill, gave you extra points for the nsquery command (learn something new every day).
I will have to take this information with me for the next test, to make a checklist of things to check.