SSH, REMSH, IPv6, resolver, nsswitch issues

[Bob_Vance]

Since this is longwinded, I'll ask my questions first,
then you can read the post:

1) any comments on the following are welcome

2) How can I make REMSH not do a IPv6 lookup



I've had a resolver problem ever since moving to 11.23 with SSH and REMSH ,
specifically when using shortname instead of Fully Qualified Domain Name (FQDN). However, there are also issues when using the FQDN.

It manifested as
a very long delay (usually ~35 secs, sometimes 105)
before resolving the address and connecting to the target.

((NOTE: I have "fixed" the SSH issue -- see later in the post
))


I did a network trace and have discovered a few issues:

****
1) /etc/nsswitch.conf
****
the new SSH, REMSH, and TELNET do not use the same resolver logic that nslookup and ping use.
Specifically, they ignore /etc/nsswitch.conf.

REGARDLESS of what nsswitch.conf contains,
their built-in resolver does:

. try DNS first
if /etc/resolv.conf does not exist or does not have nameserver entries,
it will try 127.0.0.1 -- 4 times !!

. then files


ping and nslookup *DO* honor nsswitch


****
2) IPv6 and delay
****
# ssh myhost

I discovered that the new SSH and REMSH actually try an IPv6 DNS lookup (type 28=AAAA) first.

If it does not get a hit (i.e., get an AAAA answer), it will go thru all of the resolver's logic of adding domains from /etc/resolv.conf.
When it exhausts all of those iterations it will finally try a normal IPv4 type A lookup.
Then they try an IPv4 lookup (type A)

If that's not bad enough, the second name that he tries is the unmodified name, EVEN when it is only a single domain (shortname). It probably should never try a single word.
This will cause the request to be forwarded to a DNS server that accesses the Internet and he'll flounder around trying to resolve domain "myHOST", which does not exist.

So for
# ssh myhost
the sequence is:

. modifies myhost to myhost.mydom.dom (from the /etc/resolv.conf file)
. does a AAAA name lookup (ignoring nsswitch) on myhost.mydom.dom
. gets a reply with 0 answers
. tries AAAA lookup on myhost, which causes the nameserver to go out and look for top domain
. it does this 4 times
. finally it tries a normal type A lookup on myhost.mydom.dom, which works


Anyway that's the source of the delay.


REMSH does the same thing.

TELNET does not try IPv6



****
3) with FQDN
****
A litle summary, even with a FQDN

# grep hosts /etc/nsswitch.conf
hosts: files

# cat /etc/resolv.conf
cat: No such file or directory

# ssh FQDN myHOST

IPv4 only
Did *8* TYPE A queries to 127.0.0.1 DNS
IPV6 also
Did *8* TYPE AAAAA queries to 127.0.0.1 DNS
Did *8* TYPE A queries to 127.0.0.1 DNS

REMSH always does IPv6 first
Did *8* TYPE AAAAA queries to 127.0.0.1 DNS
Did *8* TYPE A queries to 127.0.0.1 DNS

TELNET always does only IPv4
Did *8* TYPE A queries to 127.0.0.1 DNS

ping & nslookup
use ONLY /etc/hosts



****
4) SSH "fix"
****

I was able to get around the delay problem with SSH client, by modifying its config file, /opt/ssh/etc/ssh_config.
In there,
AddressFamily
defaults to "any", meaing IPv6 and IPv4
By forcing IPv4 only, by setting
AddressFamily inet
he gets a hit on the first modified name lookup and there is no delay.



tks
bv

[Bob_Vance]

In case you want to try this yourself, here is the way I did the trace:

## T=IPv4short; nettl -traceon 0x30800000 -e ns_ls_ip | tee /tmp/raw$T \
|netfmt -F -N -n -l -c /tmp/nf1 \
| tee /tmp/fmt$T

## cat /tmp/nf1
filter udp_sport 53
filter udp_dport 53


Why the 'tee' commands?

'tee'ing to the raw file allows me to do a different filter later if I want, typically just removing the "-l" to get detailed hex data.

'tee'ing the format file allows me to change the T variable and keep different outputs for later perusal.


I run the 'nettl' command on both the local server where I'm executing the
# ssh myHOST
to see its DNS requests
*and* on the DNS server being used out of /etc/resolv.conf, to see how it is handling the requests, specifically the forwarding.

tks
bv

[Bob_Vance]

BTW, a very simple way to test whether nsswitch is being used or not is:

1) set nsswitch.conf to only files:

## grep hosts /etc/nsswitch.conf
hosts: files

or more safely to files dns :

## grep hosts /etc/nsswitch.conf
hosts: files [NOTFOUND=continue TRYAGAIN=continue UNAVAIL=continue] dns


2) Add an entry to /etc/hosts with an invalid IP address that overrides the correct IP address in DNS:

daytona3 ## cat /etc/resolv.conf
search testtrack
nameserver 10.1.0.99

## dig daytona3console.testtrack @10.1.0.99
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1
daytona3console.testtrack. 3600 IN A 192.168.11.153

## nslookup daytona3console.testtrack
Using /etc/hosts on: daytona3
looking up FILES
Name: daytona3console.testtrack
Address: 2.3.4.5

So, /etc/hosts has bogus 2.3.4.5 and DNS has correct 192.168.11.153


3) do the ssh:

## ssh daytona3console.testtrack
Received disconnect from 192.168.11.153: 11: SSH Disabled


So you can see that it tried the correct DNS IP, not /etc/hosts entry.


bv

[VK2COT]

Hello Bob,

All HP-UX 11.31 and 11.23 servers that I
just tested were fine for SSH.

I do not have IPv6 networks though.

Also, I could test REMSH because the last
time I used it was about 10-15 years ago.
The environments I work in do not allow
plain-text protocols.

With SSH on HP-UX 11.23 and 11.31, I easily
confirmed that it used "files" before "dns"
if nsswitch.conf had line like (or similar):

hosts: files dns

In regards to by-passing IPv6, couple of
options:

a) Use this flag:

ssh -o AddressFamily=inet ...

or

b) Set /opt/ssh/etc/sshd_config (or
wherever your SSH installation resides):

AddressFamily inet

... which forces IPv4 to be used.

Cheers,

VK2COT

[Bob_Vance]

Thanks, VK2COT.

I mentioned in my post that I resolved the IPv6 issue for SSH by modifying ssh_config.

However, I'm intrigued that your nsswitch.conf worked. You saw the text from my testing in the post just prior to yours, right?

I have tested 2 different sets of systems:
my own 3 test systems
and a customer's 6 servers.

My test systems were installed from scratch to 11.23, while the customer systems were upgraded from 11.11.

Both my test systems and the customer's systems behave the same way, bypassing nsswitch.conf !!!

There are no permission problems on nsswitch.conf, plus I tested as root.


This is very curious!!!

I'm wondering if some patching may be required. They haven't been patched since last May.


bv

[VK2COT]

Hello,

I wonder if you could run my Perl script and
save results into text file and see if some
unusual problem is hurting you:

http://www.circlingcycle.com.au/Unix-sources/HP-UX-check-OAT.pl/.txt

By the way, older versions of OpenSSH had
that bug - for example, Name Resolving bug in OpenSSH 3.0.2...

Cheers,

VK2COT

[VK2COT]

My typing is bad. The URL is:

http://www.circlingcycle.com.au/Unix-sources/HP-UX-check-OAT.pl.txt

Just to be safe, check /etc/hosts for
corruption or some hidden (unprintable)
characters.

VK2COT

[Bob_Vance]

OK
I found the resolver/nsswitch culprit and another workaround.
One word:

... ipnodes

As usual, finding the problem explains everything ;>)


Anyway adding "ipnodes" to /etc/nsswitch.conf

daytona3 ## grep -e hosts -e ipnodes /etc/nsswitch.conf
...hosts: files [NOTFOUND=continue UNAVAIL=continue] dns
...ipnodes: files [NOTFOUND=continue UNAVAIL=continue] dns


fixed the resolver/nsswitch not looking at /etc/hosts problem.



Somehow the system is configured thinking that it supports and should use IPv6. Of course, we noticed this in the network trace where it tried type AAAA name lookups first.

Just now, in reading about IPv6 configuration, I discovered that there is a *separate* entry in nsswitch.conf, "ipnodes", to use for IPv6, instead of "hosts". I have not played with IPv6 and was not aware of that.

So, I had no "ipnodes" entry in my nsswitch.conf !!


I re-read nsswitch.conf man page and it documents "ipnodes", but makes no mention of what it is for.

But, it cleary states

... The library functions contain compiled-in default entries that are
... used if the appropriate entry in nsswitch.conf is absent or
... syntactically incorrect. The entries are as follows:

...... hosts: dns [NOTFOUND=return] nis [NOTFOUND=return] files
...... ipnodes: dns [NOTFOUND = return] files
......

Since I had no "ipnodes" entry, that explains exactly what I "discovered" in my testing and trace analysis: dns, then files ;>)


Why did I not have "ipnodes" ?
The sample files that HP provides *do* have "ipnodes" defined in them, as you would expect.

Presumably, the upgrade from 11.11 to 11.23 simply kept the original version of the file.
The release notes for the upgrade probably mentioned the IPv6/ipnodes thingy and I must have overlooked it.



bv

[Bob_Vance]

However, simply fixing nsswitch
still leaves two issues:

1) How to disable IPv6 altogether, so REMSH doesn't try IPv6 lookup.

Why? Because the resolver does the following when using DNS:

... try IPv6 lookup on modified shortname (shortname.domain)
... since we don't have IPv6 DNS, it gets 0 answers.
... then it tries the UNmodified shortname !!-- this amounts to a lookup on a non-existent Top-Level Domain and causes forwarding, retries and a long delay before he gives up on IPv6.
... finally it tries IPv4 lookup on modified shortname and it will get a hit here.

So, the IPv6 lookup is causing a long delay when your DNS does not support it.


2) Thus, it would be nice to be able to option the resolver not to try the unmodified name when it contains 0 dots. This would prevent hunting all over the place for a non-existent TLD.
E.g.,
options mindots:1

option ndots:n
is sorta the obverse of this



bv

[Avinash20]

Chk if this helps

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=882565

[Avinash20]

Oops.. you already know :)
But this is very interested,,

[Bob_Vance]

It should be emphasized that
"ipnodes" in nsswitch does
NOT *prevent* IPv6 lookups.

All it does is to allow the resolver to correctly use "files" first when you have:

ipnodes: files dns


As I mentioned previously,
to see this, simply do a
# remsh SHORTNAME_not_in_hosts_but_in_DNS

it will :

. try /etc/hosts

. not find it

. try IPv6 DNS lookup on SHORTNAME_not_in_hosts_but_in_dns.SEARCH_DOMAIN

. get a reply from your DNS with 0 answers (because your DNS is not supporting IPv6 AAAA records)

. treat this as a no-hit

. and then try IPv6 lookup on UNMODIFIED SHORTNAME_not_in_hosts_but_in_DNS
!!

. this is essentially a non-existent TLD that will be forwarded and retried causing a large delay

. finally give up on IPv6

. try *IPv4* DNS lookup on SHORTNAME_not_in_hosts_but_in_dns.SEARCH_DOMAIN

. get correct answer

. and go Ahhhhh


You can turn off IPv6 lookup for SSH in ssh_config via:
/opt/ssh/etc/ssh_config :
# AddressFamily any
AddressFamily inet

I have not yet found how to stop REMSH from doing it.



bv

[VK2COT]

Hello Bob,

Aha, famous "missing ipnodes" issue.

It is actually one of my 600-odd tests for
HP-UX in the script I listed before.

As far as remsh is concerned, I sincerely
suggest - stop using it. It is primitive and
insecure :) You already have ssh, so
migrate your scripts to use it...

Cheers,

VK2COT

[Bob_Vance]

I was just about to use your script when I happened on ipnodes for myself. I probably could have saved myself a couple hours research had I used it -- but it wouldn't have been as much fun ;>)

vis-a-vis remsh, we of course use SSH. I just used it to see if it behaved differently than SSH for this problem.
But now it kinda annoys me that I can make SSH not use IPv6, while I cannot do the same for the R-utils =:|


bv

[Bob_Vance]

((the **N means see footnote.
))

There doesn't seem to exist a very good solution to this. **1

So, I'm closing the thread.

bv






**1
Long delay when trying ssh shortname_not_in_etc_hosts




**************
Some postmortem thoughts:
**************

The issue is really with short-name lookups in general, but is exacerbated with the new IPv6 stuff. **2


1) IPv6

SSH can be configured to avoid IPv6 -- good.

REMSH, et al., cannot -- not so good. But don't really need them if you have SSH, anyway.

However, I ran into the same issue with 'xauth', but did not really follow up on it.

TELNET doesn't even try IPv6.


Probably the best fix here would be to have a system setting that tells the resolver(s) not to try IPv6 AAAA lookups.



2) short names
OTOH, the issue is really only a problem when you use short names instead of FQDNs (or make a mistype in the TLD). **3

So, maybe the resolver option that I proposed would help; viz.:

do not try a DNS lookup on the unmodified name unless it contains at least one dot.


However, this brings to mind an admonishment I was given many years ago by a DNS guru. He was of the opinion that the resolver search list was a lame, lazy way to do things that clogged the name servers.

Actually he phrased it very succinctly.

"don't F'n use F'n short names !!!!





**2
Some of the resolver(s) are now coded to try *first* IPv6 AAAA DNS lookups on hostnames. If your hostname does not end with a dot, then the resolver will try the name modified by appending the domain to it. If you do not have IPv6 set up, then your DNS server will send back a 0-answer reply. The resolver then tries the unmodified name, but still an IPv6 lookup.

When he has cycled thru all the search domains, he will try "normal" IPv4 lookups.

So, when we are not using IPv6 we have a bunch of extraneous IPv6 lookups.



**3
The first modified name IPv6 lookup will be fast, because your DNS server will be authoritative for the FQDN and immediately will send back a 0-answer reply, since you have no IPv6 records in it.

Now, the resolver tries the un-modified name.
Since your DNS server most likely will *not* be authoritative for the unmodified name (certainly so when it does not contain a dot), it will ultimately be forwarded out to the Internet. In addition, because of timing issues, the resolver will do this several times.

Finally (after a long preceived delay), when all the IPv6 lookups are exhausted, he will do the same thing with IPv4 type A lookups, with another delay !!


The issue here is that the problem is not just a local, long delay. The unmodified, no-dot short-name lookup can ultimately get forwarded out to Internet DNS servers, and that's just not being a good netizen.