ssh session hangs during authentication and then disconnects

[Mel Burslan]

I have a bizarre problem in my hands. I have this server, moose, which sits in a demilitarized zone, behind a set of heavy firewalls. But the firewall rules allowed me to ssh to this server from my internal network. Sometime about 2 weeks ago, this communication was broken and nobody noticed it until last Friday. And since then, we are having a ping-pong match between the network team and UNIX team.

Here is the nitty-gritty:

I am on an hpux server, joe, inside the internal network and want to ssh to moose (with debugging enabled). Here is the debug trace:

[root@joe:/root]
# ssh -vvv mburslan@moose

OpenSSH_4.4p1-hpn12v11, OpenSSL 0.9.7l 28 Sep 2006
HP-UX Secure Shell-A.04.40.006, HP-UX Secure Shell version
debug1: Reading configuration data /opt/ssh/etc/ssh_config
debug3: RNG is ready, skipping seeding
debug2: ssh_connect: needpriv 0
debug1: Connecting to moose [x.x.x.x] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/3
debug1: identity file /root/.ssh/id_rsa type -1
debug3: Not a RSA1 key file /root/.ssh/id_dsa.
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /root/.ssh/id_dsa type 2
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.4p1-hpn11
debug1: match: OpenSSH_4.4p1-hpn12v11 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_4.4p1-hpn12v11
debug2: fd 4 setting O_NONBLOCK
debug3: RNG is ready, skipping seeding
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfourr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfourr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@ope6
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@ope6
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfourr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfourr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@ope6
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,hmac-ripemd160,hmac-ripemd160@ope6
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_init: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_init: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 129/256
debug2: bits set: 489/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: check_host_in_hostfile: filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 105
debug3: check_host_in_hostfile: filename /root/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 106
debug1: Host 'moose' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:105
debug2: bits set: 497/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/id_rsa (0)
debug2: key: /root/.ssh/id_dsa (4002ef98)
Connection closed by x.x.x.x

connection closed message comes after a timeout of some sort


Also when I am on the console of moose, and want to ssh into the same server as myself:

# ssh -vvv mburslan@moose
OpenSSH_4.4p1-hpn12v11, OpenSSL 0.9.7l 28 Sep 2006
HP-UX Secure Shell-A.04.40.007, HP-UX Secure Shell version
debug1: Reading configuration data /opt/ssh/etc/ssh_config
debug3: RNG is ready, skipping seeding
debug2: ssh_connect: needpriv 0

and the session trace output stops at this point, going no further for a reasonably long amount of time before I hit ctrl-c to gain access to the console again.

I have compared the sshd_config files between this server and another similarly configured server and found them to be the same with the same version of ssd running on both servers. But the other one works and this is not.

Does anyone have any clue what might be wrong here. I am not very good at reading the trace outputs but can not see anything glaringly wrong in this output. But again, I might very well be mistaken.

Any help is greatly appreciated.

[Steven E. Protter]

Shalom,

Clearly there is two way communication between your client system and moose.

If relevant, use a linux system to test nmap and see that port 22 on moose is indeed open fully. This is a portscan tool and you may need permission from networking to run it.

It is possible the NIC card in moose or even the client has gotten flakey and is the cause of this problem. I'd run hardware diagnosis on both and you may find a problem. nmap is also a potential source.

Doesn't sound like the ping pong department cares, but a change in switch settings or even ios could trigger this issue as well.

You may wish to run wireshark or tcpdump and get a big bad ugly dump on the thing.

You may just be experiencing a timeout due to another problem and want to increase that setting in sshd_config on the server.

Looking again:

debug3: RNG is ready, skipping seeding

My that is strange. I think this is further evidence of network card problems. Seeding should not be skipped.

SEP

[Mel Burslan]

running the sshd in debug mode showed us that, there is a reverse dns lookup issue due to an resolv.conf file being placed where there should be no dns lookups on this server. It is funny that,nsswitch.conf only references "files" and no sign of dns in it but still the server has a resolv.conf file and sshd picks thatup when it needs to make a reverse lookup. Great coding practice huh ?

[Armin Kunaschik]

One short note afterwards:
"RNG is ready, skipping seeding" means, that ssh uses /dev/random for seeding, not ssh_prng_cmds. So this is pretty normal and no error.

Armin