Unable to SSH to iLO2 with OpenSSH 6.2

[sl4mmy]

Howdy-

I initially posted this in reply to the v2.15 release announcement, but I'm starting a separate thread now because I reproduced the issue with another SSH client (the Ruby Net::SSH library from http://net-ssh.github.io/net-ssh/).

Basically, I'm unable to connect to iLO2 via SSH from my Linux workstation.  I tried with servers running iLO2 firmware v2.06, v2.12 and the recently released v2.15, all without success.  My workstation is running ArchLinux:

$ uname -a

Linux arch-sl4mmy 3.7.10-1-ARCH #1 SMP PREEMPT Thu Feb 28 09:50:17 CET 2013 x86_64 GNU/Linux

Here is some sample debug output when attempting to connect using OpenSSH 6.2:

$ ssh -vvv ilo01

OpenSSH_6.2p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/sl4mmy/.ssh/config
debug1: /home/sl4mmy/.ssh/config line 15: Applying options for ilo*
debug1: /home/sl4mmy/.ssh/config line 24: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to ilo01 [192.168.254.11] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 999 ms remain after connect
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/sl4mmy/.ssh/deploy-key" as a RSA1 public key
debug1: identity file /home/sl4mmy/.ssh/deploy-key type 1
debug1: identity file /home/sl4mmy/.ssh/deploy-key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version mpSSH_0.1.1
debug1: no match: mpSSH_0.1.1
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "ilo01" from file "/home/sl4mmy/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/sl4mmy/.ssh/known_hosts:274
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug2: dh_gen_key: priv key bits set: 126/256
debug2: bits set: 503/1024
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
Received disconnect from 192.168.254.11: 2: Client Disconnect

I also reproduced what I believe is the same issue using Ruby's Net::SSH library (the output below was captured by setting the :logger option to Logger::DEBUG):

D, [2013-05-03T10:17:09.222960 #2074] DEBUG -- net.ssh.transport.session[d272c4]: establishing connection to ilo01:22
D, [2013-05-03T10:17:09.224466 #2074] DEBUG -- net.ssh.transport.session[d272c4]: connection established
I, [2013-05-03T10:17:09.224580 #2074]  INFO -- net.ssh.transport.server_version[d24970]: negotiating protocol version
D, [2013-05-03T10:17:09.231403 #2074] DEBUG -- net.ssh.transport.server_version[d24970]: remote is `SSH-2.0-mpSSH_0.1.1'
D, [2013-05-03T10:17:09.231458 #2074] DEBUG -- net.ssh.transport.server_version[d24970]: local is `SSH-2.0-Ruby/Net::SSH_2.6.7 x86_64-linux'
D, [2013-05-03T10:17:09.485039 #2074] DEBUG -- tcpsocket[d26400]: read 200 bytes
D, [2013-05-03T10:17:09.485179 #2074] DEBUG -- tcpsocket[d26400]: received packet nr 0 type 20 len 196
I, [2013-05-03T10:17:09.485255 #2074]  INFO -- net.ssh.transport.algorithms[d21a40]: got KEXINIT from server
I, [2013-05-03T10:17:09.485329 #2074]  INFO -- net.ssh.transport.algorithms[d21a40]: sending KEXINIT
D, [2013-05-03T10:17:09.487958 #2074] DEBUG -- tcpsocket[d26400]: queueing packet nr 0 type 20 len 1620
D, [2013-05-03T10:17:09.488054 #2074] DEBUG -- tcpsocket[d26400]: sent 1624 bytes
I, [2013-05-03T10:17:09.488096 #2074]  INFO -- net.ssh.transport.algorithms[d21a40]: negotiating algorithms
D, [2013-05-03T10:17:09.488242 #2074] DEBUG -- net.ssh.transport.algorithms[d21a40]: negotiated:
* kex: diffie-hellman-group1-sha1
* host_key: ssh-rsa
* encryption_server: aes128-cbc
* encryption_client: aes128-cbc
* hmac_client: hmac-sha1
* hmac_server: hmac-sha1
* compression_client: none
* compression_server: none
* language_client:
* language_server:
D, [2013-05-03T10:17:09.488321 #2074] DEBUG -- net.ssh.transport.algorithms[d21a40]: exchanging keys
D, [2013-05-03T10:17:09.489091 #2074] DEBUG -- tcpsocket[d26400]: queueing packet nr 1 type 30 len 140
D, [2013-05-03T10:17:09.489145 #2074] DEBUG -- tcpsocket[d26400]: sent 144 bytes
D, [2013-05-03T10:17:09.490307 #2074] DEBUG -- tcpsocket[d26400]: read 40 bytes
D, [2013-05-03T10:17:09.490411 #2074] DEBUG -- tcpsocket[d26400]: received packet nr 1 type 1 len 36

And then it disconnects.

$ ruby --version

ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]

$ gem list | grep net-ssh

net-ssh (2.6.7)

Has anyone else encountered similar problems?  Please let me know if I can provide any more information to help identify and fix this issue.

[sl4mmy]

I came across this thread from two years ago http://www.gossamer-threads.com/lists/openssh/dev/51909 that describes a similar issue with OpenSSH 5.8, but unfortunately the recommended work-arounds no longer seem to work with OpenSSH 6.2.

[Oscar A. Perez]

Did you try the option 

HostKeyAlgorithms=ssh-rsa

[sl4mmy]

Hi, Oscar-

Yes, I did try that.  Sorry for not being more clear, but that's what I meant about "recommended work-arounds no longer seem to work with OpenSSH 6.2."

Are you able to successfully to connect to iLO2 via SSH with OpenSSH v6.2 using that option?

Here is the output with that option on my machine:

$ ssh -vvv -o HostKeyAlgorithms=ssh-rsa ilo01

OpenSSH_6.2p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/sl4mmy/.ssh/config
debug1: /home/sl4mmy/.ssh/config line 15: Applying options for ilo*
debug1: /home/sl4mmy/.ssh/config line 24: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to ilo01 [192.168.254.11] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 999 ms remain after connect
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/sl4mmy/.ssh/deploy-key" as a RSA1 public key
debug1: identity file /home/sl4mmy/.ssh/deploy-key type 1
debug1: identity file /home/sl4mmy/.ssh/deploy-key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version mpSSH_0.1.1
debug1: no match: mpSSH_0.1.1
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug2: dh_gen_key: priv key bits set: 129/256
debug2: bits set: 506/1024
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
Received disconnect from 192.168.254.11: 2: Client Disconnect

[Oscar A. Perez]

Ok, I'll debug it and hopefully it is an easy fix.  I'm getting tired of fixing iLO2 SSH server everytime a new OpenSSH version is released.

[sl4mmy]

Hi, Oscar-

Great, thanks for offering to look into the issue further!  Let me know if there are any further options you'd like me to try or other tests you'd like me to run.

[Jimmy Vance]

I know many years ago when there was an issues with iLO and openSSH not working together the workaround was to add "-o ForwardAgent=no -o ForwardX11=no " The issue I had at the time was the ssh client wouldn't use the "-o" options properly from the command line. I had to put them in a file and launch ssh with the "-F configfile" option to read the options correctly. Not saying that is the issue here, but you might give it a try creating a file with "HostKeyAlgorithms=ssh-rsa" and see if it changes anything

[sl4mmy]

Hi, Jimmy-

Thanks for your suggestion, unfortunately I'm still unable to connect:

$ cat ssh_config

Host *
  KexAlgorithms diffie-hellman-group1-sha1
  HostKeyAlgorithms ssh-rsa

$ ssh -vvv -F ssh_config ilo01

OpenSSH_6.2p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data ssh_config
debug1: ssh_config line 1: Applying options for *
debug3: kex names ok: [diffie-hellman-group1-sha1]
debug3: key names ok: [ssh-rsa]
debug2: ssh_connect: needpriv 0
debug1: Connecting to ilo01 [192.168.254.11] port 22.
debug1: Connection established.
debug1: identity file /home/sl4mmy/.ssh/id_rsa type -1
debug1: identity file /home/sl4mmy/.ssh/id_rsa-cert type -1
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/sl4mmy/.ssh/id_dsa" as a RSA1 public key
debug1: identity file /home/sl4mmy/.ssh/id_dsa type 2
debug1: identity file /home/sl4mmy/.ssh/id_dsa-cert type -1
debug1: identity file /home/sl4mmy/.ssh/id_ecdsa type -1
debug1: identity file /home/sl4mmy/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version mpSSH_0.1.1
debug1: no match: mpSSH_0.1.1
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-cbc hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-cbc hmac-md5 none
debug2: dh_gen_key: priv key bits set: 136/256
debug2: bits set: 531/1024
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
Received disconnect from 192.168.254.11: 2: Client Disconnect

[Oscar A. Perez]

I had to make lots of changes to the mpSSH server code to get it to work with the new OpenSSH 6.2p1.  

I hope this is the last time we have to make changes like this one. iLO2 memory is very limited and already full so,  we won't be able to spin new firmware releases, every time the OpenSSH folks decide to increase the size of the payload during Key Exchange.

[sl4mmy]

Hi, Oscar-

That's great, I'm glad you were able to fix the problem.  I'm not sure what can be done about futureproofing, but I appreciate your time and effort on this!

Thanks again!

Best,

Kent

[sl4mmy]

Hi, Oscar-

I posted a general question on the OpenSSH dev list about compatibility with memory constrained embedded SSH implementations.

However, after taking a look at some of the relevant standards I don't think this can be fairly called an OpenSSH problem.  Reading RFC 4253 sections 6.2 - 6.5 it seems the standard allows for additional compression algorithms, encryption algorithms, key exchange methods, etc. in the future.  Also, section 7 doesn't say anything further about maximum payload size during key exchange so I assume only the requirements from section 6.1 apply.  Looking at the conversation between my OpenSSH client and an iLO2 interface running firmware v2.15 it looks like the largest packet is only 1.4k, far short of the 35k max size that implementations must support.  So while I realize there is limited memory for iLO2, it really sounds like mpSSH's responsibility to handle large payloads and ignore unknown algorithms, etc.

I hope that doesn't come across as too snarky.  Thanks again for your help with this issue!

Best,

Kent

[Oscar A. Perez]

Thanks for that info.

mpSSH in iLO2 v2.15 would only handle 1280 bytes payload during key exchange. I increased that to 2Kb in version 2.20 (ETA later this month) and there isn't an easy way I could prevent mpSSH from disconnecting on packets larger than 2Kb without making significant changes to the code. 

I forgot to debug why the workaround of using the option “HostKeyAlgorithms=ssh-rsa” isn't working anymore. Not sure if the option got broken in OpenSSH 6.2p1 or there is something else going on within mpSSH. I will take a look because, looking forward this option might be the only way to connect to iLO2, if OpenSSH increases the payload beyond 2Kb

[sl4mmy · Accepted solution]

Hi, Oscar-

Ah-ha!  Your message just triggered an idea, and sure enough: it works!  :)

The -o HostKeyAlgorithms=ssh-rsa option doesn't work anymore with OpenSSH 6.2p1 because it's insufficient to keep the key exchange payload small enough for mpSSH to handle.  Some other value during key exchange has grown enough that the payload is still over the 1280 byte limit.  Looking at the output from a failed connection attempt the list of MAC algorithms sent by the client is the largest.

Sure enough, OpenSSH 6.2 is able to connect successfully when configured to only offer hmac-sha1:

$ ssh -vvv -o MACs=hmac-sha1 ilo01

OpenSSH_6.2p1, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /home/sl4mmy/.ssh/config
debug1: /home/sl4mmy/.ssh/config line 15: Applying options for ilo*
debug1: /home/sl4mmy/.ssh/config line 24: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug2: ssh_connect: needpriv 0
debug1: Connecting to ilo01 [192.168.254.11] port 22.
debug2: fd 3 setting O_NONBLOCK
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug3: timeout: 999 ms remain after connect
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version mpSSH_0.1.1
debug1: no match: mpSSH_0.1.1
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "ilo01" from file "/home/sl4mmy/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/sl4mmy/.ssh/known_hosts:274
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@open
ssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-sha1
debug2: kex_parse_kexinit: hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-sha1
debug1: kex: server->client aes128-cbc hmac-sha1 none
debug2: mac_setup: found hmac-sha1
debug1: kex: client->server aes128-cbc hmac-sha1 none
debug2: dh_gen_key: priv key bits set: 168/320
debug2: bits set: 557/1024
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug1: Server host key: RSA 84:ee:9f:9c:2e:46:8f:10:2d:30:07:5c:eb:94:a8:b4
debug3: load_hostkeys: loading entries for host "ilo01" from file "/home/sl4mmy/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/sl4mmy/.ssh/known_hosts:274
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "192.168.254.11" from file "/home/sl4mmy/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /home/sl4mmy/.ssh/known_hosts:278
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'ilo01' is known and matches the RSA host key.
debug1: Found key in /home/sl4mmy/.ssh/known_hosts:274
debug2: bits set: 523/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
...

So, long story short, I still believe the long term fix is to figure out how to make mpSSH handle larger payloads during key exchange more gracefully, but as a work-around it seems that whenever OpenSSH adds support for new algorithms users should explicitly use a subset of the algorithms that are known to work with mpSSH.  Perhaps you can recommend a canonical set of algorithms for each type that will always be guaranteed to work, for example:

$ ssh -o HostKeyAlgorithms=ssh-rsa,ssh-dss -o KexAlgorithms=diffie-hellman-group1-sha1 -o Ciphers=aes128-cbc,3des-cbc -o MACs=hmac-md5,hmac-sha1 username@ilo-hostname

[Oscar A. Perez]

Awesome!

You're pretty much showing all what is supported by mpSSH:

HostKeyAlgorithms: ssh-rsa, ssh-dss

KexAlgorithms: diffie-hellman-group1-sha1

Ciphers: aes128-cbc, 3des-cbc

MACs: hmac-md5, hmac-sha1

I will add this info to the Customer Advisory. :)

[Oscar A. Perez]

Fixed in iLO2 version 2.20

http://h30499.www3.hp.com/t5/ITRC-Remote-Lights-Out-Mgmt-iLO/iLO-2-Firmware-version-2-20-released/td-p/6067313

[sl4mmy]

Hi, Oscar-

That's great, thanks for releasing the fix so quickly!

[Renaud_]

What I have done is putting the following in the profile file, and it works with OpenSSH 6.2. You only have to use ilossh instead of ssh to login

I know that the problem is in the server, not the client, but that helps if you need to access older machines for which the ILO patch is not available.

alias ilossh='ssh -o PasswordAuthentication=yes \
-o ChallengeResponseAuthentication=no \
-o GSSAPIAuthentication=no \
-o HostbasedAuthentication=no \
-o PubkeyAuthentication=no \
-o RSAAuthentication=no \
-o Compression=no \
-o ForwardAgent=no \
-o ForwardX11=no \
-o KexAlgorithms=diffie-hellman-group1-sha1 \
-o MACs=hmac-md5,hmac-sha1 \
-o Ciphers=aes128-cbc,3des-cbc \
-o HostKeyAlgorithms=ssh-rsa,ssh-dss '

[ben-Nabiy]

I am currently running a DL380 G5 with iLO2 Advanced License

Current Firmware: 

1.35   07/16/2007

Am I able to upgrade the firmware to the 2.2 to overcome the SSH issue? If yes, do I need to step through the upgrades one by one?

I am afraid of ruining my currently working install (minus the ssh issue) to try updating if it is not going to work, and I have no way to bring it back to the current version.

[Oscar A. Perez]

No, you don't need to step through every upgrade one by one but, since I've never tested upgrading from such old firmware version directly to a version 2.xx, I would advise you to write down all the important iLO2 configurations, including your advanced license key and then upgrade that iLO2 to version 1.82.

If everthing seems fine after upgrading to v1.82 then, upgrading to version 2.23 should go smooth.

[ben-Nabiy]

Thank you for the prompt response.

I have downloaded the 2.2 firmware, but am unsure how to get the 1.82. The directory structure of

ftp://ftp.hp.com/pub/softlib2/software1/sc-linux-fw-ilo/p1285463034/v85709/

does not make it very evident.

If you could link me to the proper firmware for what you would recommend, I would much appreciate it!

[ben-Nabiy]

Please disregard that previous post. I have the right firmware, but just have a question as to which settings will get reset upon firmware update. Will my users and SSH info and IP address change?

Also, is there any way to download the current firmware just in case something happens and the upgrade fails?

[ben-Nabiy]

Two Step Upgrade went Flawless.

1.35 -> 1.82 -> 2.23

[phoeniXfury]

Hi there,

Normally I would be opposed to hijacking a thread. However, we seem to be having the same issue with our GbE2c switches in our blades. Connecting from OpenSSH 6.2 and higher seems to trigger a payload related error on the switch console output.

The workaround used here "-o MACs=hmac-sha1" also works for us. Any chance this fix can be done there or is it a complete different team?

I've made a post in the correct section but thought I'd link it here hoping it may help speed things up. Here's the link:

http://h30499.www3.hp.com/t5/HP-BladeSystem-Network-Blades/GbE2c-OpenSSH-6-2-Compatibility/td-p/6611884

[HaraldH]

I noted the same thing on a J9775A HPE Aruba 2530-48G switch. Unable to SSH the switch until I used '-o MACs=hmac-sha1'. Many thanks for the tip, this drove me crazy. Where could I report it?

[HPC-EzeE]

I am logging into bl280c G6 blades with iLO 2 and *updated iLO firmware* to the latest revision!  This problem is NOT resolved:

[root@h01 export]# ssh -vvv -o MACs=hmac-sha1 admin@ilon01
OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ilon01 [192.168.10.1] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug3: Incorrect RSA1 identifier
debug3: Could not load "/root/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version mpSSH_0.1.4
debug1: no match: mpSSH_0.1.4
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "ilon01" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:70
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-sha1
debug2: kex_parse_kexinit: hmac-sha1
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes256-ctr,aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: aes128-ctr,aes256-ctr,aes128-cbc,3des-cbc
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha2-256
debug2: kex_parse_kexinit: hmac-sha1,hmac-sha2-256
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit: none
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-sha1
debug1: kex: server->client aes128-ctr hmac-sha1 none
debug2: mac_setup: setup hmac-sha1
debug1: kex: client->server aes128-ctr hmac-sha1 none
debug1: kex: diffie-hellman-group14-sha1 need=20 dh_need=20
debug1: kex: diffie-hellman-group14-sha1 need=20 dh_need=20
debug2: bits set: 1007/2048
debug1: sending SSH2_MSG_KEXDH_INIT
debug1: expecting SSH2_MSG_KEXDH_REPLY
debug1: Server host key: RSA 83:18:6f:60:04:86:e4:68:4b:a7:a3:15:94:15:d1:36
debug3: load_hostkeys: loading entries for host "ilon01" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:70
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "192.168.10.1" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:69
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'ilon01' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:70
debug2: bits set: 999/2048
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
Received disconnect from 192.168.10.1: 2: Client Disconnect