SSH - Known Hosts - how to sync?

[Geoff Wild]

We are now using ssh with keys for our backup server. Unfortunately, when we move the package to the other node, the first time we try to run a backup, it fails as it is waiting for an answer - something like "man in the middle attack - are you sure you want to continue?" as we connect to the floating ip address. So we have to manualy connect to the server the first time to accept adding to the known hosts file.

Other option I thought of was to have 2 different versions of the known_hosts file, and when we fail over, make a ssh connection from the node in the cluster to the backup server and copy the known_hosts.node2 file to known_hosts.

Anyone know a better workaround?

Rgds...Geoff

[Oviwan]

Hy

you can also run a script which copy the known_hosts file when you move the package. there are two function "function customer_defined_run_cmds" and "function customer_defined_halt_cmds" in the packagename.cntl file.

Regards

[Ralph Grothe]

Hi Geoff,

for cluster nodes that essentially form a single host I would use the same SSH host keys on all of the nodes that belong to that cluster.
The host keys should reside in /opt/ssh/etc.
However, if you don't fancy the idea you could also make use of the HostKeyAlias option like

ssh -o hostkeyalias=nodeA nodeC

when your client actually comes from nodeB
(presumably owe to a package failover) instead of nodeA what he pretends by the HostKeyAlias.
But this is an ugly kludge, that I usally use when doing port forwarding through firewalled
routes.
In my opinion a common cluterwide hostkey
is the better choice.

[Geoff Wild]

As I said, we connect to the floating ip (not the real hosts names) as that is needed for oracle.

The issue is, when you ssh to floatip, it has a mac address.

When you ssh to the same floatip - but now on another host - it sees that the mac has changed.

The hostkeyalias is for making multiple hosts (IE in VSE) use the same MAC.

But, looking through the ssh_config man page, I see this option:

CheckHostIP
If this flag is set to ``yes'', ssh will additionally check the
host IP address in the known_hosts file. This allows ssh to
detect if a host key changed due to DNS spoofing. If the option
is set to ``no'', the check will not be executed. The default is
``yes''.


Maybe I'll try that.


Rgds...Geoff

[Steven E. Protter]

Shalom,

You can build into the cluster statup script a command that copies the active nodes known_hosts file to the proper location.

You may be able to play with the backup clients /etc/hosts file and make this go away. The problem of course is due to the hostname not matching.

SEP

[Denver Osborn]

I'd add "-o StrictHostKeyChecking=no" and "-o PreferredAuthentications=publickey" to your syntax.

The other option is to keep the HostKey the same between all nodes in the cluster. ITRC Doc KBRC00011982 mentions HostKeys being copied from one node to another.

-denver

[Olivier Masse]

One thing you can try is having a second ssh daemon that listens to another port on each of your nodes (no need for a package for this), and have it use the same host key on each server in your cluster. That's what I'm planning to do soon.

Let's say you decide to use port 822. When you connect to server:22, you access the normal ssh service of the node. But when you connect to floating:822, you access a high-availability ssh service of some sort, for which the host key will be consistent across your cluster.

I suggest you restrict a lot the service on this port, for instance make it ONLY allow logins with certificates, and use AllowUsers to further limit who can access it.

Another thing I've been doing in the past is manually adding multiple host keys in known_hosts for the same floating address, AFAIK it used to work two years ago with openssh derivatives but I had to forget with other third-party SSH tools (winscp, etc). However, I'm not sure it still works with more recent version of HP-UX Secure Shell.

Good luck

[Ermin Borovac]

Here is another option.

For example if your SG cluster consists of two hosts and you might have the following two lines in your known_hosts file.

, ssh-rsa
, ssh-rsa

Edit known_hosts file adding SG package name and IP to both lines as shown below.

,,, ssh-rsa
,,, ssh-rsa

Also make sure to remove any other lines in known_hosts file that contain or .

[Ralph Grothe · Accepted solution]

Hi Geoff,

I can't share your difficulties.
The HostKeyAlias option for the ssh client
is exactly provided for cases as yours
I beleive.
As said, I regularily make use of it whenever
I tunnel some service via ssh over our admin server to e.g. avoid in-between copying
because the copying source host isn't allowed to make direct connections to the destination host (so called local port forwarding).
Naturally in a situation as such I would get a clash of hostkeys between my client's know_host entry and the now forwarded to deviating destination host's host key.

Admitedly my explanation was wrong because I mixed up whose host keys clashed.

So I reconstructed your case with one of our clusters (only spoofed IP addresses and removed fingerprints here for paranioa's sake)
My admin host (i.e. where ssh client connects from) be 10.10.10.10
and my package's "floating" IP be 123.123.123.123
that before the switch over should run on nodeA.
I deleted host keys for IP 123.123.123.123 in me's known_host on 10.10.10.10 prior to tests, why I am asked on 1st time connection for confirmation of nodeA's hostkey:

$ ssh me@123.123.123.123 hostname
The authenticity of host '123.123.123.123 (123.123.123.123)' can't be established.
RSA key fingerprint is
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '123.123.123.123' (RSA) to the list of known hosts.
Last login: Thu Dec 7 14:11:11 2006 from 10.10.10.10
(c)Copyright 1983-2000 Hewlett-Packard Co., All Rights Reserved.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-1992 Sun Microsystems, Inc.

nodeA

Now nodeA's host key for IP 123.123.123.123 should reside in me's known_host.

I then login on cluster nodeB as root
and switch packageA from nodeA over to nodeB


[root@nodeB:/root]
# cmhaltpkg packageA && cmrunpkg -n nodeB packageB
One or more packages has been halted and will not be started automatically. To start these packages, enable AUTO_RUN via cmmodpkg -e .
cmhaltpkg : Completed successfully on all packages specified.
cmrunpkg : Completed successfully on all packages specified.
[root@nodeB:/root]
# cmviewcl -p packageA

PACKAGE STATUS STATE AUTO_RUN NODE
packageA up running disabled nodeB


and verify that IP 123.123.123.123 is now active on nodeB


[root@nodeB:/root]
# netstat -in|grep '123.123.123.123'
lan0:3 1500 123.123.123.0 123.123.123.123 6 0 0 0 0


When I now retry to connect to that IP
from me's management host 10.10.10.10 as before SSH discovers that the host key
entry in me's known_host for this IP
isn't the one that nodeB now is advertising,
and rightly refuses to continue.


$ ssh me@123.123.123.123 hostname
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is

Please contact your system administrator.
Add correct host key in /home/me/.ssh/known_hosts to get rid of this message
.
Offending key in /home/me/.ssh/known_hosts:949
RSA host key for 123.123.123.123 has changed and you have requested strict checking.

Host key verification failed.


But knowing that 123.123.123.123 is now bound to nodeB I can tell the ssh client to instead refer to the host key entry of nodeB,
so that it can continue to RSA public key autorization and execute the command.


$ ssh -o hostkeyalias=nodeB me@123.123.123.123 hostname
nodeB


If you want to make sure that your command
doesn't get hung because of possible host key clashes or other reasons,
you can always add -o batchmode=yes to your ssh command, or simply -B if you are doing scp.

You can also put these, or any other options
in your client's $HOME/.ssh/config file
so that you need not supply clumsy command lines.


Nevertheless, I still would maintain to use
just a single hostkey on all nodes of a cluster to avoid all this.
On the other hand you usually also allow
root's remsh and rcp commands (trusted host environment) between cluster nodes,
which is merely another manifestation of "one-host concept".

Regards
Ralph

[Florian Heigl (new acc)]

Hi Geoff,

You should attach a specific sshd instance with the cluster package.
Only clean solution :)

Regards,
Florian

[Florian Heigl (new acc)]

Roughly:

1. Set the sshd on nodeA to listen only on ip_of_nodeA:22
2. Set the sshd on nodeB to listen only on ip_of_nodeB:22

3. Launch an extra ssh daemon with the package startup script listening to vip_package:22

I didn't know the hostalias option until just now, and I'm too tired to fully understand it, but I _very_ _much_ _doubt_ it's the right tool for the job.


The only tricky part from my point of via is doing a clean install of the extra sshd, I personally would probably just copy it, but with more insight one could also either try relocating the SD package or repackage it. I just fail at building my own packages for too long to bother anymore.

[PeterWolfe]

Ralph's/Denver's original answers were the cleanest solution to the
the problem:

>for cluster nodes that essentially form a single host I would use the same
>SSH host keys on all of the nodes that belong to that cluster. The host keys
>should reside in /opt/ssh/etc.

The cluster is a single security domain. It makes sense to for each member to
have the same host key (/opt/ssh/etc). DSAU uses this exact configuration in
order to support SG package failover when the remote clients are using ssh
tunnels to connect to the package. Package failover and the subsequent ssh
reconnect works correctly. The exact docs on the procedure are in the
Distributed Systems Administration Utilities User's Guide in the section s"sh
Port Forwarding to a Serviceguard Cluster Log Consolidator"


Geoff, IOW, I have never seen this problem:
>When you ssh to the same floatip - but now on another host - it sees that the
>mac has changed

I am able to non-interactively ssh successfully to the floating IP or
hostname for that IP no matter which member is hosting the package.
This assumes that you've done the appropriate key exchange for the floating
address. Tools like DSAU's csshsetup make the key exchange a painless
process.

sshd has many options, so it could be that there is some option that makes
this sensitive to MAC addresses. If that's the case, I'd like to know which
option that is. It's not a default option for example (at least not for the
11.23 systems I'm testing on).

[Geoff Wild]

Okay - reopening this thread - as here's what I get with your way Ralph - it works - but not clean.

Here's my known_hosts entries on the backup server:

nodea,10.10.10.16 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA7GapBwjmC+g5Qd5y3i/usDJLi6KZuYYmmj1Ixpf+L1wwNAnpqBu+1wd0exFoyu3H0+5+SCZL3g7SqzpBESk2EPa1jfKfbAHR4d0oj/h5K0td9wCL+s3yFOvJd/Qm6s5xw595IWGBt9RW/J6uSp74TVAVgLhqp/aFQnC4e/7sZXX=
floatip,10.10.22.31 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEA7GapBwjmC+g5Qd5y3i/usDJLi6KZuYYmmj1Ixpf+L1wwNAnpqBu+1wd0exFoyu3H0+5+SCZL3g7SqzpBESk2EPa1jfKfbAHR4d0oj/h5K0td9wCL+s3yFOvJd/Qm6s5xw595IWGBt9RW/J6uSp74TVAVgLhqp/aFQnC4e/7sZXX=
nodeb,10.10.10.28 ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEArXV3DYn1WoOU/4IH4ek/R36MDYnmjs2DlcCJuOoRnxpgrGQ0JVbUcc9w6hirJZycmm4JJiuBrY3+W9FZJbCOlalKKbIIQLu1fHVMLq6FVE9pd2YdNhiB7ar8jKC440v8A4on0AKcHcveiBaAqOPhwKz+ZG5L2wWZd9+Nq/WnVPO=

floatip is now in a package running on nodeb

Running the following from the backup server:


# ssh -o hostkeyalias=nodea -o StrictHostKeyChecking=no floatip date
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX.
Please contact your system administrator.
Add correct host key in /.ssh/known_hosts to get rid of this message.
Offending key in /.ssh/known_hosts:27
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.



Tue Jan 16 08:59:59 MST 2007


# ssh -o hostkeyalias=nodeb -o StrictHostKeyChecking=no floatip date
Warning: the RSA host key for 'nodeb' differs from the key for the IP address '10.10.22.31'
Offending key for IP in /.ssh/known_hosts:28
Matching host key in /.ssh/known_hosts:29


Tue Jan 16 09:08:52 MST 2007


This happens regardless if both nodes have the same keys in /opt/ssh/etc

If I don't do StrictHostKeyChecking=no it fails:

# ssh -o hostkeyalias=nodeb floatip date
Warning: the RSA host key for 'nodeb' differs from the key for the IP address '10.10.22.31'
Offending key for IP in /.ssh/known_hosts:28
Matching host key in /.ssh/known_hosts:29
Are you sure you want to continue connecting (yes/no)?

# ssh -o hostkeyalias=nodeb -o batchmode=yes floatip date
Host key verification failed.


Rgds...Geoff

[PeterWolfe]

Geoff,

I think if you start from scratch
it all works cleanly.

After copying all the keys from
/opt/ssh/etc/ cluster-wide, did you
restart sshd on all the members?

Then, on the client, delete
all the entries for the cluster members
and the floating IP's from known_hosts
and use the new key. I think that's sufficient.


I actually had deleted the /.ssh directories on all the members and the clients and started from scratch there as well (and used cssshetup to redistribute everything). That works with no hitches.

[Geoff Wild]

Ah Ha - restarting sshd was the culprit!

Rgds...Geoff

[Geoff Wild]

Yes, had to make /etc/opt/ssh/*key* the same on both nodes.

Had to restart sshd (very important!)

Command to run from backup server now works with no errors:
ssh -o HostKeyAlias=nodea floatip

Rgds...Geoff

[PeterWolfe]

You don't need the HostKeyAlias options at this point. The common cluster config makes it work.....