secure shell durring service guard package failover

[Paul Wasik]

Recently we found that while a service guard package was failed over to the 'other' cluster node, the secure shell key was invalidated (due to new MAAC address) and ssh access was denied.

has anybody else encountered this and what automated solutions were implemented.

[Jeff_Traigle]

In sshd_config, there is a setting for IgnoreUserKnownHosts. By default, this is "no". Changing it to "yes" should avoid the problem.

[Jeff_Traigle]

Actually, that may have been the solution to another problem I saw in my last job when the host key got changed during a Secure Shell upgrade on a few systems. (I actually just had the few people who noticed remove the affected entries from their known_hosts files.)

Are the /opt/ssh/etc/ssh_host_key files the same on both nodes in the cluster? In our clusters, we have them the same and I haven't seen any SSH denials switching packages from one node to another.

[Paul Wasik]

Taking a look at the files (on cluster nodes) does reveal there are differences using both 'diff' and 'cmp'. this file also appears to differ between any two servers.

I'll get through a test as soon as possible but any further info or suggestions would be helpfull.

[Olivier Masse]

I've had the same problem, as when you connect to the HA address the host key changes from the cached one and it is considered by the ssh client as a security breach. Very tricky if you have automated tasks.

There are three workarounds to this that I know of, from the simplest (less secure) to the more complicated (more secure).

1. Put the same host key on both your servers. Not very good in terms of security.

2. Add manually the host keys for all your servers in the .ssh/known_hosts file of the user which initiates the connection. You can put more than one host key for each IP address. This does not work with all SSH clients, but OpenSSH-based clients did work last time I tried.

3. Configure an alternate SSH server, with a separate sshd_config file, that listens on another port than port 22. Associate this server with a particular host key that you propagate to each node of your cluster. If a user connects to port 22, he gets the host key of the "real" server. But on the alternate port, the hostkey of the HA service.

Olivier

[Logu_2]

What was the exact error message you received. Do you mean to say that the server authentication failed? Or you have problem with public keyauthentication?

-logu

[Paul Wasik]

Here is the exact message:

root@somehost: ssh service_pkg
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
6a:55:43:8a:ca:a6:a5:4a:1a:aa:9a:74:2a:01:a6:aa.
Please contact your system administrator.
Add correct host key in /ts/root/.ssh/known_hosts to get rid of this message.
Offending key in /ts/root/.ssh/known_hosts:36
RSA host key for service_pkg has changed and you have requested strict checking.
Host key verification failed.

my temp solution was to delete the service_pkg entry from the named file (known_hosts) and renegotiate with a ssh over to the service guard package. this will work untill the next failover.

[Doug O'Leary]

Hey;

There's actually a pretty slick work around for this exact problem. I'm assuming you're using openssh.

Let me restate the problem. If I'm not accurate in what I'm trying to fix, you can (and probably should) ignore the rest of the post.

When the package (pkg) is running on the primary node (PM), you can ssh to ${pkg}'s IP address w/o issue.

If ${pkg} fails over to the adoptive node (AD), you get an error message about failed Host checking when you ssh to ${pkg}'s IP address.

The fix is to use host key aliasing. Details can be found at:

http://www.olearycomputers.com/ll/ssh_cluster.html

HTH;

Doug

[Paul Wasik]

After researching through the above responses, I tried something similar to Masse #2 and O'leary suggestions.

first of all, this is what I get when I look at the ssh versioning:
root@: ssh -V
OpenSSH_3.9, OpenSSL 0.9.7d 17 Mar 2004
HP-UX Secure Shell-A.03.91.002, HP-UX Secure Shell version

second, I have a two node cluster with 2 packages; node1, node2, package1, package2. root is the ID that submits all of the ssh connections.

the solution I chose is here:
1. within ~root/.ssh/known_hosts, I removed the key's specific to package1.
2. node1 and node2 key entries were updated with the package info. for example here are the entries prior to change:

node1,123.456.78.90 ssh-rsa {KEY}
node2,123.456.78.91 ssh-rsa {KEY}
package1,123.456.78.92 ssh-rsa {KEY}
package2,123.456.78.93 ssh-rsa {KEY}

the package1 (haven't altered package2 yet) entry was removed and the node1, node2 key was updated with the package1 info:

node1,123.456.78.90,package1,123.456.78.92 ssh-rsa {KEY}
node2,123.456.78.91,package1,123.456.78.92 ssh-rsa {KEY}
package2,123.456.78.93 ssh-rsa {KEY}

subsequent ssh connections durring failovers worked for package1 when hosted on node1 or node2.

This seems really simple and far too easy ... please scrutinize this for validity and potential issues. please advise.

[Paul Wasik]

see my last post