Managing host keys / global known hosts file

[Reiner Rottmann]

I'm wondering what you are using to maintain a ssh global known host file on your network.

What do you use to maintain your global known hosts in a feasable way?

Currently I am using a script that dumps the dns zone file and collects all CNAMEs and A records
for any given IP. Then the script queries the ssh host keys and dumps them as a global known host file
for DSA and RSA keys. After manually diff'ing the resulting file, I distribute it on the servers.

However in our infrastructure the servers get frequently (re-)installed and DNS aliases come and go.

Moreover this script cannot update the global known hosts file automatically because rouge servers
would be blessed that way.

As we are using an DNS appliance that does not support SSHFP records, it is rather difficult to maintain the hosts keys and we cannot force ssh StrictHostKeyChecking at the moment...

Do you know any remedy?

[Matti_Kurkela · Accepted solution]

> Moreover this script cannot update the global known hosts file automatically because rogue servers would be blessed that way.

Exactly. This is more of a procedural problem at the human level: if nobody coordinates server (re)installations and DNS alias management, there will be no way to programmatically tell legitimate servers from rogue ones. A purely technical, fully-automatic solution probably doesn't exist.

For example: you could set up an easy, well-documented procedure to follow in case of legitimate (re)installations. The procedure should be as automated as reasonably possible, and should be noticeably easier for the server administrator than doing the equivalent things manually one by one. The procedure should include authentication or some other way to verify legitimacy.

The idea is that the new procedure should be a *more reliable* and *easier* version of the tasks a server administrator needs to do when (re)installing a server or adding/changing/removing a DNS alias.

When you have a procedure that demonstrably works well and is found useful by at least some portion of your server admins, you may want to add something that automatically provides gentle reminders for those that aren't using the correct procedure.

For example, a variation of your key collection script could run nightly, compare the state of the network to a list of known-good servers and then send the list of hosts with new or changed host keys to a department-wide mailing list: "If you currently maintain one of these servers, please ".

The aim would be to generate some low-level peer pressure to minimize the amount of junk mail in everyone's inbox. As a side benefit, you're keeping people informed about server changes that might currently be in a "work-in-progress" state.

You might also consider re-thinking your network topology, to separate the "stable infrastructure" from the "chaotic lab/development/playground". This would be a major effort, but sometimes nothing else works for reining in the chaos. After you've established a degree of separation, you could then delegate the management of the SSH hostkeys of the chaotic portion to someone closer to the cause of the chaos.

MK