root password was changed now rsync does not work

[Neil Alertsen]

Hello,

We changed our root password on both of our servers. Now, rsync is failing to authenticate.

I did not install the rsync software so I am not familiar with configuring it. Is the password in some file that needs updated? Will rsync need to be re-compiled?

Thank you for any help you can give me.

Neil

[James R. Ferguson]

Hi Neil:

This sounds like you have used:

--password-file=FILE

Look at your 'rsync' script parameters. If you see the above, change the password in 'FILE' appropriately.

Regards!

...JRF...

[Neil Alertsen]

Hi JRF,

Please pardon my ignorance but where would the rsync parameters be defined? Are they defined in a file or at the command line?

The command that we are attempting to run is:
/usr/local/bin/rsync -av --delete -e "ssh" /lvol2b/production/ ihcc2:/datatel/co
ll18/production >>/var/adm/syslog/rsync.log

Thank you for your lightning quick response.

Neil

[James R. Ferguson]

Hi (again) Neil:

OK, so you are using 'rsync' over 'ssh' and not a yukky password file. That's good. Beside the root password change (which should not have affected this) what else was changed? Perhaps did someone due a "security" adjustment based on an auditor's "mandate"?

Regards!

....JRF...

[Jim Walls]

Two things you might do:

Look for ssh errors in the remote server's /var/adm/syslog/syslog.log (or possibly local0.log). That might give you a clue.

Try modifying the rsync call as follows to get more information and post the output here.

rsync -av -e "ssh -v" ... ... ...

Placement of the second "v" between the quotes is important!

I am assuming you use default ssh keys, which is why you do not usually need to supply a password.

[Neil Alertsen]

About that same time, I disabled the root login via telnet from anywhere but the console. I also disabled the SSH port on the source box.

I just tested the "-v" option and the rsync was successful. I was required to type in the root password to make it work.

This is part of what I saw:
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Trying private key: /.ssh/identity
debug1: Trying private key: /.ssh/id_rsa
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey,password,keyboard-interactive
debug1: Trying private key: /.ssh/id_dsa
debug1: Next authentication method: keyboard-interactive
Password:
debug1: Authentication succeeded (keyboard-interactive).

Any ideas?

Neil

[Jim Walls]

What else did you change? ;O)

Do you see any messages in the remote server's syslog?

Please post the output from:

ls -al ~root/.ssh

from both servers.

[Neil Alertsen]

This started between July/August. I cannot recall any other changes at this time but I'll continue to update you as I remember or find my more of my notes.

The source has:
# ls -al ~root/.ssh
total 64
drwx------ 2 root sys 8192 Jul 14 15:30 .
drwxr-xr-x 26 root root 8192 Sep 14 18:00 ..
-rw------- 1 root sys 1675 Jul 14 15:30 id_rsa
-rw------- 1 root sys 1424 Apr 29 14:53 known_hosts
#

The destination has:
# ls -al ~root/.ssh
total 64
drwx------ 2 root sys 96 Sep 15 2008 .
drwxr-xr-x 25 root root 8192 Sep 12 05:27 ..
-rw-r--r-- 1 root sys 1111 Sep 15 2008 authorized_keys
-rw------- 1 root sys 1111 Sep 10 2008 ihcc-250-rsync-key.pub
-rw------- 1 root sys 1111 Sep 10 2008 temp
#

Neil

[Neil Alertsen]

Here is the syslog errors for the failed atttmpts:

Sep 14 17:59:17 ihcc2 sshd[19351]: error: PAM: Authentication failed for root from ihcc
Sep 14 17:59:15 ihcc2 sshd[19349]: Failed password for root from 192.168.193.250
port 54141 ssh2
Sep 14 17:59:21 ihcc2 sshd[19351]: Failed keyboard-interactive/pam for root from
192.168.193.250 port 54147 ssh2
Sep 14 17:59:21 ihcc2 sshd[19351]: Failed password for root from 192.168.193.250
port 54147 ssh2
Sep 14 17:59:21 ihcc2 sshd[19351]: error: PAM: Authentication failed for root from ihcc
Sep 14 18:00:03 ihcc2 above message repeats 2 times
#

Neil

[Jim Walls]

The permissions look OK.

You haven't daid if there are any messages in the syslog.

I would expect to find something like...
Sep 15 11:16:21 svrasdf sshd[4320]: Accepted publickey for root from 10.10.10.193 port 50854 ssh2

If it worked... and summat else when it fails on the publickey; followed by acceptance of the password.

[Jim Walls]

I think the public key stored in the remote server may be invalid (or may not be installed)

Did you have to recover something from a backup on the remote server?

The date of the LOCAL server's private key (~root/.ssh/id_rsa) is 14 July this year.

However, the authorized_keys file on the REMOTE server is dated Sep 2008 (indeed, all the ssh files are at least two years old).

You should probably generate a new private key on the local server, using ssh-keygen and append the resulting id_rsa.pub file to the authirized_keys file on the remote server. You will need to append the same key to every other server you want to access using passwordless SSH.

[Neil Alertsen]

The last time it accepted a publickey for root was at 15:17:48 on July 14.

Jul 14 15:17:48 ihcc2 sshd[2049]: Accepted publickey for root from 192.168.193.250 port 58579 ssh2

There are no publickey lines in the syslog on the destination server, from where the rsync failed on July 14 to today.

On Jul 14, 16:00, it starts failing on the keyboard-interactive only:
Jul 14 16:00:23 ihcc2 sshd[3240]: SSH: Server;Ltype: Version;Remote: 192.168.193.250-59946;Protocol: 2.0;Client: OpenSSH_5.3p1+sftpf
ilecontrol-v1.3-hpn13v5
Jul 14 16:00:26 ihcc2 sshd[3240]: error: PAM: Authentication failed for root from ihcc
Jul 14 16:00:30 ihcc2 sshd[3240]: Failed keyboard-interactive/pam for root from 192.168.193.250 port 59946 ssh2
Jul 14 16:00:30 ihcc2 sshd[3240]: Failed password for root from 192.168.193.250 port 59946 ssh2

[Jim Walls · Accepted solution]

Those timings fit nicely with the timestamp of the local server's private key, which has a timestamp of 14 Jul 15:30.

The source has:
# ls -al ~root/.ssh
total 64
drwx------ 2 root sys 8192 Jul 14 15:30 .
drwxr-xr-x 26 root root 8192 Sep 14 18:00 ..
-rw------- 1 root sys 1675 Jul 14 15:30 id_rsa
-rw------- 1 root sys 1424 Apr 29 14:53 known_hosts
#

The last "good" access was at:
Jul 14 15:17:48 ihcc2 sshd[2049]: Accepted publickey for root from 192.168.193.250 port 58579 ssh2

And the first failed access at:
On Jul 14, 16:00, it starts failing on the keyboard-interactive only:

If you can't find the corresponding public key (id_rsa.pub) for 14 July then you had better generate a new key pair.

[Neil Alertsen]

Here is the syslog with two unsuccessful attempts and one successful attempt in the middle.

Neil

[Neil Alertsen]

Ok, Jim. I think I have a backup of that before it happened.

I'll check and let you know.

Thank you very much for your help. I am glad you were here today.

Neil

[Neil Alertsen]

Jim,

I did not have a id_rsa file but I did have a id_dsa.pub and it worked beautifully.

Thank you very much in your assistance in getting this fixed.

Neil

[Neil Alertsen]

I restored the file id_dsa.pub from a back up from before the rsync broke.

[Jim Walls]

But that is a Keyboard/Interactive success - not a public/private key exchange! It means someone was prompted for a password and then typed in the correct one.

If a public/private key is used, one gets something like this:
Sep 15 10:40:00 pocuast2 sshd[2938]: SSH: Server;Ltype: Version;Remote: 10.81.105.17-49187;Protocol: 2.0;Client: OpenSSH_5.1
Sep 15 10:40:00 pocuast2 sshd[2938]: Accepted publickey for root from 10.81.105.17 port 49187 ssh2
Sep 15 12:19:30 pocuast2 sshd[2938]: SSH: Server;LType: Throughput;Remote: 10.81.105.17-49187;IN: 96208;OUT: 17952;Duration: 5970.2;tPut_in: 16.1;tPut_out: 3.0

If you need to run this rsync periodically using cron on the source (or local) server, you must use a private/public key-pair that does not require a pass-phrase - otherwise it will fail because there is no way to supply the authentication password in a non-interactive environment such as the cron.

If, on the other hand, you run it manually, then you can supply the necessary password but if a valid key-pair was present, that is tried first; only if the key exchange fails will a password be requested.

[Jim Walls]

No worries... I seem to have got a bit out of sync in the interaction here ;O)