rsync / ssh

[Marty Metras]

Here is what is happening.
I use rsync with ssh to keep files updated on my standby server. been doing this for a couple years with no problems. The last week or so I noticed that it starts out moving data at the rate of 120kb and then it drops to 10-12 bytes. (not kb but bytes) I can use sftp to jam data thry to the standserver and it will run at full speed for ever.
I tried using the same rsync commands from the command line and it also get stuck.
SOme times it will move 200MB and other times 2-3GB before rsync locks.
I find no message is syslog it just stops doing any thing and hangs up.

I think my brain is starting to freeze up on this. Can you point me in a direction to start looking for a cause.

Marty

[RAC_1]

Are there any network problems on the system??. Do you experience same network slowdown problems with rcp/scp??

How is the ping/telnet response when this happens??

Anil

[Steven E. Protter]

Network problems would effect all protocols and sftp does not seem effected.

I don't know anything about rsynch, but the configuration of that tool or patching is where I'd look first.

SEP

[Denver Osborn]

To help troubleshoot, provide more options with your ssh syntax for rsync. Make your ssh syntax to include "ssh -vvv" for debug output. This will show us what's happening on the client. You can also startup sshd on the standby server with "-dd" for debug output.

Assuming your ssh and rsync syntax hasn't changed any, look at ~/.ssh/ssh_config or /opt/ssh/etc/sshd_config and ssh_config files to see recently modified. Maybe something there will help. Also check for any sshd errors in the syslog.log on the standby node.

post a reply and attach the ssh debug output if you need help looking through it.

are these nodes on the same subnet?

-denver

[A. Clay Stephenson]

By any chance are you using -c or --checksum. I've seen cases where that can be very slow especially on large files. It would also be useful to add the --stats and --verbose option to see if this is related to a particular file.

[Sandman!]

try using rsync w/o compression by removing "-z" as one of the options to rsync.

man rsync for details

cheers!

[Stuart Browne]

What is the link between the two servers? Are they on the same LAN segment or are they at separate offices connected by some sort of WAN link?

I noticed something like this happening once when the number of files involved reached a critical point (but this was on another OS, and the number of files was up around the 400,000 mark). What size of data-set are you using? (both number of files and amount of data?). THere's a check-sum-block-size of which you might be able to tweak if it's the latter.

[Marty Metras]

The command I am using looks like this.
rsync -azv -e ssh /filepath/filenam.* hostname:/path/to/where/to/put/files/
bandwith is 125K (t1) different network/subnet.
After checking most of the thing you talked about her is what I found out.
root has the problem mostly.
been using the same command for a very long time.
Just found out yesterday that the network guy has been locking down the securty between here and the co-loaction. He is very good at what he does but some times he is too good. We want the trafaic to only go to the co-location and not back. He may have locked it down too tight.
He kind of has a tunnel in a tunnel in a tunnel and an automatic switch over if the connection fails so it may be having commucating back when using rsync.
sftp runs at full bandwidth for 4 hours with out missing a beat. rsync gets sucks intermintly. This only happens when I am pumping 80gb. The other normal rsynd for the database archives. 2-300 megs a day never fails.
for now I wait for Bob to tell me more about his security.
Thanks,
Marty

[Marty Metras]

I have checked and recheck this issue using most all for you you have suggested.
Nothing seamed to make any difference or shined light on the cause.
Now I can't make it not work.
It had to be some thing happening on the network that was out of my control.
I don't like it when a plan comes together and starts working when the was no plan to start with.
I'll kist set this one aside and see what happens.
Thanks,
Marty