Tar files and directories to another server

[Coolmar]

We have to move some files and directories to another server using:

tar cf - * | ssh server "cd /copy to dir ; tar xvf -"

The problem is that it is very very slow. Usually it is pretty quick, but now it is slow. The network is fine as we can use the cp -Rp command and it is substantially less time. The only thing that I can think of is that the directory structure is very very deep (up to 17 deep which is why we are moving everything). Could that cause the slowness? Why wouldn't it be slow then with cp? Any other ideas or switches to make it faster?

[Ivan Ferreira]

Why do you use tar instead of scp?

Use for example:

scp -rp * remotehost:/copy_to_dir

[Coolmar]

scp doesn't get links....I have just found tar to be a little more reliable.

[Calandrello]

Friend
which the volume of data that you this trying to copy? exist many small archives? this degrades the performance. it tries to use a NFS I believe that he is more fast.

[Coolmar]

It is alot of data, but what it takes cp 17 minutes, it is taking tar 2 hours.

[MarkSyder · Accepted solution]

tar has a reputation for being slow. I would use cpio.

cd /old_filesystem
find . â xdev â depth â print|cpio â pmd /new_filesystem

Mark Syder (like the drink but spelt different)

[Coolmar]

Thanks Mark...what character(s) are missing in your answer? Instead it put boxes and "a"s

[Coolmar]

oh and can cpio be used across servers?

[MarkSyder]

That's what I get for copying and pasting from a web page on a unix server!

find . -xdev -depth -print|cpio -pmd /new_filesystem

Mark

[MarkSyder]

Yes, cpio can be used across servers, provided you are using nfs.

Mark

[TwoProc]

A) from source machine to dest machine
- on source machine

>cd /sourcedir
>tar cvf - . | ssh destmachine "cd /destdir ; tar xvf -"

B) from dest machine - getting data from the source machine
- on destination machine
>cd /destdir
>ssh sourcemachine "cd /sourcedir; tar cvf - . " | tar xvf -

FWIW - I prefer using tar to move things also, it preserves file ownership, symbolic links, etc.

[Coolmar]

No we aren't using NFS....it would have to be with SSH. I saw the following command in another thread...you think this will work? It says though that it will just move the directories and not the data....but I presume that is jsut with switches.

# find /source -type d | cpio -ov | remsh server " cd /copy ; cpio -idvum "

What switches can I use to copy the directories and data and subdirectories?

Thanks

[TwoProc]

The command you put looks correct for cpio. Cpio has the advantage of being able to move special files as well (the "x" option), but it is usually slower than ftp.

Re: directories and subdirectories - that's what the "d" in the extract command is for, and you already have it. You may have to add that option to the first "cpio" as well as the extraction part to make it fly.

[Jean-Philippe HENRY_1]

Hello,

Your command may be improved inserting gzip at the source server and gunzip at the target.

The v option of tar on the target can also increase the execution time.

I suggest you to try this :

tar cf - * | gzip -c | ssh server "cd /copy_to_dir ; gunzip -c | tar xf -"

Regards,

JPH