topic Re: tar taking 2 hrs for filesystem with deep directory structure in Operating System - Linux

Tar taking 2 hrs for filesystem with deep directory structure

chindi — Fri, 21 Aug 2015 13:13:29 GMT

Hi ,

We have a VM in which we want do local copy of 6.7GB .

Taking more than 2 hrs to copy .

What is the fastest way to copy data from one filesystem to another ?

Have tried throught tar -cvf its taking almost 1.5 hrs.

Re: Tar taking 2 hrs for filesystem with deep directory structure

Steven Schweda — Sat, 22 Aug 2015 00:57:52 GMT

> [...] with deep directory structure

   Define "deep".

> Taking more than 2 hrs to copy .

   When you "copy" _how_, exactly?

   As usual, showing actual commands with their actual output can be
more helpful than vague descriptions or interpretations.

> Have tried throught tar -cvf its taking almost 1.5 hrs.

   I assume that that's only half (or less) of your actual command. I
normally use a "tar" pipeline for such jobs. I don't know if it affects
the speed much, but I would not use "v" (on either side of the "tar"
pipeline), _especially_ for a large job. If there's some kind of error,
I'd like to see the message, not have it get lost in voluminous "tar v"
output.

Re: Tar taking 2 hrs for filesystem with deep directory structure

chindi — Mon, 24 Aug 2015 04:56:05 GMT

Hi Steven ,

We need /backup of 6.7GB to be copied to /temp_backup on same VM ( Redhat linux 6.1 )

cd /backup

tar -cvf /temp_backup/backup.tar *

Re: tar taking 2 hrs for filesystem with deep directory structure

Dennis Handly — Mon, 24 Aug 2015 05:09:17 GMT

>tar -cvf /temp_backup/backup.tar *

By using a tar pipeline, you don't have to write to the 6.7 GB tarfile and then read back from it. You just send that data through the pipe.

Assuming you are later doing a "tar -xf"

Re: tar taking 2 hrs for filesystem with deep directory structure

chindi — Mon, 24 Aug 2015 05:35:58 GMT

Hi Dennis ,

Tar pipeline ? Can you please share the same

Re: tar taking 2 hrs for filesystem with deep directory structure

Dennis Handly — Mon, 24 Aug 2015 17:15:34 GMT

>tar pipeline?

cd /temp_backup

tar -cf - -C /backup . | tar -xf -

Re: tar taking 2 hrs for filesystem with deep directory structure

Steven Schweda — Mon, 24 Aug 2015 17:31:06 GMT

> We need /backup of 6.7GB to be copied to /temp_backup on same VM
> [...]

   Do you want a giant "tar" file at the destination, or do you want a
file tree like the source file tree?

> cd /backup
> tar -cvf /temp_backup/backup.tar *

   That will give you the giant "tar" file. Again, I'd omit "v". Also,
"*" will not catch "hidden" files (".login", and so on). If I wanted a
giant "tar" file, I'd do:

      ( cd /backup ; tar cf /temp_backup/backup.tar . )

   If the bottleneck in your scheme is the disk-writing speed, and your
CPU is speedy. then you might save some time by compressing the data.
For example:

      ( cd /backup ; tar cf - . | gzip -c > /temp_backup/backup.tgz )

   On a modern GNU/Linx system, GNU "tar" probably has built-in
compression options, but using an explicit compression program in the
pipeline (as shown above) works on systems with older and/or less
capable "tar" programs.

      http://www.gnu.org/software/tar/manual/html_node/gzip.html#SEC134

> Tar pipeline ? Can you please share the same

   Your Forum/Web search found nothing useful? One example (of many):

      http://h30499.www3.hp.com/t5/x/x/m-p/3966579

   Substituting "&&" for ";" in those example pipelines can improve
behavior when errors occur.

Re: tar taking 2 hrs for filesystem with deep directory structure

TwoProc — Tue, 08 Sep 2015 19:01:04 GMT

Since you're on the same machine - cpio might be your friend.

Run this as root:

cd <sourcedirectory>

find . | cpio -pdmvu <destdirectory>

ex: sourcedirectory = /src, destdirectory = /dest

$> cd /src

$> find . | cpio -pdmvu /dest

-- This might run faster than tar - I would suspect that it would.