Operating System - Linux
1748163 Members
3613 Online
108758 Solutions
New Discussion

Tar taking 2 hrs for filesystem with deep directory structure

 
SOLVED
Go to solution
chindi
Respected Contributor

Tar taking 2 hrs for filesystem with deep directory structure

Hi ,

We have a VM in which we want do local copy of 6.7GB .

Taking more than 2 hrs to copy .

 

What is the fastest way to copy data from one filesystem to another ?

Have tried throught tar -cvf its taking almost 1.5 hrs.

 

 

 

7 REPLIES 7
Steven Schweda
Honored Contributor

Re: Tar taking 2 hrs for filesystem with deep directory structure

> [...] with deep directory structure

   Define "deep".

> Taking more than 2 hrs to copy .

   When you "copy" _how_, exactly?

   As usual, showing actual commands with their actual output can be
more helpful than vague descriptions or interpretations.

> Have tried throught tar -cvf its taking almost 1.5 hrs.

   I assume that that's only half (or less) of your actual command.  I
normally use a "tar" pipeline for such jobs.  I don't know if it affects
the speed much, but I would not use "v" (on either side of the "tar"
pipeline), _especially_ for a large job.  If there's some kind of error,
I'd like to see the message, not have it get lost in voluminous "tar v"
output.

chindi
Respected Contributor

Re: Tar taking 2 hrs for filesystem with deep directory structure

Hi Steven ,

We need  /backup of 6.7GB to be copied to /temp_backup on same VM ( Redhat linux 6.1 )

cd /backup

tar -cvf /temp_backup/backup.tar  *

 

 

 

Dennis Handly
Acclaimed Contributor

Re: tar taking 2 hrs for filesystem with deep directory structure

>tar -cvf /temp_backup/backup.tar  *

 

By using a tar pipeline, you don't have to write to the 6.7 GB tarfile and then read back from it.  You just send that data through the pipe.

Assuming you are later doing a "tar -xf"

chindi
Respected Contributor

Re: tar taking 2 hrs for filesystem with deep directory structure

Hi Dennis ,

 

Tar pipeline ? Can you please share the same

Dennis Handly
Acclaimed Contributor
Solution

Re: tar taking 2 hrs for filesystem with deep directory structure

>tar pipeline?

 

cd /temp_backup

tar -cf - -C /backup . | tar -xf -

Steven Schweda
Honored Contributor

Re: tar taking 2 hrs for filesystem with deep directory structure

> We need  /backup of 6.7GB to be copied to /temp_backup on same VM
> [...]

   Do you want a giant "tar" file at the destination, or do you want a
file tree like the source file tree?

> cd /backup
> tar -cvf /temp_backup/backup.tar  *

   That will give you the giant "tar" file.  Again, I'd omit "v".  Also,
"*" will not catch "hidden" files (".login", and so on).  If I wanted a
giant "tar" file, I'd do:

      ( cd /backup ; tar cf /temp_backup/backup.tar . )

   If the bottleneck in your scheme is the disk-writing speed, and your
CPU is speedy. then you might save some time by compressing the data.
For example:

      ( cd /backup ; tar cf - . | gzip -c > /temp_backup/backup.tgz )

 

   On a modern GNU/Linx system, GNU "tar" probably has built-in
compression options, but using an explicit compression program in the
pipeline (as shown above) works on systems with older and/or less
capable "tar" programs.

      http://www.gnu.org/software/tar/manual/html_node/gzip.html#SEC134

> Tar pipeline ? Can you please share the same

   Your Forum/Web search found nothing useful?  One example (of many):

      http://h30499.www3.hp.com/t5/x/x/m-p/3966579

   Substituting "&&" for ";" in those example pipelines can improve
behavior when errors occur.

TwoProc
Honored Contributor

Re: tar taking 2 hrs for filesystem with deep directory structure

Since you're on the same machine - cpio might be your friend.

Run this as root:

 

cd <sourcedirectory>

find . | cpio -pdmvu <destdirectory>

 

ex: sourcedirectory = /src, destdirectory = /dest

 

$> cd /src

$> find . | cpio -pdmvu /dest

 

-- This might run faster than tar - I would suspect that it would.

We are the people our parents warned us about --Jimmy Buffett