The best method for duplicating/coping a file system onto another

[Yogeeraj_1]

Dear all,

this question has always been pestering my mind.

What is the best and efficient way of copying a whole file system to another one - both found on the same server.

For example, i have the following setup where all the files in /d02 should be copied/moved to /d01:

SLX2:>bdf /d01 /d02
Filesystem kbytes used avail %used Mounted on
/dev/vgappinst/lv_appinst 20963328 3472 20632368 0% /d01
/dev/vg02/lvol1 17776640 11557928 6121600 65% /d02
SLX2:>

Possible options: (i hope i don't miss any of them)
a. cp -p -R /d02 /d012
b. cd /d01; tar -cvf D02.TAR /d02 then tar -xvf D02.TAR
c. cd /d02; find . -depth -print | cpio -pd /d01
d. dd if=/dev/vg02/lvol1 of=/dev/vgappinst/lv_appinst (??)
e. Backup/Restore using omniback

Which one gives the best performance?

Thank you for your replies.

Best Regards
Yogeeraj

[Rajeev Shukla]

I have tried the tar (b) method and never got any problems. All files are copied and never got any complaints. But i'll still look forward from people if if they hare tried other method.

Rajeev

[Ravi_8]

Hi,

In such situations i always opt for dd, which is faster than tar

[avsrini]

Hi Yogeeraj,
I feel cpio and dd is the fastest method than other options. I have to check this thread for more options if anybody have better options.

mirroring is one option but you have 2 diff vg's :(.

Srini.

[S.K. Chan]

The "dd" method would give you the best performance with a price of course because dd does an image copy without any error check. I usually use the "find" and piping it to "cpio" but the only caveat is cpio (also tar and pax) I think does not support files bigger than 2GB in size, so you got to be careful there.

[S.K. Chan]

The "dd" method would give you the best performance with a price of course because dd does an image copy without any error check. I usually use the "find" and piping it to "cpio" but the only caveat is cpio (also tar and pax) I think does not support files bigger than 2GB in size, so you got to be careful there. If that does not apply here then go with option c (if I'm doing it).

[S.K. Chan]

The "dd" method would give you the best performance with a price of course because dd does an image copy without any error check. I usually use the "find" and piping it to "cpio" but the only caveat is cpio (also tar and pax) I think does not support files bigger than 2GB in size, so you got to be careful there. If that does not apply here then go with option c (if I'm doing it).

[S.K. Chan]

The "dd" method would give you the best performance with a price of course because dd does an image copy without any error check. I usually use the "find" and piping it to "cpio" but the only caveat is cpio (also tar and pax) I think does not support files bigger than 2GB in size, so you got to be careful there. If that does not apply here then go with option c (if I'm doing it).

[S.K. Chan]

Oops .. no points please for the dups !

[Michael Tully]

I've to do this many times when I've had to things off one diska array onto another and find that 'find' in conjunction with 'cpio' works best for me.

create new filesystem and mount on say /new
# cd /oldfilesystem
# find . -xdev -print | cpio -pdumv /new

[Artyom Voronchihin]

If you have MirrorDisk/UX installed on your system, you can create mirror on necessary logical volumes using lvextend(1m) and split the mirror after synchronization using lvsplit(1m) command. So you will get two identical copies of the logical volumes.

[John Palmer]

Personally I'd use cpio as per Michael's reply - it's the only thing that I do use it for. Be careful if you have any largefiles (>2Gb) though as cpio doesn't handle them (or user/group ids > 60k) - see then man page.

dd is a possibility but there are a few things to be aware of:-
- both filesystems should be unmounted before running dd.
- both logical volumes ought to be the same size. If the target is larger then you can extendfs after dd.
- dd will copy ALL the data, not just the files in use so it may be slower if the filesystem isn't full.

One other possibility is to pipe vxdump into vxrestore (see the man pages).

Regards,
John

[Leif Halvarsson_2]

Hi,
If the filesystems is very large and you have fast tape devices (LTO) I would suggest using OmniBack rawdisk backup/restore. In other cases I always use find ... |cpio ... .

[Yogeeraj_1]

Thank you are for you replies!

I guess in my case, the only plausible options are:
1. cpio
2. Omniback backup/restore

dd option is a bit more complicated, as i already doubted it, because of the difference in sizes of the LVs as John mentioned.


The cpio option does not seem to be working properly:
---------------------------------------------------------------------
SLX2:d02>find . -depth -print | cpio -pd /d01
22357001 blocks

[took about 20 mins!!]

SLX2:d02>bdf /d01 /d02
Filesystem kbytes used avail %used Mounted on
/dev/vgappinst/lv_appinst 20963328 11565664 9250880 56% /d01
/dev/vg02/lvol1 17776640 11557928 6121600 65% /d02
SLX2:d02>cd /d01
SLX2:d01>ls
lost+found product
SLX2:d01>ll
total 0
drwxr-xr-x 2 root root 96 Jan 21 09:44 lost+found
drwxrwxr-x 5 ias902 oinstall 96 Jan 22 11:02 product
SLX2:d01>ll lost*
total 0
SLX2:d01>du -sk product
11523064 product
SLX2:d01>cd ../d02
SLX2:d02>du -sk product
11515516 product
SLX2:d02>
---------------------------------------------------------------------

The target (11523064) and the source (11515516) sizes do not match after the execution of the command.

Will now attempt the omniback option.

Will update this as soon as completed.

Best Regards
Yogeeraj

[Pete Randall]

Yogeeraj,

My standard answer would be dd piped to cpio but I have, on occasion, run into errors. In those cases I resort to the omniback method. If you already have the backup, it's relatively quick and it's definitely foolproof.

Pete

[John Palmer]

As I replied earlier, cpio is reliable provided you don't have files > 2Gb.

The difference in size reported by du can usually be attributed to directory sizes although I would have expected the source to be larger than the target.

I reckon your cpio copy was OK and expect that you'll get similar du results after your restore.

Regards,
John

[harry d brown jr]

Yogeeraj,

I just did a quick test, and I found "cp -Rp" to be faster and consume less resources:

# time cp -rp * /dbf4

real 4:32.3
user 0.1
sys 35.3

# time find . -xdev -print | time cpio -pdumv /dbf4

real 0.0
user 0.0
sys 0.0
/dbf4/DB2003-1PGN.ZIP
/dbf4/DB2003-1.pgn
/dbf4/unzipped.pgn
5249489 blocks

real 4:54.2
user 1.5
sys 43.3
# ll
total 5249522
-rw-rw-rw- 1 root sys 74481664 Dec 30 00:57 DB2003-1.pgn
-rw-r----- 1 root sys 558082162 Dec 30 16:46 DB2003-1PGN.ZIP
drwxr-xr-x 2 root root 96 Aug 14 2001 lost+found
-rw-rw-rw- 1 root sys 2055174021 Dec 30 16:54 unzipped.pgn
#


live free or die
harry

[Marco Santerre]

I'd have to agree with harry on this one, if you are looking for speed, your cp might very well do the job. In my case, though, when I can afford the time, I'd rather go with dd whne in comes to LV

[James R. Ferguson]

Hi:

While 'cp' may be faster than 'cpio', the later has the advantage that file creation timestamps can be preserved. My preference is to use 'cpio' to copy while filesystems in the original question's context.

Regards!

...JRF...

[RolandH]

But keep in mind cp will change the permissions on the file.

See following example:

# touch text
# ll text
-rw-rw-rw- 1 root sys 0 Jan 23 16:09 text
# chown www:sys text
# cp text text1
# ll text*
-rw-rw-rw- 1 www sys 0 Jan 23 16:09 text
-rw-rw-rw- 1 root sys 0 Jan 23 16:10 text1

So if you choose a copy method (dd,cpio,tar,cp ...etc)
for your lvol copy think about your permissions.

Roland

[Bill Hassell]

The fastest method is very dependent on the quantity of files and directories and how full the lvol might be. cpio will be very fast for thousands of files when compared to cp. For a small set of files, cp and cpio are fairly close, tar is just a bit behind. dd can be quite fast unless the bs= option is not specified and/or the filesystem isn't very full. dd copies empty space too. Using disk mirroring will be one of the slowest methods due to the copy+verify atomic (aka, exclusive) method used to ensure accuracy. tape solutions are always slower since you need to access the tape twice, once to write and again to read.

As far as the size difference between the source and destination, *only* dd will preserve the disk usage. Anything that reads the files and copies them to a new location will cause the total usage to be larger or smaller than the original.

Here's why:

Smaller disk usage:
When lots of files are added to a directory (which itself is a file), the directory is increased in size to accomodate all the new names. However, if you add 10,000 files to a directry and then remove them, the directory does NOT shrink. It remains the same with just a bunch of empty entries. When the directory is 'copied', it is actually created as a 96 byte file and then increased in sized if needed. Lots of directories that used to have lots of files will be recreated much smaller than the source.

Larger disk usage:
Two words: sparse files. A sparse file is one in which a few random records (perhaps the first and the 10,000th) are written and then closed. The occupied space is two records but read the file serially (as happens in a backup or copy) and the filesystem fills in the all the missing records with zeros. Databases are commonly created as sparse files. So when the file is copied, it will be much larger. Create your own sparse file with:

dd if=/etc/issue of=/var/tmp/sparse bs=2048k seek=1

where you will see the original file is just a few dozen bytes, the result with ls -l or wc -c shows a 2 meg file, but du will show the file as occupying just a bit more than the original /etc/issue file. A cp of the file will create a new file that is the same size (using ls -l or wc -c) but du will now show a MUCH larger size than the original file.

Another common file that is often sparse is a core file from a crashed program. cp the core file and it is often larger than the original.