copy a filesystem best solution (local host or remote host)

[Billa-User]

what's the best way to log the copy command and copy process in a external logfile. i will put copy command's in a script.

cd /old_fs
SHOW_FS_LOGFILE=/tmp/copy_$( date +%Y%m%d_%H%M%S ).log

# logging in a external logfile

# option 1
exec 4>&1
tee ${SHOW_FS_LOGFILE} >&4 |&
exec 1>&p 2>&1

find . ! -name lost+found -depth -print | cpio -padlmuv /new_fs

# option 2
find . ! -name lost+found -depth -print | cpio -padlmuv /new_fs 2>&1 > ${SHOW_FS_LOGFILE}

regards

[Bill Hassell]

dd will be by far the fastest and most accurate copy, given the restrictions you mentioned above (umount, VG version, etc). Using the raw lvol device and a 1MB block size, you will be limited only by disk speeds. Using fbackup or tar or even cp will mean that sparse files will be expanded so that a a multi-million record file with just two defined records will be expanded into fully occupied multi-million records. rsync will try to guess about sparse files but the result is guaranteed to be a sector-by-sector copy, only a functional copy. Sparse files are very common with database files. And while only fbackup handles ACLs, dd doesn't care about anything unique in the filesystem including large files -- it's just a raw block copy. And especially for small files, the overhead in the system to open and close each file can be a big slowdown for both source and destination.

[Michael Steele_2]

"..what's the best way to log the copy command and copy process in a external logfile..."?

################################

>OUTFILE
echo command | tee -a OUTFILE 2>&1
command | tee -a OUTFILE 2>&1

[Bob_Vance]

@Bill

Does the -s option on 'frecover' really do anything ?


bv

[Steven Schweda]

> [...] Using fbackup or tar or even cp will
> mean that sparse files will be expanded
> [...]

That could depend on whose "tar" you use, and
how.

http://www.gnu.org/software/tar/manual/html_node/sparse.html

> [...] dd [...] -- it's just a raw block
> copy. [...]

So it doesn't do any defragmentation or other
tidying, either.

As usual, many things are possible, and "best
solution" is seldom a well-defined entity.

[Bill Hassell · Accepted solution]

@Bob > Does the -s option on 'frecover' really do anything ?

All I know is what I read in the funny papers -- aka man pages. ;-)

So I tested it:

Create a sparse file:
# dd if=/etc/issue of=/tmp/sparse bs=2048k seek=1

Demonstrate that it is sparse (ll versus du)
# ll sparse
-rw------- 1 root sys 2097703 May 13 12:28 sparse

# du sparse
16 sparse

Yep, it is a sparse file. Now copy it:
# cp sparse sparse.cp
# du sparse sparse.cp
16 sparse
4112 sparse.cp

Yep, that's what I expected -- cp expanded the file. Note that it is the same contents -- cksum agrees:

1925132811 2097703 sparse
1925132811 2097703 sparse.cp

And that's because cksum reads the file serially and gets the records full of nulls.

Backup the file:
# fbackup -i sparse -f sparse.fb
fbackup(1004): session begins on Fri May 13 12:36:56 2011
fbackup(3024): writing volume 1 to the output file sparse.fb
fbackup(3055): total file blocks read for backup: 4098
fbackup(3056): total blocks written to output file sparse.fb: 4112

Recover normally:
# frecover -x -i sparse -f sparse.fb

Check the size with du:
# du sparse sparse.orig
16 sparse
16 sparse.orig

Now try -s with frecover:
# frecover -x -i sparse -f sparse.fb

Check the size with du:
# du sparse sparse.orig
16 sparse
16 sparse.orig

No difference. So the answer (experimentally) is that -s does nothing, but frecover always seems to preserve the sparse nature of a file. (or more likely, looks for repeated null records and does lseeks to move the record pointer but stores nothing on the disk).

[Bob_Vance]

Actually, I was trying similar when your response came in,
so you saved me some work!


However, I decided to test with larger files -- I wanted t obviate any minimum size issues because of the heasder, etc.


It appears to WORK, after a fashion!

((I noticed that your second frecover was missing a -s,
but that may not have reflected what you actually executed.
))


## dd if=/stand/vmunix of=/tmp/sparse bs=10240k seek=20

## nd /tmp

## ll sparse
-rw-r--r-- 1 root sys 237744936 May 13 13:14 sparse


## fbackup -i sparse -f sparse.fb

## du sparse*
54752 sparse
464368 sparse.fb

## mv sparse sparse.O

## frecover -x -i sparse -f sparse.fb

## du sparse*
472448 sparse
54752 sparse.O
464368 sparse.fb

## mv sparse sparse.fromfb

## frecover -s -x -i sparse -f sparse.fb

## du sparse*
86016 sparse
54752 sparse.O
464368 sparse.fb
472448 sparse.fromfb

## (ll sparse ; du -sk sparse)
-rw-r--r-- 1 root sys 237744936 May 13 13:14 sparse
27384 sparse


So, the -s recovered file is DEFINITELY a sparse file, but not as small as the original.

Checksums still match:

## cksum sparse*
2273740371 237744936 sparse
2273740371 237744936 sparse.O
2292610191 237752320 sparse.fb
2273740371 237744936 sparse.fromfb


But, it tries its best, I'm sure as you say skiping zero-data blocks.

bv

[Billa-User]

@ Bob Vance: umount /mount; mount -r /mount
@ for read-only mode.

i am writing now a script, which should include all the input's of this thread like

./copy_cpio.sh -s/src_mount -d/dst_mount

but when i want to mount a filesystem, which doesn't exist in fstab (example:service-guard filesystem), i need the lvol for mounting ?
so

./copy_cpio.sh -s/src_mount -d/dst_mount -l/dev/vgcpio/lvol

or how can i detect the best way of a mount filesystem ? bdf ?

@@ sparse file

thx for the useful,great informations, i have to write a script to detect "sparse file" ? or does a command exist like "find ... -sparse " :-) ?

@@ cksum
a cksum for source-directory (like "/src_mount") and destination-directory ( like "/dst_mount" ) don't exist ?
cksum /src_mount
cksum /dst_mount

how can i check, if dest filesystem is the same like src filesystem ?

regard

[James R. Ferguson]

HI:

> but when i want to mount a filesystem, which doesn't exist in fstab (example:service-guard filesystem), i need the lvol for mounting ?

Correct.

> or how can i detect the best way of a mount filesystem ? bdf ?

That assumes that the filesystem is already mounted.

> does a command exist like "find ... -sparse " :-) ?

No. As you know now, if you 'cp' a file and compare the number of disk blocks for the original versus the copy, a sparse file will be inflated. that is, the copy will show a larger number of disk blocks.

> a cksum for source-directory (like "/src_mount") and destination-directory ( like "/dst_mount" ) don't exist ?

Correct. You will have to roll-your-own. One way is to checksum every file in the source directory; independently checksum every file in the destination directory; write the results of each to file and compare the two files with a 'diff'.

> A simple count of the number of files (short of a cksum as I suggested above) is probably sufficient given that the replication mechanism returned a good exit code.

Do *not* be mislead by *directory* size differences between the source and target directories. A directory for which a large number of files *once* existed but for which many were deleted, will shrink in size when copied. That is, disk blocks once allocated for a directory are never returned. Rather they remain to be re-used.

Regards!

...JRF...

[Bill Hassell]

> i want to mount a filesystem, which doesn't exist in fstab

fstab provides the required 2 filenames for mount (source and destination). The fact that you can give just one (either one) is because fstab fills in the other. So, yes, you'll need both.

> or how can i detect the best way of a mount filesystem ? bdf ?

Well, bdf won't be of any value -- it only shows what is already mounted. You can always create a temporary mount point (which is nothing more than a directory), then give the full mount command:

mkdir /mnt1
mount /old_fs /mnt1

Once completed, you can umount /mnt1 and remount the copied lvol to it's final location. But why not just mount it to it's final location first?

> how can i check, if dest filesystem is the same like src filesystem ?

cksum is useless for checking the contents of a directory. It only returns the results value for a single file. You might simply check the size of the source and destination:

bdf /old_fs /new_fs
or
du -s /old_fs /new_fs

but this will almost always be different. If the /old_fs had a huge directory with thousands of files and all the files were removed, the size of the directory (which is just another file) will still contain thousands of empty directory slots. But create the same directory on /new_fs and you will see a 96 byte empty directory. Both are identical but the /new_fs will be smaller. Similarly, sparse files will be expanded by cpio and thus /new_fs could be much larger.

A better way to verify that everything has been copied is to count the files and directories. These will be identical except for lost+found so use these commands to compare the resultant copies:

find /old_fs ! -path /old_fs/lost+found -type d | wc -l
find /new_fs ! -path /old_fs/lost+found -type d | wc -l

find /old_fs ! -path /old_fs/lost+found -type f | wc -l
find /new_fs ! -path /old_fs/lost+found -type f | wc -l

And finally, to check for sparse files, I have attached a script that will look for sparse files. It will look at one or more directories and show each file as a . if it is not sparse but stop and report of files where ll and du do not match. You can test it by creating a sparse file (see previous answers) and running checkSparse on that directory.