Copy sizes are different

 
N.D
Occasional Advisor

Copy sizes are different

I am copying from one filesystem to another;
' cp -rp /source_dir /destination_dir '

but when i do a bdf the sizes are different and so is the file sizes i have tried recopying, deleting etc, but still the same and i need to make sure the data copied over is identical
7 REPLIES 7
David_246
Trusted Contributor

Re: Copy sizes are different

Hi,

Better is to use tar for these occasions. Copi-ing directories may resume in less data. For each file a specific amount of space is used per file. do an ls -lad of all directories in both directories. This shows you the difference.

Also cp does not copy special files. tar does (not all, but read the man page for those).

So :

tar cvf - . | ( cd /to/new/dir; tar xvf - )

Of course you inittiate this command from the source directory.

Regs David
@yourservice
Robert-Jan Goossens
Honored Contributor

Re: Copy sizes are different

Hi,

You can also use cpio,

# find /source_dir | cpio -pcmudv /dest_dir

Hope it helps,

Robert-Jan.
Stefan Farrelly
Honored Contributor

Re: Copy sizes are different

If you want to be sure the copy sizes are identical then tar up the source files first, creating a single tar file which you can then run sum on - to get a checksum value, then ftp to remote destination, run sum and confirm the same which means they are identical, then untar it.
Im from Palmerston North, New Zealand, but somehow ended up in London...

Re: Copy sizes are different

Hi,

Are you sure its not just the directorys that are different sizes?

Directory files grow, but never shrink, so when you get more than a certain number of files in a directory, the directory itself takes up slightly more space, but if some of the files in the dir are deleted it doesn't get smaller again.

A better way of checking whether your copy was successful is to look at just the files. Try this:


cd /source_dir
find . -type f -exec ll {} \; > /tmp/source

cd /destination_dir
find . -type f -exec ll {} \; > /tmp/destination

diff /tmp/source /tmp/destination

If this reports no differences your OK.

HTH

Duncan

I am an HPE Employee
Accept or Kudo
Elena Leontieva
Esteemed Contributor

Re: Copy sizes are different

To verify the data integrity, compare checksum and size:

cksum filename

Elena.
A. Clay Stephenson
Acclaimed Contributor

Re: Copy sizes are different

You are almost certainly a victim of a "sparse" file.

Here is how 1MB file that occupies on two bytes can be created.

char buf = 'x';
fdes = open("myfile",O_RDWR | O_CREAT,0664);
write(fdes,&x,1);
lseek(fdes,1000000,0);
write(fdes,&x,1);
close(fdes);

This creates "myfile" writes "x" at the beginning of the file, skip 1000000 bytes, and writes a second "x" --- a sparse file.

This file would ls as 1000000 bytes but bdf as two blocks --- the intervening bytes are never allocated BUT when myfile is copied those intervening bytes will be padded with NUL's on the copied file and instantly bdf will report very different numbers. When directories are copied, the "holes" (artifacts of deleted files in that directory) are not copied so that directories commonly don't copy exactly.

bdf and ls are not very good tools to determine if a file has copied correctly. After all, the SIZE of the files may be identical but that says nothing about the CONTENTS of the file. A much better test is to run cksum of each file.

Here is one method:

On the source dir:
find . -type f -exec cksum {} \; | sort > srclist
Now on the destination:
find . - type f -exec cksum {} \; | sort > destlist

diff srclist destlist -- if any lines are reported you have problems.
If it ain't broke, I can fix that.
Sundar_7
Honored Contributor

Re: Copy sizes are different


Couple of things missing.

char buf = 'x';
FILE *fdes;
fdes = open("myfile",O_RDWR | O_CREAT,0664);
write(fdes,&des,1);
lseek(fdes,1000000,0);
write(fdes,&des,1);
close(fdes);
Learn What to do ,How to do and more importantly When to do ?