Windows cifsmount Unix and Rsync

[mlutz]

Hello

I got a issue with rsync (3.0.6)

I've mounted a share coming from a win server on my HPUX V2 server via the cifsmount command.

Once this is done I'm triing to do a rsync -vvlrtz of some specific folder. to that win. share

It sounds working however sometimes filesize of the transfered file is not correct, when this happens, it always displays a size of 1GB instead of maybe 100kb.
Even sometimes the file stays under its temp form .blablabla (with 1Gbsize).

Does anybody know why files are sometimes transfered with a bad size ?

Thanks in advance for your help

Mathieu

[Bill Hassell]

When you copy a file (cp, rcp, rsync, etc), the file is read sequentially. Unix has the concept of a sparse file, that is, one that was written randomply. For example, a database might write a record at location 1 and then another at location 1000000, then close the file. All the records in the middle are undefined and do not occupy any space. If you read the file, the filesystem code fills in the missing records with a string of zeros. So when you copy it to another disk or another system, the file will be larger, sometimes much larger.

So when you copy a file, do not use the size to verify the copy, Use a checksum and you'll find that the files have the same data.

[mlutz]

Hello,

I've not fully understood you ... does that mean the file is really consuming 1GB on the new system (win.) or is it only a false view displayed by the system ?

(I thought file management was done through an allocation table which store/manage all file disk blocks)

[Dennis Handly]

>does that mean the file is really consuming 1GB on the new system (win.)

Yes.

>I thought file management was done through an allocation table which store/manage all file disk blocks

Yes but it is the copying program that is expanding the holes.

[Bill Hassell]

> (I thought file management was done through an allocation table which store/manage all file disk blocks)

You are comparing a PC fileystem to much more sophisticated filesystem management. As I mentioned, when you copy the sparse file, all the missing records are created as a stream of zeros. The PC has no way to know that this did not exist in the original file -- nor does it care. Both forms of the file have identical data. The sparse file uses less space but when the sparse records are written with some data, the local file will be the same size as the copy.

Alwaysa use the cksum command to verify file copies. (oh, you'll have to find cksum for PCs -- Windows doesn't have that tool).

[Matti_Kurkela]

How are you determining the original file size (100kb or so)?

If you use "du", be aware that with sparse files, it shows the "compressed" size. The du command does exactly what it says: it tells you how much disk space this file occupies - on this particular filesystem. A different filesystem may have different file compression properties, and therefore get a different result.

But if you use "ls -l", that displays the number of bytes in the file - including the runs of zero bytes that are not actually stored anywhere because of the sparse file mechanism.

At work, my mini-lecture on "sparse files in Unix-style filesystems" is usually triggered by an applications guy or a junior sysadmin noticing that the size indicated by "du" is much smaller than the size indicated by "ls -l".

But if "ls -l" indicates your original file has a size of 100kb, there may be a problem. Try to copy the file to a different Unix-style filesystem:

cp /somewhere/suspicious_file /var/tmp
or
dd if=/somewhere/suspicious_file of=/var/tmp/suspicious_file

If the copy again has a size of 1Gb or so, I'm afraid your HP-UX filesystem may be corrupted: the inode information says the file size is 100kb, but when actually reading the data, the filesystem gives out a much longer datastream. In that case, I'd schedule a maintenance break as soon as possible, unmount the filesystem and run a full file system check on it.

If errors are found, the file contents are probably already corrupted (most likely the contents of some other file are mixed in). In that case, it would be best to find the latest uncorrupted backup of the file and use it to verify the data.

http://en.wikipedia.org/wiki/Sparse_file

MK

[mlutz]

Hello,

Ok in between I had a look in the sparse file stuff. too.

But I wonder one thing on my issue.
Tell me if I'm wrong... If I got a sparse file, it would mean that each time I'm "rsyncing (rsync -vvlrtz) the file" it should become a 1GB file ... correct ?
Unfortunately this is not my case :(
Here it happens randomly ...

Moreover du and ls commands provides me non-spare file results e.g. :
du z.txt : 194 (512b blocks)
ll z.txt : 98852

I discovered the --sparse option in rsync and I tried it, first I was quite happy I though it was working but then after around 4000 files copied I got again some couple of 1GB files :S

I really asking my self if this is due to my exotic stuff (remote windows share cifs client mount + rsync Unix to Win on that CIFmount)

Any Idea or comment ?

[Steven E. Protter]

Shalom,

You might find more consistent file copying by doing cifsmount and then copying.

You lose the advantage rsync provide you which it the ability to not copy files already copied across. However the file protocols are more compatible.

SEP

[mlutz]

Hello

What a pitty, I wanted to make daily update of my stuff ...and not a kind of delete a copy each night :/

but sounds you're correct with cp it works ..no sparse Gb sized files are create.

maybe issue lays within rsync :/