Strange rsync problem

[Steven_Chen]

Hi Admins,

My customer face a strange problem on using rsync to move file from local file system to NFS file system mount on same host by samba.

some files during rsync moving will generate several times size on destination and unreadable.

example:
file a ~ file z
all files size around 10MB, but after moving these files using rsync, some files size will became to 1.4GB
is this bug for rsync ?

OS version is 11.23 on ia64, and rsync version is v 3.0.6 protocol version 30.

[Bill Hassell]

Are these files part of a database? It is common for databases to create sparse files, that is, files that have a few records but they are positioned far apart logically. The 10MB files are really more than 1GB in size but only a small number of records have been written. When the file is copied, all the empty space is read as a series of zeros and written to the destination. Unless you use the program that created these files to copy the data to another file, the file will be expanded with any file reading tool (cp, tar, cpio, rsync, etc). Perfectly normal.

[Steven_Chen]

Hi Bill,

No , it just simply text files.
that's why I said its really weird.

[Bill Hassell]

Check the files with xd. You may find a lot of zeros (empty or non-existent records) that are not visible:

xd -xc filea

Zeros are not normal in a plain text file but some application may have created streams of nulls (zeros) by skipping records. The original file will be small, the copy will be much larger.

[Steven_Chen]

New situation here:

when execute rsync -av $SOUR_PATH/ $DEST_PATH
the first 10 min see all fine,but at the moment all changed, some files start growth to 1.1GB, and I just let it keep running, more worth is some file growth to 9GB!

after I kill the rsync process, and list these wrong size files to a file as list, then re-write a script to process these wrong size files, the result is some of them are complete in normal size (10MB) and some others growth to 1.2GB again.

what can I do now?
I really haven't face this strange thing before !

[Steven_Chen]

Update:

I build another windows server and mount its share folder on same network/subnet by cifsmount.

now is more intresenting, using cp to copy file have no any issues, but using rsync, every file became 100 times from 10MB > 1.2GB!

is there anything wrong on this system?

[Dennis Handly]

>but using rsync, every file became 100 times from 10MB > 1.2GB!

Are these files sparse? What does ll(1) show for their size?

[Steven_Chen]

>Are these files sparse? What does ll(1) show for their size?

No, these files are formatted data generate by customer's application.
the file size using ll and ls -la are same result with wc command get.

I do another sets of test with this host by using cifsmount to mount another windows server shared file system.
scenarios:
1) HPUX run rsync from /local/path/files to /mnt/WINDOWS_FS > all files size became 100 times.
2) HPUX share local file system by samba and mount to windows server as net driver, copy files from HPUX to Windows local file system, no problem.
3) run rsync and cp on HPUX box, source and destination is same HPUX box local file system, no problem.
4) ftp from windows server to HPUX get all files back on HPUX, no problem.

can I narrow down the problem root cause is the HPUX box and seems its cifsclient library problem?
and this server's CIF server(samba) is not install from depot as package, its previous admin compile from source.