topic Re: tar command changed directory size in Operating System - HP-UX

tar command changed directory size

jane zhang — Tue, 02 Dec 2003 19:00:57 GMT

Hi all,

I used command to move one directory from one box to another.
In box A
cd EN2001.1; tar -cvfh - . | remsh csihp08 "(cd /homes/boxB/EN2001.1; tar -xvf - .)"

The original size in boxA is
# du -sk EN2001.1
1723142 EN2001.1

After tar command, in boxB, the size got much bigger.
# du -sk EN2001.1
4501585 EN2001.1

Can you explain?

Jane

Re: tar command changed directory size

A. Clay Stephenson — Tue, 02 Dec 2003 19:11:55 GMT

Almost certainly that is a result of 1) sparse files or 2) symbolic links being followed.

Sparse files are those with "holes" in them. For example, write 1 byte at offset 0 then seek to offset 1,000,000 and write 1 byte. You now have a file which occupies only 2 blocks but ls -l reports its size as 1MB. When a sparse file is copied the "missing" bytes are filled in with NUL's. More advanced backup like OmniBack or fbackup are able to preserve sparse files.

Re: tar command changed directory size

jane zhang — Tue, 02 Dec 2003 19:15:46 GMT

Hi,

I do not really want to go back to omniback tape to restore if I can do it online.

the big size difference is definitely not desired. Does tar command have any option to reduce the holes? I also need to preserve the symbolic links as well.

Thanks,

Jane

Re: tar command changed directory size

A. Clay Stephenson — Tue, 02 Dec 2003 19:30:14 GMT

You should be able to change your tar|tar pipeline to an fbackep | frecover -s pipeline. Man fbackup (which doesn't really know about sparse files) and frecover (which does) for details.

Vanilla tar or cpio cannot preserve sparse files.

Re: tar command changed directory size

Bill Hassell — Tue, 02 Dec 2003 21:27:22 GMT

There is no way for any backup program to tell the difference between a normal and sparse file since (as with all Unix flavors) a regular file's format and contents are determined solely by the creating program. What frecover and Omniback will do is to see a string of nulls, stop copying them to disk until non-zero data is encountered and perform an lseek to skip to that particular record and continue writing. The filesystem remembers the empty space by not assigning inodes that to the data areas not written.

It is very important that 1723142 is NOT the true size of the file. If you use cp or tar or cpio or pax or dump, they will all create a copy that is 4501585 in size. This is the true size of the file and if this is a database file, the unoccupied space may eventually become occupied, eventually increasing the orginal file's size. It is very important to note that the file behaves exactly the same way whether the null records exist or not. To see this, use the cksum command on both files. You can also use the cmp command to compare the files. Other than the result from du, the files are truly identical.

It is very seldom the case that a backup then restore of a set files and directories will be the same size. A sparse file can make the copied result larger, but a sparse directory (a directory that once had thousands of files but they were deleted) when copied will be much smaller. This is normal behavior.

As far as symlinks, this can be quite complicated to resolve. A symlink is really just a string that contains the link. There is nothing truly connecting the symlink to a real file or path. So if you restore the link on another system, it may point to nothing. A hard link is just a directory entry that points to the same file and will always work when restored.

Re: tar command changed directory size

Elmar P. Kolkman — Wed, 03 Dec 2003 01:24:26 GMT

What you can do is a find on boxA to find symbolic links to make sure they are the cause of your problem:
find EN2001.1 -type l -exec ls -lad {} ';'

Or, to get some more info

find EN2001.1 -type l | while read link
do
echo "`ls -lad $link` -> `ls -adlL $link' `du -sk $link`"
done

When you add up the results from the du's in the loop, they could very well add up to your difference, in which case the -h option of tar is the cause of your size difference. If not, the sparse file option could very well be the cause. In which case you have to look at the type of files and need to think about better ways to duplicate the data, for instance a dump and import.

Re: tar command changed directory size

jane zhang — Wed, 03 Dec 2003 13:40:10 GMT

Hi,

Thanks you all, I will need to consider the other method to move the directory which contains symbolic links.

Jane

Re: tar command changed directory size

Michael Schulte zur Sur — Wed, 03 Dec 2003 13:49:43 GMT

Hi,

according to man pages tar does not follow links unless explicitly told so. Have you compared individuell file sizes? What about block size?

greetings,

Michael

Re: tar command changed directory size

Dietmar Konermann — Wed, 03 Dec 2003 16:49:49 GMT

Just a short hint... the pax(1) command also copies sparse files as sparse (and of course handles symbolic links). Have a look at the man page and watch out for the -rw option.

Best regards...
Dietmar.