Why does tar not work?

[Thomas Schler_1]

Hi,

I need to create a tar archive using a combination of find, xargs, and tar. E.g.

find tmp | xargs tar fcv tmp.tar

(Assume some more options of find for filtering purposes.)

find produces an output of several hundred paths. All these files shall be put into one tar archive "tmp.tar". But this does not work. Although, I do not get an error message, the resulting tar archive never reflects the paths delivered by the find process.

I tried some other variants:

% tar fc tmp.tar `find tmp`

% tar fc tmp.tar any_file
% find tmp | xargs -n 10 tar fr tmp.tar

% tar fc tmp.tar any_file
% find tmp -exec tar fr tmp.tar {} \;

Nothing works. What is going wrong? What is the solution?

(Is it time to change from HP-UX 11.00 to Linux?)

[Patrick Wallek]

No, no need to change to Linux. You just need to get your command line options in the correct order.

Your 'f' option should immediately preceed the name of the file you are taring t.

# tar -cvf tmp.tar filenames

So in your examples:

# find tmp | xargs tar cvf tmp.tar -

Note the '-' at the end of the tar command as well. That tells tar to read the file names from standard input.

[RAC_1]

Is it find tmp | xargs tar fcv tmp.tar
or find /tmp | xargs tar -fcv tmp.tar

Anil

[Sanjay_6]

Hi,

the command works for me. I just ran the command that thomas has mentioned and got a tar file. What is the error you are getting. Hope you are not running out on filesystem space.

Hope this helps.

Regds

[Thomas Schler_1]

Patrick,

unfortunately, there's no effect using '-' or using it not. Again, no error message, but:

% find tmp | xargs tar cvf tmp.tar -
% find tmp | wc
336 337 8656
% tar ft tmp.tar | wc
35 35 675

(Using xargs, '-' should be obsolete.)

Something is going wrong here. I assume, it has something to do with LINE_MAX, see xargs(1) and limits(5).

But, what is the work-around?

[Thomas Schler_1]

Sanjay,

thank you for your test; no, file space is ok.

Please, see my answer to Patrick. Would you, please, again do the test with some hundreds of files, and, say about, 50 MB?

[Marvin Strong]

#find dir | xargs tar cvf dirnam.tar -

works for me here, provided I'm in the correct directory.

[Thomas Schler_1]

Marvin,

please, do following test:

% find dir | xargs tar cvf dirnam.tar
% find dir | wc
output??

% tar ft dirnam.tar | wc
output??

[Jason W. Neiss]

> % tar ft dirnam.tar | wc

Again, the 'f' (file) option should come right before the filename:

% tar tf dirnam.tar | wc

Looks to me like this is consistently your issue...

Jason

[Marvin Strong]

# tar tvf mfstrong.home.tar | wc
79 633 6292
root@hp19rm4 [/tmp]
# find /home/mfstrong | wc
105 106 3789

hmm I see your point.

[Thomas Schler_1]

Jason,

in my opinion, there's no problem with the order of 'ft' or 'tf' or such things.

Marvin, Sanya,

anything new??

Patrick,

already at home? Really, no need to change to Linux?

[Jeroen Peereboom]

L.S.

I think the problem is within the command sequence.

Xargs takes as many arguments as possible, and starts the tar command with all these arguments. This means that tar will be started multiple times, each time overwriting the previously made tarfile. Your tarfile probably only contains the last XX files.

Don't use xargs, just do
find .... | tar cvf tarfile.

JP.

[Marvin Strong · Accepted solution]

don't know whats up with tar, but if your aren't tied to tar, you could use cpio.

find dir | cpio -oc > output

my results:
# find scripts | wc
68 68 1528
root@hp19rm4 [/home/mfstrong]
# find scripts | cpio -oc > scripts.cpio
349 blocks
# cpio -ict < scripts.cpio | wc
349 blocks
68 68 1528

[RAC_1]

what shell you are using? Seems csh.
type args, type tar

Anil

[Shannon Petry]

There can be problems with the order arguments are given to many commands. I do not think that this is the case here, but best practices should be to give "f" as the last argument, and the action first, in this case "c".

The reason for this is that other arguments may require additional arguments. "b" immediately comes to mind.
I.E.
tar -cvbf 2048 -

To fix the problem, try a different approach.

tar -cvf myfiles.tar `find . -depth -print`

Notice the grave marks enclosing the find command. The grave marks tell the shell to execute the command enclosed to get it's arguments.

Regards,
Shannon

[Shannon Petry]

I just wanted to add a few things after re-reading a bit in the thread.

"find tmp" is not compiant usage of the find command. This could produce an empty list, or pathing issues.

You do not state that an arcive is not made, just that the path is not reflected.

Is an archive made at all? if so what is the contents? There should be something in archive.

If you want absolute pathing in the file, then find needs to give an absolute path.

find /tmp -depth -print
makes the absolute path /tmp

find . -depth -print
makes the cwd the absolute path.

Another problem I noticed is that in your -exec example you do not give the path to tar, which is required for execution.

find /tmp -depth -exec /usr/bin/tar rf myfile.tar {} \;

Try not to be lazy with command arguments. While it may work under certain circumstances, it will not work in all cases.

Lastly, which tar are you using? and are you using a hybred find? It can make a difference if LD_LIBRARY_PATH needs to be adjusted.

Regards,
Shannon

[Jeroen Peereboom]

I just checked on a Linux system.If you have many files, only the files found 'last' will be in the archive.So you may want to use an 'append' or 'update' mode. Linux tar has an 'r' flag, for append mode.This gave a weird result: the tar arhive contained twice as many files as were found by find. I do not understand (yet).So, using xargs causes the problem.My suggestion to use find ... | tar cvf is wrong. Won't work.JP

[Patrick Wallek]

Probably the easiest thing to do is:

# cd /tmp
# find . -print > /dir/file_list
# tar -cvf file.tar $(cat /dir/file_list)

[john korterman]

Hi,
I think I once experienced something similar, namely that the argument list for xargs was too long. I also seem to remember that xargs does only produce an error message if you use it with the -x option. Try testing your find commands like this:
# find | xargs -x 1>/dev/null

If it producces an error message like "xargs: arg list too long" we know what the problem is. If you can provoke that error message then try executing the above command without the "-x" option and without redirection.

regards,
John K.

[Jeroen Peereboom]

John,

by definition xargs cannot produce an argument list that is too long. Xargs will split the arguments such that the list is not too long, and the command (tar in this case) is called as few times as possible.

My point is: xarsg will call tar a few times because there are so many files. Only the files (arguments) in the last invocation of tar (by xargs) will be in the tar archive. All previously created archives are overwritten.

JP

P.S. Of course, if you add an argument manually, the arglist may become too long.

P.S. 2. The number of arguments in the list is smaller if filenames are longer.

[Mike Stroyan]

tar will recursively include the contents of directories. All you need to tar all of the tmp directory is-

tar cf tmp.tar tmp

When you used the plain find command it listed both files and directories. That would cause tar to include multiple copies of files. When you use find to filter you also need to use a "-type f" filter so you only pass file names and not directory names.

The easy way to create a tar file with a really long list of names is to use pax instead of tar. It will write an archive format that tar can read. Pax will take a list of file names from stdin, so you can just pipe the output of find to it.

find tmp -type f | pax -w -f tmp.tar

[Dave Johnson_1]

The problem you are encountering is the 'c' command creates a new archive file each time tar is started. This way you loose all the files stored in the last run. So you need to use the 'u' command to update the archive with the new files to be added. The new problem is that the 'u' command will not create a new archive. So do this:

touch dummy
tar cf tmp.dir dummy
find tmp -tpye f | xargs uvf tmp.dir

and you will find you have what you are looking for plus the dummy file.

Hope this helps.
-Dave

[Thomas Schler_1]

Thank you all for helping solving my problem.

Special thanks to:

Marvin -- using cpio is a good work-around

Mike --
Your hint using '-type f' was the key. Also, using pax is really an easy way to create a tar file without running tar with a huge argument list, I didn't know that. Thank you.

Dave --
It's a good work-around, too. But, some little modifications on your solution must be done (using 'r' instead of 'u'; using '-n' option on xargs).