Ignite-UX tar errors.

[Kevin Boatright]

I"m having an issue with a make_tape_recovery on HPUX 11.11. Ignite was working and then stopped.
I originally had 7.7.89 loaded. I have attempted to reinstall ignite and downgraded to an earlier version.


* Creating local directories for configuration files and archive.

======= 01/29/09 09:36:12 EST Started /opt/ignite/bin/make_tape_recovery.
(Thu Jan 29 09:36:12 EST 2009)
@(#)Ignite-UX Revision C.7.7.98
@(#)ignite/net_recovery (opt) Revision:
/branches/IUX_RA0809/ignite/src@75250 Last Modified: 2008-08-25
11:39:14 -0600 (Mon, 25 Aug 2008)

* Testing for necessary pax patch.
* Recovery Archive Description = /dev/rmt/0mn

* Recovery Archive Location = /dev/rmt/0mn

* Number of Archives to Save = 2

* Pax type = tar


In? dsk/vg name minor# Associated disks/mountpoints
2 v /dev/vg00 0x00 /dev/dsk/c2t1d0 /dev/dsk/c2t0d0
/dev/vg00/lvol1 /stand 2
/dev/vg00/lvol2
/dev/vg00/lvol3 / 2
/dev/vg00/lvol4 /tmp 2
/dev/vg00/lvol6 /opt 2
/dev/vg00/lvol7 /usr 2
/dev/vg00/lvol8 /var 2
0 v /dev/vg01 0x01 /dev/dsk/c4t0d1 /dev/dsk/c6t0d1 /dev/dsk/c8t0d1 /dev/dsk/c10t0d1 /dev/dsk/c4t0d2 /dev/dsk/c6t0d2 /dev/dsk/c8t0d2 /dev/dsk/c10t0d2
/dev/vg01/lvol9 /home 0
/dev/vg01/lvol10 /home/emstat 0
0 v /dev/vg02 0x02 /dev/dsk/c4t0d5 /dev/dsk/c6t0d5 /dev/dsk/c8t0d5 /dev/dsk/c10t0d5
/dev/vg02/lvol11 /usr/local 0
/dev/vg02/lvphys /home/phys 0
/dev/vg02/lvconfig /home/config 0
0 v /dev/vg03 0x03 /dev/dsk/c4t0d3 /dev/dsk/c6t0d3 /dev/dsk/c8t0d3 /dev/dsk/c10t0d3 /dev/dsk/c4t0d4 /dev/dsk/c6t0d4 /dev/dsk/c8t0d4 /dev/dsk/c10t0d4
/dev/vg03/lvol12 /or_dump 0
/dev/vg03/lvtables /home/oracle/oradata/PR/tables 0
* Checking Versions of Ignite-UX filesets
* Creating System Configuration.
* /opt/ignite/bin/save_config -f
/var/opt/ignite/recovery/2009-01-29,09:36/system_cfg vg00
* Backing Up Volume Group /dev/vg00
* /usr/sbin/vgcfgbackup /dev/vg00
* Creating Map Files for Volume Group /dev/vg00
* /usr/sbin/vgexport -s -p -m /etc/lvmconf/vg00.mapfile /dev/vg00

* Backing Up Volume Group /dev/vg01
* /usr/sbin/vgcfgbackup /dev/vg01
* Creating Map Files for Volume Group /dev/vg01
* /usr/sbin/vgexport -s -p -m /etc/lvmconf/vg01.mapfile /dev/vg01

* Backing Up Volume Group /dev/vg02
* /usr/sbin/vgcfgbackup /dev/vg02
* Creating Map Files for Volume Group /dev/vg02
* /usr/sbin/vgexport -s -p -m /etc/lvmconf/vg02.mapfile /dev/vg02

* Backing Up Volume Group /dev/vg03
* /usr/sbin/vgcfgbackup /dev/vg03
* Creating Map Files for Volume Group /dev/vg03
* /usr/sbin/vgexport -s -p -m /etc/lvmconf/vg03.mapfile /dev/vg03

* Creating Control Configuration.
* Creating Archive File List
* Creating Archive Configuration

* /opt/ignite/lbin/make_arch_config -c
/var/opt/ignite/recovery/2009-01-29,09:36/archive_cfg -g
/var/opt/ignite/recovery/2009-01-29,09:36/flist -n 2009-01-29,09:36 -r
pa -b 64 -d /dev/rmt/0mn -t 1 -i 1 -m t
* Saving the information about archive to
/var/opt/ignite/recovery/previews
* Creating The Tape Archive
* Checking configuration file for Syntax
tar: Cannot add file -: No such file or directory
tar: Cannot add file 20: No such file or directory
tar: Error exit delayed from previous errors
make_medialif: ERROR: Cannot tar and gzip script files; possibly insufficient disk space in /var/tmp/make_medialif6965.

======= 01/29/09 09:36:48 EST make_tape_recovery completed unsuccessfully

I did not find any files labeled - or 20
There is 5GB of free space in /var.

I was able to tar the filesystems directly to tape without any issue.

I did find this other thread that had the same issue but didn't see a resolution.

http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1233322172759+28353475&threadId=1171617


Any ideas?

Thanks,
Kevin

[Michael Mike Reaser]

Somewhere on your system you have a file whose name is of the form

[string-of-characters][:space:]20[:space:][string-of-characters]

IOW, you have a file, somewhere, that has the string " 20 " buried within its name.

I ran into something similar with Ignite several years ago, where Oracle was using Unix-unsociable names eleventeen-gazillion levels deep within their directory structures. After having the DBA either rename 'em or remove 'em (I just don't remember which), tar was happy as a clam.

[Avinash20]

You could find the unprintable character via ls -b

[Steven Schweda]

> I did find this other thread [...]

My thoughts are unchanged from back then. I
still find it hard (bordering on impossible)
to believe that any file names are causing
this problem, but if someone can actually
_demonstrate_ a "tar" failure caused by goofy
file names (in the file system, not on the
"tar" command line), instead of merely
conjecturing with no evidence, then I'd be
willing to reconsider.

> [...]
> tar: Cannot add file -: No such file or directory
> [...]

Clearly "tar" is unhappy, but I can't see
the actual "tar" command line, so I can only
guess what bad things might be there. My
_guess_ is that there are some misplaced
options which are being interpreted as file
names, but the weakness of my psychic powers
means that it's only a guess. Who made the
bad command line, and how/why, remain
mysteries.

> [...] but didn't see a resolution.

If you do puzzle it out, be sure to explain.

[Kevin Boatright]

Mike, Oracle was installed on this box. Our vendor has been copying files from one box to another. No issues running ignite on the other box. It's running version 5.4.50.

I could not find any files with 20 in between spaces.

I was able to preform:
tar xvf /dev/rmt/0mn on /sbin /dev /stand /usr and /etc without any issues.

I also try using make_tape_recovery -m cpio and still receive the same tar error.

I guess I could try and find Ignite 5.4.50 and install that on the box.

[Bill Hassell]

> tar: Cannot add file -: No such file or directory
> tar: Cannot add file 20: No such file or directory

As mentioned, these are bad filenames. Ignite/UX (list_expander) and tar check every filename to be saved. The first is a very common error: The file is just a dash (-), or it is a dash surrounded by non-printing characters. (You can create a file with a dash like this: touch /tmp/-

The second is similar. It can be a file with a 2 followed by 0 but not necessarily together, for example, ESC 2 9 backspace 0 ESC or something similar.

In both cases, you will have to look carefully. You will have to escape the "-" as you search for it, something like this:

find / -name '*-*' -exec ls -dlb {} \;

Note that this will find any file with a dash somewhere. Remove the dash file with a a full path (ie, rm -i /tmp/*-*) Try a similar command for the 2..0 file:

find / -name '*2*0*' -exec ls -dlb {} \;

[sujit kumar singh]

hi

also can have a look in addition to what Bill has said at the content of the following:


/var/opt/ignite/recovery/2009-01-29,09:36/flist

regards
Sujit

[Steven Schweda]

> As mentioned, these are bad filenames.

Define "bad filename". Perhaps my VMS
experience has biased me, but is HP-UX so
lame that the
official system backup procedure can't cope
with _any_ legal file name? Really??? If
VMS had a problem like that, there'd be a mob
on the move with torches and pitchforks.

Am I the only one in the universe who finds
this explanation hard to credit?

[Steven Schweda]

And I know that I'm a bit slow, but can
someone explain why the thing says:

* Testing for necessary pax patch.

and then apparently uses "tar" instead of
"pax":

tar: Cannot add file -: No such file or directory

????

Given the limitations of the typical lame-old
"tar" program (file name length, file size,
who-knows-what-else), shouldn't it _be_ using
"pax", and not "tar"?

What obvious thing am I missing this time?

[Patrick Wallek]

pax can write output in either cpio format or tar format. The default is to write out in tar format. So as a result you have many of the same limitations as if you were to use the regular tar command.

If you happen to look at a process listing while a make_tape_recovery is being created then you would see that m_t_r does in fact use pax to create the tape.

[Steven Schweda]

> [...] The default is to write out in tar
> format. So as a result you have many of the
> same limitations as if you were to use the
> regular tar command.

Well, there's "tar" format, and then there's
"tar" format. It's been extended repeatedly
over the years, as evidenced by the ability
of GNU "tar" to deal with file names longer
than 100 characters, and files bigger than
8GB. (Eleven octal digits for the file size?
Who writes this stuff?)

> [...] m_t_r does in fact use pax to create
> the tape.

So, unless "pax" claims to be "tar" when it
gets an error, it's hard to explain those
"tar: Cannot add file [..]" complaints,
wouldn't you agree?

My (unfounded) conjecture is that somehow
someone's running "tar" using a "pax" command
line, and the (incompatible) "pax" options
are confusing/annoying "tar". Of course,
that would conflict with the prevailing "bad
file name" theory, but that's how it looks to
me. (But what do I know?)

[Bill Hassell]

"bad filenames" is perhaps a misnomer. The Ignite/UX list_expander and the various backup tools are trying to be helpful. There is problem at all in creating obnoxious names with special characters and invisible control characters. The problem is that what you see is not what you get unless you are knowledgeable (how to get rid of "--" or ";" files) and remember to use the -b option for ls as well as enclose spaces inside quotes. Unix isn't unfriendly, it is just indifferent. It follows exactly what you type, and assumes that users know the rules.

Personally I like the list_expander and pax/tar warning messages. It gives me a chance to educate users that create them.

[Steven Schweda]

> Personally I like the list_expander and
> pax/tar warning messages. It gives me a
> chance to educate users that create them.

I'm all for education, and if you can get
"tar" (any "tar") to complain about funny
file names which are in the file system but
not on its command line, I'd _love_ to be
educated.

And if you know of a way to persuade
make_tape_recovery to reveal the "pax" (or
"tar") command(s) it generates, _that_ might
be _particularly_ educational.

> [...] the various backup tools are trying
> to be helpful. [...]

Trying but not succeeding? Or are you saying
that it's more helpful to blow up in this
situation than it would be simply to get the
job done and be quiet about it? Perhaps this
is some new meaning for "helpful", with which
I'm unfamiliar.

This whole "bad filename" thing sounds to me
like a bit of folklore with no foundation in
fact. Now, I'm open to a good
counter-demonstration which will show just
how wrong I am, but a bunch of hand-waving
about restrictions which don't seem to exist
will not convince me. Surely, it can't be
very difficult to create a small file system,
populate it with (so-called) good and bad
file names, reproduce this failure, and then
remove the "bad" file names, and see how
splendidly everything works. _That_ would
_really_ be educational.

It would also seem to demonstrate that the
official HP-UX backup software is a steaming
pile of something stinky, but I'm willing to
believe that, too, _if_ someone can actually
demonstrate that it really has this rather
severe problem.

[Patrick Wallek]

OK Here's a very simple example.

I created a file called "abc - de f _"

Now, if I do:

$ tar -cvf test.tar a*
abc - de f _


It works fine.

It will even restore
$ tar -tvf test.tar
-rw-r--r-- wallekp/users 0 2009-01-30 22:17:22 abc - de f _


Now the problem comes in when you try to specify the file name on the command line, which pax/tar does with m_t_r:
$ tar -cvf test1.tar abc - de f _
tar: abc: Cannot stat: No such file or directory
tar: -: Cannot stat: No such file or directory
tar: de: Cannot stat: No such file or directory
tar: f: Cannot stat: No such file or directory
tar: _: Cannot stat: No such file or directory
tar: Error exit delayed from previous errors


Do the errors look familiar?


While Unix allows file names with spaces and other character, it can cause issues.

[Steven Schweda]

> Now the problem comes in when you try to
> specify the file name on the command line,

Isn't that what I specifically excluded? Why
do you think that I did that?

> which pax/tar does with m_t_r:

It _does_? You know that? Yikes. That
_would_ be a steaming pile. Doesn't it know
about apostrophes? I (yes, even I) do.

> Do the errors look familiar?

Only vaguely. They _are_ different.

I see that "man make_tape_recovery" (11.11)
does mention some restrictions:

[...]
File Names with Non-Printable Characters
Although permissible within HP-UX, it is inadvisable to use characters
that do not have a printable graphic on the hardware you commonly use,
or that are likely to confuse your terminal. Filenames with these
characters cause a warning message to be displayed by
make_tape_recovery. In addition, files that contain these non-
printable characters are not included in the archive.
[...]

I may see the steam beginning to rise...

Of course, I wouldn't consider a space
character to be "non-printable".

I also wouldn't call "make_medialif: ERROR:
Cannot tar and gzip script files; [...]" "a
warning message". But I may be eccentric.

I also haven't yet seen make_tape_recovery
spew the errors (under controlled
conditions), just some elementary shell
globbing problems (which may or may not be
related to what make_tape_recovery actually
does).

[Patrick Wallek]

What do you want, Steven? Do you want the developers to stand up a shout "we screwed up"?

File names with spaces, non-printable character etc., can cause problems with tar & pax.

My feeling is that when some of these tools were originally written, the developers never thought about spaces and other things in file names. Yes, there have been updates over the years, but the fact remains that in some cases Unix just doesn't deal well with some things. Is there room for more improvement? Absolutely.

However, I wouldn't call pax, tar, or the Ignite product a steaming pile of just because it doesn't handle some issues well.

If you think that little of it, don't use it. If you think that little of HP-UX, why don't you go back to your precious VMS?

[Steven Schweda]

> What do you want, Steven?

What I asked for: a quick demo that
make_tape_recovery really has trouble with
non-trivial file names. (Blame assignment,
always the most important thing, can follow.)

> File names with spaces, non-printable
> character etc., can cause problems with
> tar & pax.

But they don't, really. They _may_ cause
trouble with a _shell_, but that's not the
same thing as causing trouble with "pax" or
"tar". And characters as simple as spaces
don't need to cause trouble for a shell, if
the file names are handled well.

> However, I wouldn't call [...]

A backup program which can't back up legal
files? Boy, that's sure what I'd call it.

> If you think that little of it, don't use
> it. If you think that little of HP-UX, why
> don't you go back to your precious VMS?

I haven't actually left (HP's, not my) VMS.
I just get out a little more than some folks.

I _don't_ use HP-UX much, mostly testing the
Info-ZIP programs, and some other freeware.
(I just hang around in this forum for the
educational value.)

And I didn't say that HP-UX as a whole was
bad, just the defective backup program (if it
really is as defective as people seem to say
that it is).

[Dennis Handly]

>tar: Cannot add file -: No such file or directory
>tar: Cannot add file 20: No such file or directory
>tar: Error exit delayed from previous errors

As Steven says, something is wrong here. If ignite uses pax(1) there should be no tar messages. I get:
pax: - : No such file or directory

If you want to see what's really going on, you can use tusc:
tusc -fp -a -o tusc.out make_tape_recovery ...

(Assuming a demon isn't doing the work.)

[Bill Hassell]

The list_expander and pax/tar issues don't represent HP-UX filesystem standards. I submitted a bug report for list_expander because it could not handle very large volume groups (hundreds of lvols and pvs). Getting it fixed without introducing a new filename bug was a challenge.

Ignite/UX is a fine product, with the functionality conspicuously missing in most other flavors of Unix. But the choice of a patchwork tar or pax for today's Ignite product needs serious attention. Since Ignite/UX is 100% proprietary to HP-UX, the choice of its file backup tool should be unconstrained and fully compatible with any useable files and filenames (in my not so humble opinion). Ignite/UX tries to prevent saving (and restoring) systems with issues. Perhaps flags can be used to filter some of the warnings. Certainly the error messages can at least show the full pathname.

[Kevin Boatright]

I've looked for files that contain any special characters and either a - or 2*0. I have yet to find any.

I did what Dennis suggested and found this in the output.

[11328] execve("/usr/bin/tar", 0x40039d40, 0x40039de8) ... [entry]
argv[0] @ 0x4001ee60: "tar"
argv[1] @ 0x400219d8: "-cfb"
argv[2] @ 0x400219f8: "-"
argv[3] @ 0x40039d68: "20"
argv[4] @ 0x40039d78: "./opt/ignite/data/scripts/os_arch_post_l"
argv[5] @ 0x40039db0: "./opt/ignite/data/scripts/os_arch_post_c"

I'll continue to search.

[Kevin Boatright]

I found an old version of Ignite. Version B.5.4.50. Loaded that after removing 7.7.98.

I received the same error but it actually created the Ignite tape.

I'll try restoring to another box.

* Creating The Tape Archive
tar: Cannot add file -: No such file or directory
tar: Cannot add file 20: No such file or directory
tar: Error exit delayed from previous errors
Adding /opt/ignite/Version file to LIF.
46382+0 records in
46382+0 records out
* /opt/ignite/data/scripts/make_sys_image -c n -d /dev/rmt/0mn -t n -s
local -n 2009-02-02,11:15 -m t -w
/var/opt/ignite/recovery/2009-02-02,11:15/recovery.log -u -R -g
/var/opt/ignite/recovery/2009-02-02,11:15/flist -a 1489730

* Preparing to create a system archive

* Archiving contents of emstat via tar image to local device/dev/rmt/0mn.
* Creation of system archive complete


======= 02/02/09 11:20:30 EST make_tape_recovery completed with warnings

[Steven Schweda]

> [11328] execve("/usr/bin/tar", 0x40039d40, 0x40039de8) ... [entry]
> argv[0] @ 0x4001ee60: "tar"
> argv[1] @ 0x400219d8: "-cfb"
> argv[2] @ 0x400219f8: "-"
> argv[3] @ 0x40039d68: "20"
> argv[4] @ 0x40039d78: "./opt/ignite/data/scripts/os_arch_post_l"
> argv[5] @ 0x40039db0: "./opt/ignite/data/scripts/os_arch_post_c"

At first glance, that'd seem to me to be a
legit "tar" command:
tar -cfb - 20
(With "-f -", I'd assume that it's in a
pipeline.)

"/usr/bin/tar" would seem to be pretty
explicit, too, about which "tar" to use.

Any evidence that "/usr/bin/tar" is not what
it should be? Does a simple, similar "tar"
command work? Say,
/usr/bin/tar -cfb - 20

(That should spew to the terminal. You could
redirect it to someplace instead.)

[Dennis Handly]

>Steven: that'd seem to me to be a legit "tar" command:

Yes, it seems to work fine. Not sure why they use tar when they could use pax, possibly because those files are small?

If someone illegally replaced HP's tar by GNU tar you get similar errors:
$ PATH=/usr/local/bin tar -cfb - 20 hi.c
tar: -: Cannot stat: No such file or directory
tar: 20: Cannot stat: No such file or directory

Note: Nobody in their right mind should use the above tar syntax. Better to use:
tar -cf - -b 20 hi.c

[Kevin Boatright]

I received the following when attempting to Ignite to another box:
____________________________________________
x ./etc/passwd, 332 bytes, 1 tape blocks
x ./etc/magic, 8794 bytes, 18 tape blocks
x ./etc/hosts, 486 bytes, 1 tape blocks
x ./etc/services, 10908 bytes, 22 tape blocks
x ./etc/inetsvcs.conf, 15 bytes, 1 tape blocks
x ./etc/nsswitch.conf, 382 bytes, 1 tape blocks
x ./configure3, 454656 bytes, 888 tape blocks
x ./monitor_bpr, 24576 bytes, 48 tape blocks
* Download_mini-system: Complete
* Loading_software: Begin
ERROR: Tar: blocksize = 0; broken pipe?
ERROR: File: . not found.
ERROR: Could not load scripts from the tape.
ERROR: Missing script "/opt/ignite/data/scripts/os_arch_post_c".
* Installing boot area on disk.
* Enabling swap areas.
* Backing up LVM configuration for "vg00".
* Processing the archive source (/dev/rmt/0mn).
* Wed Feb 04 08:46:08 EST 2009: Starting archive load of the source
(/dev/rmt/0mn).
* Positioning the tape (/dev/rmt/0mn).
* Archive extraction from tape is beginning. Please wait.
_____________________________________________
#which tar
/usr/bin/tar
#what /usr/bin/tar
/usr/bin/tar:
#what /sbin/tar
/sbin/tar:
tar.c $Date: 2007/05/09 22:14:45 $Revision: r11.11/7 PATCH_11.11 (PHCO)
$ Version_11.11 May 21 2007 14:14:14 $
#cp /sbin/tar /usr/bin/tar
_____________________________________________

I reran the make_tape_recovery and I didn't receive the error:
tar: Cannot add file -: No such file or directory
tar: Cannot add file 20: No such file or directory
tar: Error exit delayed from previous errors

I reinstall Ignite 7.7.98 and ran a make_tape_recovery.
_____________________________________________
* /opt/ignite/lbin/make_arch_config -c
/var/opt/ignite/recovery/2009-02-06,11:26/archive_cfg -g
/var/opt/ignite/recovery/2009-02-06,11:26/flist -n 2009-02-06,11:26 -r
pa -b 64 -d Recovery\ Archive -t 1 -i 1 -m t
* Saving the information about archive to
/var/opt/ignite/recovery/previews
* Creating The Tape Archive
* /opt/ignite/data/scripts/make_sys_image -c n -d /dev/rmt/0mn -t n -s
local -n 2009-02-06,11:26 -m t -w
/var/opt/ignite/recovery/2009-02-06,11:26/recovery.log -u -R -g
/var/opt/ignite/recovery/2009-02-06,11:26/flist -a 1497290

* Preparing to create a system archive.

* Archiving contents of emstat via tar image to local device /dev/rmt/0m.
* Creation of system archive complete.

* Archive created and rewind the tape to beginning of tape.

* Cleaning up old configuration file directories

======= 02/06/09 11:32:24 EST make_tape_recovery completed with warnings

Looks like tar was the problem. Thanks for everyone's help!

Kevin

[Steven Schweda]

Thanks for the update.

> #cp /sbin/tar /usr/bin/tar

Wouldn't be my first choice as a solution,
but it's not totally crazy. (Especially
when it apparently actually helped.)

It'd be interesting to know more about the
origin of this quirk. (Like who's more
lame, make_tape_recovery or this particular
"tar" program.)