Problem with tar piping through openssl to tape

[Andrew Kaplan]

A colleague and I wrote a perl script that is designed to change to a specific directory that contains upwards to seventy subdirectories, and once there proceed one at a time to tar the subdirectory, encrypt the newly created tar file, and then tar to tape media the encrypted tar file. Once that was done, the script would go back to the source directory, remove the tar and encrypted tar files, and repeat the process on all additional subdirectories in the same location.

The command syntax that is used to accomplish this is the following:

###############################################
$cmd= "(cd $path; tar -cvf - $_|openssl des3 -salt -k $password > $_.des3)";
print `$cmd`;
if ($first){
$cmd= "(cd $path; tar -cvf $device $_.des3)";
} else{
$cmd= "(cd $path; tar -rvf $device $_.des3)";
}
$RC=$?;
print `$cmd`;
$cmd= "(cd $path; rm $_.des3)";
print `$cmd`;
$first=0;
}
###############################################

During a testing of the script, indications were the above process completed successfully. There was write activity on the tape drive, and no errors appeared on-screen.

After the session was completed, the tape media was rewound, and the tar -tvf command was ran to list the contents of the tape. The output indicated that the first directory was backed up successfully, but there was no indication of any additional directories being backed up to tape. The mt -f fsf 1 command was used to go to the end of the archive, and the tar -tvf command was reran to see if there were additional archives to the initial one on the tape, but there was no indication of any.

One thought that came to mind was to remove, or initially comment out, from the script the tar -rvf command, and rerun the script along with the follow-up tvf command. However, this did not appear to make any difference.

Does anyone have any ideas or suggestions? Thanks.

[Steven Schweda · Accepted solution]

What's "$device"? Is it rewinding or
non-rewinding?

[Andrew Kaplan]

Hi there --

Thanks for your reply. The item $device refers to a non-rewinding tape drive be it a dlt, superdlt, or LTO-4 device.

I did not include it in the initial posting, but the script prompts the administrator for the device path of the drive in question. One example would be:

/dev/rmt/5mn.

If people need to see the script in its entirety, I can attach it to a future response.

[Steven Schweda]

Looking at it again, "tar -r"?

http://docs.hp.com/en/B2355-60130/tar.1.html

says:

[...]
r

Add the named file to the end of the archive. The same blocking factor used to create
the archive must be used to append to it. This option cannot be used if the archive is a tape.
[...]


You're writing to a tape, right?

[Steven Schweda]

Is there anything in your script which
notices if a command fails? (Failure is not
an option?)

[Steven E. Protter]

Shalom,

I feel your methodology is overly complex.

Write to a simple file.

Use PGP (availabel at http://software.hp.com Internet Express) to encrypt the file.

Then transfer it to tape.

SEP

[Andrew Kaplan]

Hi there --

The writing is to tape, so right off the bat the -rvf command will have to be removed. One test that I ran had my commenting out the lines within the script that immediately follow the initial $cmd line to see if that made a difference, but it did not seem to do so.

I have a failure check later in the script that determine if the script completed successfully. It came back with an error code of 0.

As far as writing everything to a file, and then sending the file to tape, I can go that route if necessary, but that would require a separate directory in which to write the file. My hope is to avoid having to do that.

[Steven Schweda]

> I have a failure check later in the script
> that determine if the script completed
> successfully. It came back with an error
> code of 0.

Apparently, either your definition of success
differs from mine, or else that "failure
check" is ineffective.

[OldSchool]

as noted above...you can't use the "-r". you can write additional archives to the tape, provided that you did indeed use the non-rewind device. each archive will be a separate file on the tape.

The downside of that is that since each archive is a separate file on the tape, so you have to know its location and skip the appropriate number of file marks forward to access it, which can be messy to put it mildly.

[Andrew Kaplan]

Hi there --

It looks like I am going to have to go with the scratch directory approach where the encrypted files are initially created, and subsequently copied to tape. Although it isn't my first choice, at this point it looks like that will probably be the best avenue for success.

[Bill Hassell]

And as a programming note when using a tape in this manner. The man page for mt cautions about AT&T versus Berkeley dvice files. AT&T has some very bizarre behaviors when you close a tape file (it can move back to the beginning of the just-created file on the tape). Berkeley device files are sane -- they will not move the tape when you close it. Use the lssf command to find the Berkeley files, or use insf to create them:

lssf /dev/rmt/* | grep berkeley

[Andrew Kaplan]

I am going to try the single file approach that was suggested but use the openssl des3 encryption for the added security of the 'secret password'.

If I have to use a separate directory to create the tar file, encrypt it, and then copy it to tape so be it. However, prior to going down that route, can someone clarify if the following command syntax would work:

tar -cvf - . |openssl des3 -salt -out > /dev/rmt/5mn/archive.tar.des3

[Dennis Handly]

> ... > /dev/rmt/5mn/archive.tar.des3

Is this a tape drive or a file?

[Andrew Kaplan]

Hi there --

The /dev/rmt/5mn is a tape device, the archive.tar.des3 is the tar file encrypted by openssl.

[Steven Schweda]

> [...] can someone clarify if the following
> command syntax would work:
>
> tar -cvf - . |openssl des3 -salt -out > /dev/rmt/5mn/archive.tar.des3

Yes. I'd guess that it's doomed. Tape
devices are not, in general, file-structured
devices. Is "/dev/rmt/5mn" a directory on
your system? If not, then what is
"/dev/rmt/5mn/archive.tar.des3"?

I'd try sticking to program which knows how
to write to a tape device, like, for example,
"tar", "pax", "cpio", ...

[Steven Schweda]

> [...] can someone clarify if the following
> command syntax would work:

Why ask this question? Try it. Does it
work? Reality can be very informative.
Quick, too.

[Dennis Handly]

>The /dev/rmt/5mn is a tape device, the archive.tar.des3 is the tar file encrypted by openssl.

If that is a tape, you can't append names to the end. Just:
... > /dev/rmt/5mn

>Steven: I'd try sticking to program which knows how to write to a tape device

I suppose this might be dd(1) to reblock it in bigger blocks?

>Steven: Why ask this question?

Perhaps the question was what's better? :-)

[Steven E. Protter]

Again,

Bill's remarks should close the debate.

Whatever encryption technology is used, reporting directly to tape is not recommended.

Write your report/data to a file, encrypt and send to tape.

I know you have pride of ownership in your methodology, it seems ingenious. But it will not do the job effectively in this case.

SEP

[OldSchool]

ok, so based on your approach:

1. continue with the piece that creates the encrypted files of each subdirectory
2a. instead of backing them up to tape immediately, write a series of tar commands to a file. You want the resulting output to be something along the lines of:
cd
tar -cvf -C mypath1 ./mytar1 \
-C mypath2 ./mytar2 \
-C mypath3 ./mytar3
and so on for each of your ~70 directories
2b. write a series of delete commands to a file, you want the end result to be something like:
rm mypath1/mytar1 \
mypath2/mytar2 \
mypath3/mytar3
and so on
3. execute the file containing the tar command. the end result is one tar archive on the tape containing each of the specified encrypted tars.
4. Check status of the tar, if ok, run the delete file

this approach assumes sufficient disk space to retain each of the encrypted files. if you did this to minimize the useage, then this may not work.

You appear to have all of the basics you need to do the above, rearranging when operations happen as outlined may help.

[OldSchool]

oh yeah... in the files noted above, you want the actual (expanded) names for the various paths and files...which i'm sure you knew...

sorry the example is real clear, but hopefully you get the intent.

[Andrew Kaplan]

Hi there --

I went ahead with the scratch directory approach, and after the tar file was encrypted, the dd command was used to copy the file to tape.

A test of a related script which is designed to extract a file from the encrypted file on tape was also run with similar success.

Thank-you all for the help and suggestions.