Output of multiple commands (different lines) into one command?

[whairst]

How would I collect the output of multiple commands (on different lines) and send it to one command?

HPUX 10.20 I'm working on a script that will collect the entire contents of a tape and write it to a compressed tar file. Considering the tape could hold >20 GB of data (uncompressed), and the machine in question only has 0.5 GB memory and limited drive space, I'd really like to avoid temp files or trying to hold the entire contents in memory. Ideally the only disk space used would be the final compressed file. What I have at the moment is the following, which works but uses temp files and only gets 3 entries from the tape:

mt -t /dev/rmt/0mn rew
mkdir /tmp/tapecontents
dd if=/dev/rmt/0mn of=/tmp/tapecontents/file1 ibs=51200
dd if=/dev/rmt/0mn of=/tmp/tapecontents/file2 ibs=51200
dd if=/dev/rmt/0mn of=/tmp/tapecontents/file3 ibs=51200
tar cvf - /tmp/tapecontents/file* | compress > /home/tapecontents.tar.Z
rm -rf /tmp/tapecontents

I'm planning on expanding that into a while loop, so would like something like this pseudocode:

setup outputfile = (tar cvf - | compress > /home/tapecontents.tar.Z)
while (tape has more contents); do
dd if=/dev/rmt/0mn of=(outputfile) ibs=51200
done

How would I setup something like "outputfile" here?

(Note: Doesn't have to be tar specifically, just something I can use to store the tape contents in the same order as on the tape, so they could be written back to tape that way if needed.) 

[Steven Schweda]

> [...] collect the entire contents of a tape [...]

> [...] only gets 3 entries from the tape: [...]

   What's the structure of the tape?  Does "dd" see individual "fileX"
entities, or one big stream of data?  I don't do much with tapes or "dd"
on Unix, so I know nothing, but I'd've expected "dd" to see one big
stream (until the End-of-Tape mark).

   In which case:

      dd if=/dev/rmt/0mn [options (not "of=")] | compress > outfile

   Your tape has some kind of file structure, so that your "fileX" temp
files are different data?

> [...] I'd really like to avoid temp files [...]

   Ok.  So use more parentheses?  I don't have an HP-UX system up at the
moment, and my antique collection doesn't extend back so far as 10.20,
but on a handy Mac, getting multiple commands to feed a single stdout is
no trick:

$ (
> date; sleep 2
> date; sleep 2
> date; sleep 2
> ) > ds.out

$ cat ds.out
Thu Mar 17 14:10:09 CDT 2022
Thu Mar 17 14:10:11 CDT 2022
Thu Mar 17 14:10:13 CDT 2022

   I assume that /bin/sh anyplace could do the same.  And the three
loose command lines here could have been a loop.

> [...] something like this pseudocode:

   That (especially the "tar" command) makes no sense to me.

   Also, why "compress" instead of, say, "gzip" (or bzip2)?

[whairst]

On a tape, there's no filesystem, just raw data. dd reads until it sees an EOF (really a tape mark). "End of tape" is really two EOFs/tape marks in a row. The example I gave was for reading the output of one particular program, which wrote to the tape 3 times, so there's 3 entries on the tape. A 4th dd returns success but with 0 blocks transferred, while a 5th gives an error.

How would I use parentheses when getting an unknown number of "files"/entries off of the tape (number of dd commands determined at runtime)?

tar was listed simply to stick the entries together before compression. compress is what I'm familiar with, and I know this system has it; I'm not sure about gzip or bzip2. I'm trying extremely hard not to add additional software aside from a few small scripts.

[Steven Schweda]

> On a tape, [...]

Ok. That's all plausible.  (Just unfamiliar.)

> tar was listed simply to stick the entries together before
> compression. [...]

   So far as I can see, if you want some kind of name attached to each
of these files, then you'll need to have the files (or something like
them) on a disk where "tar" can see that name.

   I might try extracting a file from the tape through the compression
program into a compressed file on disk.  For example:

      i=0
      stop=0
      while [ ${stop} -eq 0 ] ; do
        i=` expr ${i} + 1 `
        dd if=/dev/rmt/0mn | gzip -c > file${i}.gz
        if [ $? -ne 0 ] ; then
          stop=1
        else
          tar rf out.tar file${i}.gz
        fi
        rm file${i}.gz
      done

   I'd expect something like that (untested, probably defective)
scriptlet to suck a file off the tape, push it through gzip into a new
(numbered) disk file (compressed).  (Feel free to use "compress" if you
wish, but "gzip" compresses better (bzip2 still better), and if you have
a system without it, then I'd try to remedy that.)  Then, the compressed
(and named) file is appended to "out.tar", and the original compressed
(and named) file is deleted.

   With the "while" loop, it's supposed to repeat until the "dd" command
fails, but you might need to waste more disk space to get the right exit
status used (from "dd", not "gzip"):

        dd if=/dev/rmt/0mn > file${i}
        if [ $? -ne 0 ] ; then
          stop=1
        else
          gzip file${i}
          tar rf out.tar file${i}.gz
          rm file${i}.gz
        fi
        rm file${i}

   Either way, you should get a "tar" archive with a bunch of (named)
compressed files in it, which could be extracted and (individually)
expanded.  But note that that's a "tar" archive of compressed files,
_not_ a compressed "tar" archive of the original (unnamed) files.  So,
extracting the data from "out.tar" will be cumbersome, too.

   I don't see a way to get the kind of compressed "tar" archive which I
assume that you'd really like.  Without, that is, extracting all the
tape files onto a disk (where they'd have names).  If you don't have the
disk space to do that, then something like the above suggestion is what
I'd try.

[whairst]

That's quite clever. It still copies to disk first, but only stores one file at a time, creating the compressed archive along the way. I'll have to give that a try and see how it goes. Thank you very much.

Extraction would be essentially the reverse - use tar tvf to count the files in the archive, then for each, use tar xvf to extract it, decompress it, then write it to tape.

Incidentally, I don't necessarily need each "file" to have a name; the whole process is more about making a copy of the whole tape contents (minus the blank parts after the end-of-tape mark), compressed to save space. (Compressing it before or after tar doesn't really make a difference here.) That way I can store (and backup) the contents electronically instead of on a physical tape, restoring them to tape if needed.

[Steven Schweda]

> That's quite clever. [...]

   At least, something like that should work.  I'll settle for providing
some inspiration.

> Incidentally, I don't necessarily need each "file" to have a name;
> [...]

   You do, _if_ you want them in a "tar" archive.  (I claim.)  As I
said, what you were trying to do with "tar" was unclear to me.

> [...] the whole process is more about making a copy of the whole tape
> contents [...]

   Ok.  If so, then "tar" may provide no benefit other than providing a
single package for the data from one tape.

   As usual, if there is some actual problem which you are trying to
solve, then it might help if you explained what that is, rather than
asking how to implement some particular "solution", which might or might
not make much sense.

    Again, I know nothing, but I'd guess that various computer emulators
provide some kind of support for virtual tapes as well as virtual disks,
so there may be some established standard(s) for storing tape data in a
disk file.  If your tape hardware is as old/obsolete as your OS, then
I'd be looking for a way to escape from all of it.  Tapes on the shelf
don't last forever, either.

[whairst · Accepted solution]

Unfortunately, this system uses a bit of proprietary hardware, so I can't virtualize it, and some of the proprietary software writes to tape with no option of writing elsewhere. The goal is to be able to back up the contents of a tape to a file, and restore it later, given the blocksize used on the tape.

For those in a similar situation, check out https://www.arraid.com/data-storage-products/product/asfd-2-tp.html , a "tape drive" that really writes to CF card. (The host machine can't tell the difference - it acts like a tape drive.) It's faster and more reliable than tape, and by transferring the contents to another medium (like using the scripts below), one card can be used many, many times.

I've tested out these two scripts on one of our backups, and they seem to recreate the exact tape contents (again, given the blocksize).

tapeCopy.sh:

dest=/home/`date "+%Y-%b-%d"`.tar
let i=0
stop=0
mt -t /dev/rmt/0mn rew
rm -rf /tmp/tapeCopy
mkdir /tmp/tapeCopy
while [ $stop -eq 0 ]; do
    let i=$i+1
    dd if=/dev/rmt/0mn of=/tmp/tapeCopy/file$i ibs=51200
    if [ $? -ne 0 ]; then
        echo dd returned $?. Stopping.
        stop=1
    elif [ -s /tmp/tapeCopy/file$i ]; then 
        gzip /tmp/tapeCopy/file$i
        if [ i -eq 1 ]; then
            tar cf $dest -C /tmp/tapeCopy/ file$i.gz
        else
            tar rf $dest -C /tmp/tapeCopy/ file$i.gz
        fi
        rm /tmp/tapeCopy/file$i.gz
    else
        echo Found empty file. Stopping.
        stop=1
    fi
done
rm -rf /tmp/tapeCopy

tapeRestore.sh:

origin=$1
startdir=`pwd` # To restore when done
let i=0
stop=0
mt -t /dev/rmt/0mn rew
rm -rf /tmp/tapeCopy
mkdir /tmp/tapeCopy
cd /tmp/tapeCopy
while [ $stop -eq 0 ]; do
    let i=$i+1
    tar xf $origin -C . file$i.gz
    if [ -s /tmp/tapeCopy/file$i.gz ]; then
        echo Contains file$i.gz
        gunzip /tmp/tapeCopy/file$i.gz
        dd if=/tmp/tapeCopy/file$i of=/dev/rmt/0mn obs=51200
        rm /tmp/tapeCopy/file$i
    else
        echo Doesn\'t contain file$i.gz. Stopping.
        stop=1
    fi
done
cd $startdir
rm -rf /tmp/tapeCopy

I'll mark this post as the solution, since it contains the full code, but credit is really due to Steven Schweda. Thank you once again.

[Steven Schweda]

Looks reasonable to me, except:

>         if [ i -eq 1 ]; then
>             tar cf $dest -C /tmp/tapeCopy/ file$i.gz
>         else
>             tar rf $dest -C /tmp/tapeCopy/ file$i.gz
>         fi

   Are you working too hard, or is your (fossil) "tar" that lame?  I'd
expect "tar rf $dest" to be equivalent to "tar cf $dest", if "$dest"
doesn't exist.  It certainly was around here, with:

$ tar --version
bsdtar 2.8.3 - libarchive 2.8.3

> [...] Thank you once again.

   Glad to help.

[whairst]

> Are you working too hard, or is your (fossil) "tar" that lame?

The fossil one. "tar rf doesntexist.tar somefile" returns "tar: cannot open doesntexist.tar". I can't find a version switch for it (tar --version complains about s being an unknown option); the man page for it says HP-UX release 10.20: July 1996.

It gets the job done, though.

[Steven Schweda]

> [...] "tar: cannot open doesntexist.tar". [...]

   Thanks for the info.  On the bright side, your scheme should work
with a newer/better "tar" as well as it does with an older/lamer "tar".

   Despite "Change is bad," being my motto, some things have clearly
improved since HP-UX 10.20.

> It gets the job done, though.

   That might matter to some folks, I suppose.