'tar' & 'cpio' backup to DLT tape - different time recorded but both too slow

[Rita Li]

new HP9000 L1000 Enterprise server, HP-UX 11.0, HP DLT8000 tape drive - set w/ hw data compression on, same DLT type IV tape used

test w/ backup a file system of ~3GB

1. w/ cpio

a. backup to tape w/ this command :

cd && (/bin/find ./develop -depth -print) | /bin/cpio -oBc > /dev/rmt/1m

requires 67 mins to finish

b. list tape w/ this commad :
/bin/cpio -iBvtc < /dev/rmt/1m

requires 40 mins to finish

2. w/ tar

a. backup to tape w/ this command :

cd && tar cvf /dev/rmt/1m -C /develop/*

requires 30 mins to finish

b. list tape w/ this command :

tar tvf /dev/rmt/1m

requires 20 mins to finish

I can't understand :
Q1 : why 'tar' takes 1/2 backup/list time compare w/ 'cpio', what parameter in my cpio is set wrong?
Q2 : 'tar' is taking ~6G/hr vs 'cpio' is taking ~3G/hr to backup to the DLT tape, both slow compare w/ the 20G/hr (native non compressed mode) as claimed, w/ hw compression on, I am still expecting a rate of 10-12G/hr

Pls shed me light on this.

Thanks

-Rita

[Dan Hetzel]

Hi Rita,

This is easy to understand, once you know how tar and cpio are 'blocking' data.
tar uses a default block size of 20x512=10240 characters, cpio uses 5120 characters when given the 'B' flag. This explains why tar is faster in this case (larger chunks of data)

Your DLT is a streaming device, which will work a lot better when receiving a continuous stream of data, or when really large blocks are written.

You could try bypassing this 'limitation' the following way:

cd && (/bin/find ./develop -depth -print) | tar cf - | dd bs=64k of=/dev/rmt/1m

This forces writing 64K blocks, you may try different values and see if it works faster.

The same should work with cpio.

Keep in mind that, as the blocking factor isn't the standard tar blocking factor, you'll have to use 'dd' to read it, like this:
dd if=/dev/rmt/1m bs=64k | tar xf -

Let me know if it improves your tape creation speed.

Best regards,

Dan

[Tommy Palo]

Instead of piping, try the following:

find ./develop -cpio /dev/rmt/1m

Should be more efficient (acc. to man cpio)

[Darrel Louis]

Rita,

To identify what the blocking factor is, try the following:

time dd if=/develop of=/dev/rmt/1m bs=64k count=5000
time dd if=/develop of=/dev/null bs=64k count=5000

time dd if=/develop of=/dev/rmt/1m bs=256k count=5000
time dd if=/develop of=/dev/null bs=256k count=5000

By giving different Block-sizes you can test which blocksize is the best to keep your tape-drive in streaming mode. Writing to /dev/null will check I/O speed on disk.
You can test the following aswell.
time dd if=/dev/dsk/c?t?d? of=/dev/null bs=64k count=5000
etc.

[Pete Ellis]

tar has it own blocking factor, the default is 20. But when used thru a pipe it is set to 1. The max is 64; ignore the man page, enter tar cvfb /dev/null 65 /tmp to enter a wrong value and get the true max value.
I don't know if adding a blocking factor of 64 would help when using standard input, the man page states:
If the name of the file is -, tar writes to standard output
or reads from standard input, whichever is appropriate,
and the default blocking factor becomes 1.
But I don't always believe the man pages!

Also up'ing the blocking factor may only help with big files, the man pages state files over 2 gig can not be backed up, this I can believe!! or does anybody know better?

[Frederic Soriano · Accepted solution]

Hello Rita,

You could also have a look at the pax(1) command, as it can backup your data in either cpio or tar (more precisely 'extended tar format') formats.

Thanks to pax, you can overcome the tar blocking size limit (10240), and set it to a maximum value of 32256 in increments of 512.

I recently switched one of my backup methods from tar to pax (running on a DLT7000), and I saved a lot of time !

I hope this helps !

Best regards.

Fred.

[Dan Hetzel]

Pete,

I agree with you when you're saying that pipe uses a default blocking factor of 1 when writing to a pipe.

But, in the example given, that blocking factor will only impact the standard input of 'dd' and shouldn't impact the blocksize of what is written to the tape.

Rita,

The 64k blocksize is an example, you could try with higher values and see if it gives better writing stream.


Best regards,

Dan

[Pete Ellis]

Dan,

Sorry; my comment about tar's limitation was just in relation to tar itself; you are right about dd writing in 64k blocks in your example and it can go much higher.

Rita,

fbackup is quicker than tar or cpio, but files cannot be restored on non-HPUX systems, is this why you are using it?
Also with dd you can't restore individual files or dirs, I don't know if you can with pax.

[Dan Hetzel]

Hi Pete,

No need to be sorry.
It was no more than a clarification point.
If your name appeared at the top, it was to help Rita in seeing to what message this answer was related.

You were right, I said it !

Cheers ;-)

Dan

[Rita Li]

Thanks everyone for the great idea contributed. I have found solution to my problem & also found using 'pax -x cpio' gives me the best time with a block size of 32256. Dump files to tape is ~20G/hr now.

Thanks again.

-Rita