topic how to make TAR to run faster in Operating System - HP-UX

how to make TAR to run faster

Steven Chen_1 — Wed, 23 May 2001 19:46:03 GMT

Hi,

I have 7G data needed badly to extract from a 12G tape drive. It runs forever. The server is K260 with 2 cpus. Not much other processes going on.

I guess there is some command to make tar to be running faster.

Thanks.

Re: how to make TAR to run faster

Patrick Wallek — Wed, 23 May 2001 19:55:51 GMT

As far as I know there is not much way to speed up tar. The only thing I can think of that might help a little is to NOT use the verbose '-v' option when extracting. Other than that you'll just have to let it run. It will finish eventually.

Re: how to make TAR to run faster

A. Clay Stephenson — Wed, 23 May 2001 19:59:29 GMT

Hi Steven,

Try this, we are going to use dd to icease the input blocking:

dd if=/dev/rmt/0m ibs=2000b obs=1b | tar xvf -

This will reblock input to 2000 512 byte blocks.

You may need to adjust the obs to perhaps 20b depending on the original tar blocking factor.

Before you actually extract, I would pipe the dd to a tar vtf - to attempt a listing first. That way you can adjust the obs setting until you get consistant results. You can kill (or interrupt) the process until you get the settings right. Again, i would start with obs=1b, then try 20b.

This should speed up the process by at least 30x.

Clay

Re: how to make TAR to run faster

Steven Chen_1 — Wed, 23 May 2001 20:00:51 GMT

Patric,

Thanks for the help.

Would "tar x ...." just enough?
or "tar xf ..." if not use xvf?

I am using xvf that is as slow as whatever.

Re: how to make TAR to run faster

John Poff — Wed, 23 May 2001 20:01:40 GMT

If the tape drive and the disk drive are on the same SCSI bus I guess there could be a traffic problem as you might be reading and writing the data on the same bus. The only way I know to get around that is to somehow read the tape from another system and send the output through the LAN. I've thought about doing that before but I've never tried it.

Re: how to make TAR to run faster

Patrick Wallek — Wed, 23 May 2001 20:07:06 GMT

Yes you could use 'tar -x filetorestore' or 'tar xf /dev/rmt/?m filetorestore'. The 'v' option is what causes everything to display to your terminal, so it could slow you down.

Clays solution looks interesting. I may have to play with that myself.

Re: how to make TAR to run faster

Steven Chen_1 — Wed, 23 May 2001 20:08:57 GMT

Clay,

Thank you for the thought.

Those 7G data are Oracle datafiles used to recover database. I would need to make sure any method would not cause data corruption.

I was using tar to backup, so I use tar to extract.

Re: how to make TAR to run faster

Steven Chen_1 — Wed, 23 May 2001 20:55:02 GMT

another question related:

I use oracle user to tar cvf file. While I tar xf extracting file, would I have to use oracle user too? Would root account run faster? Or that would change the file bits?

Re: how to make TAR to run faster

Carlos Fernandez Riera — Wed, 23 May 2001 23:42:27 GMT

Talking about blokages:

Use tar cvbf 64 /dev/rmt ...

Or tyr pax.

Re: how to make TAR to run faster

Bill Hassell — Thu, 24 May 2001 00:00:55 GMT

tar is not in itself the problem unless your machine is very slow (less than 50 Mhz). tar reads the file as fast as the opsystem allows and outputs the result to the tape. To see just how fast tar is, change the output tape device to /dev/null...it should complete in a very short time.

So the problem is getting the data to the tape fast enough to prevent data starvation. Like all streaming drives, if the data does not keep ahead of the device, the drive stops, backs up and resyncs, a very costly task (wastes a lot of time).

The tape drive must be on a dedicated bus or on a bus with no activity on any other connected peripheral. If the tape and the disk are on the same bus, there is nothing you can do to improve performance.

It's also important to note that tar (pax, cpio, dump) is NOT recommended for large data backup tasks. It is a couple of decades old and designed for reel-to-reel tape drives holding a whopping 150 megs. Industry standard tar can never handle a file bigger than 2 Gb and has no error recovery procedures. If this data is important, strongly consider a commercial quality backup program. Or as a minimum, use fbackup which does handle large files and has error recovery built-in. Do not use fbackup without a config file to set large records and a large number of helper proceses.