tar, ssh, and dd with Ultrium 1 tape

[Paul Tibbitts]

We have a 2-cpu x86 proliant, 32-bit rhel 3 kernel. It has a scsi card with an HP Ultrium 1 drive as the only device on that controller.

If I do a tar from the local disk to tape, default 10240-byte blocks, I get about 8mbytes/sec writing or reading (tar c/tvf.) That's fine for what we need.

If I do tar cvf - . | dd of=/dev/st0 bs=10240, and then read that back, I get about the same results.

But, if I do tar cvf - . | ssh localhost "dd of=/dev/st0 bs=10240", I get dramatically slower write speed. And surprisingly to me, reading a tape created using the ssh (reading without the ssh, using the tar tvf directly from /dev/st0) is even slower than writing!

Tar doesn't complain (or even comment) about block sizes, etc. and all the data is there, it's just too slow to be useful, by an order of magnitude. Obviously, sshing to localhost is pointless in this example, I just did that to eliminate one variable when I started using this computer for remote tars (from hp-ux 10.20, of all things) and encounted the performance problem.

I have tried various combinations of dd options (conv=block, etc.) with seemingly no improvement.

Any suggestions will be appreciated.

Paul

[Steven E. Protter]

Shalom Paul,

This problem could be made worse by ssh itself.

What version of ssh are you running? What is the networking environment as far as speed and such?

Those Ultrium drives are very, very fast and should be able to do a 50 GB Ignite style backup in about 20 minutes.

Its also possible the scsi drivers for Linux are not correct.What is the distribution of Linux you are using? Is the kernel up to date?

SEP

[Ivan Ferreira]

You can use star instead of tar to improve performance.

I think that your 8MB/s is very low, with LTO-1, you should get near to 34 MB/s.

The SSH adds overhead because of encryption.

[Paul Tibbitts]

I should clarify that

1. 8mb/sec is ample for what we need, but less than 800kbytes/sec is a problem.

2. the ssh is only being used to *write* the tape, not read it back in. Tapes written with the ssh pipeline read back in (not using ssh) 10x slower than tapes written without the ssh pipeline.

Star is an interesting suggestion, but given that the eventual objective is to back up various OSes that won't have star, I still need to solve the ssh problem. All I can count on is tar and ssh on the client systems.

I'm mainly interested in finding out how/why the data is being written differently due to the use of the ssh pipe and what I can do to fix it.

I'm guessing that dd isn't waiting for the pipeline to fill its input buffer, and writing partial blocks, although I'm not exaclty sure how the Ultrium drive handles that. If that's happening, should tar be able to restore the data without error (which it can)???

Paul

[Paul Tibbitts]

I forgot to answer the question about the Linux version. It's rhel 3, with very recent "up2date" patches. Kernel 2.4.21-37.ELsmp. Obviously while rhel 3 is still current in the support sense, Linux and RH have moved on. But we need rhel 3 and the 2.4 kernel for app compatibility.

I don't have an Ultrium drive on anything else so I can't test it with other OSes.

Paul

[Steven E. Protter]

Paul,

Try booting the system if possible on the non-SMP kernel.

I've been having some I/O problems with the SMP kernel under certain bizarre situations.

We suspect storage, but it still could be the kernel.

SEP

[Paul Tibbitts]

Thank you for the kernel suggestion, but it'll be a while before I can try it. Since I couldn't figure out what to do I set up rsh/rmt and the "real" remote backup is running now from the 10.20 system. It'll take a couple of days to make the tapes we need, given the 1.5mbytes/sec or so that we're writing at (which I don't think is that horrible, given the theoretical 10mbytes/sec network speed.)

The tapes written using rmt at 1.5mbytes/sec read locally on the linux box at the same speed as tapes written locally without the ssh pipe. Local reading/writing is about 8mbytes/sec, which as I said is fine for what we're doing.

Reading each ssh tape was going to take more like a week, so using rmt is providing a big improvement vs. the ssh pipeline.

I will update when I can experiment further.

Paul