topic Re: DDS3 TAPE Compression in Operating System - HP-UX

DDS3 TAPE Compression

Joseph C. Denman — Mon, 05 Feb 2001 18:26:40 GMT

Need a little help folk!!!

Attached is an ioscan of my tape devices. I have a DDS3 internal tape and a DDS3 autochanger. I'm using the autochanger, but the same results on both. I am using 'HP C5708A DDS-3 Data Cartridge, 24GB' tapes. I attemp to tar around 12GB of data to a tape and it runs out of space. I can only get about 6GB to the tape. I am using /dev/rmt/1m. Is this the device I should be using? Will one of the other give me better compression? Do I have the correct devices loaded?

Any help would be appreciated.

...jcd...

Class I H/W Path Driver S/W State H/W Type Description
======================================================================
tape 0 10/4/8.3.0 tape2 CLAIMED DEVICE HP C1537A
/dev/diag/rmt/c1t3d0 /dev/rmt/0mn /dev/rmt/c1t3d0BESTb
/dev/rmt/0m /dev/rmt/0mnb /dev/rmt/c1t3d0BESTn
/dev/rmt/0mb /dev/rmt/c1t3d0BEST /dev/rmt/c1t3d0BESTnb
tape 1 10/12/5.0.0 stape CLAIMED DEVICE HP C1537A
/dev/rmt/1m /dev/rmt/1mnb /dev/rmt/c4t0d0BESTn /dev/rmt/c4t0d0DDSb
/dev/rmt/1mb /dev/rmt/c4t0d0BEST /dev/rmt/c4t0d0BESTnb /dev/rmt/c4t0d0DDSn
/dev/rmt/1mn /dev/rmt/c4t0d0BESTb /dev/rmt/c4t0d0DDS /dev/rmt/c4t0d0DDSnb

Re: DDS3 TAPE Compression

Andy Monks — Mon, 05 Feb 2001 19:59:47 GMT

6gb sounds a bit small. Do an 'lssf /dev/rmt/1m' and ensure it says 'BEST' somewhere in the description (which it should). If it doesn't, use the 'BEST' device file.

Also, what size are your files? It is lots of very small files? If so, tar etc really hate those, but I wouldn't have thought it hates them that much.

Re: DDS3 TAPE Compression

James R. Ferguson — Mon, 05 Feb 2001 20:03:24 GMT

Hi:

The amount of data that can be copied to a tape is highly dependent upon the data. The best compression will be achieved for strings of repetitive data.

You will obtain the most-for-the-money by using the 'BEST' device files. Normally these are already linked to '/dev/rmt/?m'. You can verify that you are using the best density (compression) by comparing the minor numbers of the device files for "?m" with "c?t?d?BEST" and noting identical values. For example, 0x030000 for /dev/rmt/0m as well as for /dev/rmt/c3t0d0BEST would denote equivalent devices. You can also verify that compression is available by doing an 'lssf ', as for instance, 'lssf /dev/rmt/0m'.

The presence of 'sparse' files (common for databases) can consume considerable amounts of tape storage. You can be mislead when evaluating space requirements with 'ls' or 'bdf'. Whereas 'ls' shows the apparent file size (which for sparse files will be smaller than actual); 'du' will return the actual file size (in 512-byte blocks).

...JRF...

Re: DDS3 TAPE Compression

Joseph C. Denman — Mon, 05 Feb 2001 21:06:32 GMT

Thanks guys, but everything looks as is should.

The files are oracle datafiles and configuration files from cold backups. I have gzipped all files. I do a cold backup every week in a respective directory. Each weekly backup directory is a little over 3GB after gzip. 'du -sk directory is around 3200000'. I have about 4 weeks of backups I'm attempting to put to the tape. I can only get 2 weeks worth to a tape. If I try 3 weeks, I get the old end tape message.

Any other ideas on why I can only get 6GB on the 24GB tapes?

Re: DDS3 TAPE Compression

paul courry — Mon, 05 Feb 2001 23:35:08 GMT

Yank the drives out of the machine, check the DIP switches on the bottom. You might have compression forced off.

Set the one that has problems so it looks like the one that works.

Re: DDS3 TAPE Compression

Boyd Johnson — Tue, 06 Feb 2001 01:22:01 GMT

In this case the data has already been compressed with gzip, so you are going to be getting virtually no additional compression on the drive. The jumpers won'd make any difference. I believe DDS-3 stores 12 GB native. That doesn't explain why it runs out of tape after 6 GB unless you are storing many files and there is a lot of wasted tape space.

Re: DDS3 TAPE Compression

Andy Monks — Tue, 06 Feb 2001 08:45:07 GMT

I think your sufferring from the problem of trying to compress, compressed files.

If your backing up gzip'ed files and then the tape drive is trying to compress them, well it can't. In fact, it does worse and probably makes them bigger!.

Just try doing this for a test :-

cd /tmp
cp /stand/vmunix /tmp
gzip /tmp/vmunix
compress -v /tmp/vmunix.gz

You'll see that the compress command reports a negative compression and doesn't then compress it. Not sure tape drives are that clever.

Therefore, you could try using a non-compression device file (or just not gzipping the files in the first place). Either should work.

Re: DDS3 TAPE Compression

Carlos Fernandez Riera — Tue, 06 Feb 2001 09:54:27 GMT

1- Do a rmsf fo each /dev/rmt special file and recreate using insf -e -C tape. Your dev/rmt/1m* and linked files seem to force write in DDS format.

2- Notes about using compresion device with compresed file are absolute true.

3- Tape drives need to be very busy, to avoid stream. For this reason block size must be increased. Use tar -cvfb /dev... 64 or try pax. For comparation do a dd of one of your files and tar too. Check also busy time for disks.

4- I have put some times a program on this forum (dat3) that does a full report of drive statistics; search for Riera and dat3.( i would like if someone has checked, for me it is a good utility).

5- DDS3 tape is 12 GB native.

Re: DDS3 TAPE Compression

Joseph C. Denman — Tue, 06 Feb 2001 15:34:05 GMT

I love this forum!!!!!

Thanks for all the info. I was wrong about the amount going to the tape. It is closer to 10GB. I am sure it compress to compress that is using up the other 2GB.

Thanks again!!

...jcd...

Re: DDS3 TAPE Compression

Dave Wherry — Tue, 06 Feb 2001 18:28:32 GMT

Those numbers you see for a DDS drive are marketing numbers. With all conditions perfect in the lab you may be able to achieve that data capacity. I've never seen a DDS drive get very close to the marketing numbers in the real world. Since you are getting 10GB out of a possible 12GB, that's very good. I've had a 1GB file system that would not fit on a 2GB tape.
You've already compressed the data so it is dense and you get more on the tape.
There's one other area that wasn't mentioned, and this especially applies to DDS drives, tape quality and how clean the head is. An old, used tape and or a dirty tape head will greatly affect the amount of data you get on that tape. A brand new tape is typically dirty with residue from the manufacturing process. After it has been used a couple of times most of that will transfer to the tape head. The head needs to be cleaned. Your autoloader might have an LCD screen which will give you statistics as it is being used. See if it shows you the compression ratio. It will likely change with different tapes.
About Carlos' third comment on streaming, that's really a performance and not a capacity issue. You want to stream. Continually push data to the tape so it is moving forward. It it is not getting enough data a drive will stop and restart. It may even rewind a little to reposition. That's called shoe shining and slows down the backup. Keep it streaming for best performance.

Re: DDS3 TAPE Compression

Carlos Fernandez Riera — Thu, 10 May 2001 13:24:59 GMT

Dear Dave:

A non optimal speed trasfer will force to pad blocks ( and repositionig tape). This will cause capacity decrease directly. Reposition on tape will also cause degradation of surface that will decrease again capacity amount on tape.

This question had been checked by both theoretical (manuals) and (years) experieced ways.