Explanation of records and blocks

[Bruce Baillie]

I am doing backups using find, cpio | dd to a DDS-3 tape which holds 24GB compressed. When the backup is finished, dd reports 18777+1 records out and the backup file reports 38456242 blocks. How do I convert records and/or blocks to bytes?

[A. Clay Stephenson]

Without knowing more you can't. What did you specify as the output block size and input block size for dd? Judging from your statistic i would guess that you specified an obs=10240k. The 18777+1 records indicates that you output 18777 full blocks (of obs size) and 1 partial block.

[Bruce Baillie]

I use obs=1024k

[A. Clay Stephenson · Accepted solution]

Okay, now we know enough to answer:

The backup file is 38456242 blocks X 512 bytes/block = 19689595904 bytes

dd output 18777 blocks X 1024 * 1024 bytes/block = 19689111552 byes + a portion of one 1024KiB block because the input file size was not an exact multiple of 1024KiB.

[Bruce Baillie]

Thanks, just what I needed.

[Bill Hassell]

And to clarify: The DDS-3 only holds 12Gb, period. If you backup data that has 90% zeros in every file, then the DDS-3 can store more than 100 Gb, but random data is simply not compressible so there will be no compression. The 2:1 compression value is an old marketing number from a decade ago and assumes that the data patterns are compressible to about 2:1. dd is just taking raw data from the disk so there may be both random and highly compressible data. In general, vg00 from a standard HP-UX system compresses to about 1.3:1.

[Bruce Baillie]

Bill, thanks for the information. The backups I am doing are file systems that contain CAD data only. If the command, compress, is any indication of how much these files can be compressed when written to a DDS-3 tape, I found it to be over 2:1. That is the reason I am getting almost 20GB on a single tape.

[Tim Nelson]

Bruce just out of curiosity why are you using dd ? raw disk ?

fbackup, cpio, tar are all much better for tape control, standard blocking, header info and volume sets?

A restore from a dd will require that the person knows all the switches and blocking factors used.

[Bruce Baillie]

Tim, The scripts were written long before I got here. I have been here 10 years. They run from cron hourly and daily. The monthly is done from a menu as are recoveries, if we need to recover data. If I make changes, and I am not here if someone needs to recover data, then confusion starts. Everything is well documented so even a cave man can do it. I just decided to leave things as-is, it works.

[Bruce Baillie]

Tim, oh, by the way,the script does a find, pipe to cpio then pipe to dd.

[Bill Hassell]

Well, as seasoned sysadmins say: dd uses the "send-and-pray" protocol. No individual file recovery, no error retry of recovery from a tape error, no way to identify what was saved or when it was saved (except for a sticky label and exceptional attention to recording the details on every tape), etc.

As far as compressibility, you can sort of guess at the compression ratio by running a dd and piping the output into compress then into wc -c. The compress utility probably doesn't use the same algorithm but it will be close.

By the way, when was the last time these tapes were tested by performing a restore?

[Bruce Baillie]

Bill, Here is the script, edited down. You can see that cpio creates a listing of the tape before it goes to dd.

find /dir1 /dir2 /dir3 /dir4 /dir5 -depth | cpio -ov 2> /daily_files/`date -u +%d%b%y`.data | dd of=/dev/rmt/2m obs=1024k

I know there is a risk with magnetic media and these are backups not archive. A couple of years ago I had to trash about 100 9 track tapes from a VMS system that were over 20 years old. I recovered all the data and put it in CDROM's.

[A. Clay Stephenson]

At the very least I would change those absolue paths to relative paths, so that:

find /dir1 /dir2 /dir3 /dir4 /dir5 -depth | cpio -ov 2> /daily_files/`date -u +%d%b%y`.data | dd of=/dev/rmt/2m obs=1024k

becomes

cd /
find ./dir1 ./dir2 ./dir3 ./dir4 ./dir5 -depth | cpio -ov 2> /daily_files/`date -u +%d%b%y`.data | dd of=/dev/rmt/2m obs=1024k

so that restore to alternate locations become at least an order of magnitude easier.
The dd'ing is being done to better block the i/o so that it streams better.

[Bruce Baillie]

Thank you for the continued suggestions and help, I really do appreciate it.

[Dennis Handly]

>Clay: I would change those absolute paths to relative paths

You can always use pax(1) to change the paths so it might not be worth this change?