fbackup problems

[Carlos A. Munoz Lopez]

Hi guys. I'm using fbackup to backup a 200GB file system to an Ultrium 960 tape drive (LTO 3 SCSI, 800GB) but lately I'm having some problems. In SAM I created an automated backup scheduled to run every day at 02:00pm, so it means it's part of job described in cron. When the backup finishes and check mail, it displays the following:

br_backup: Invoking fbackup. See /var/sam/log/br_log for details.
fbackup(1004): session begins on Mon Apr 2 16:02:00 2007
fbackup(3203): volume 1 has been used 1 time(s)
fbackup(3024): writing volume 1 to the output file /dev/rmt/c13t3d0BEST
fbackup(3003): normal EOT
fbackup(3310): enter '^[yY]' when volume 2 is ready on /dev/rmt/c13t3d0BEST,
or '^[nN]' to discontinue:
fbackup(3004): writer aborting
fbackup(1002): Backup did not complete : Reader or Writer process exit

As I mentioned above, I'm using a 800GB tape, and the size of the file system is barely 200GB. The message is telling me I need another tape or volume so it can finish the backup. How could this be?? It doesn't make any sense. As far as I know, fbackup does not allow to append several sessions in a single tape, it has to rewind the tape every time a backup is done, so the next session it would overwrite the content on that tape. I don't know if fbackup assign a protection period as used by other backup tools such as DataProtector.

I've tried with different tapes and I get the same result. I would like some help on this, because I don't know what's happening. Thank you in advanced.

[A. Clay Stephenson]

You don't bother to identiy your OS version so it's difficult to look for specific patches but the very first thing that I would do is look for and apply the latest tape, scsi, and fbackup patches that are available for your OS release. Unlike the more "vanilla" backup commands such as tar or cpio, fbackup must use more advanced commands to properly position the tape and thus is very sensitive to having the latest driver software.

[Steven E. Protter]

Shalom,

Well the system seems to think the tape is full. The media may be bad and you may simply need to try a different tape. You may have stuff on the tape and your setting are telling it not to erase the tape.

More likely though is a hardware problem on the tape drive itself. I'd run cstm/mstm or xstm hardware diagnostics and see if anything comes up.

It is possible you are using non-supprted media. Just because it says 800 GB does not mean the tape recognizes this fact. Only certain 800 GB tapes are going to work.

Last idea I have is perhaps you need to recompile the kernel again with the stape driver. Take it out of the kernel, recompile and boot and put it back in. That two boot process has cleared up these issues for me in the past.

The system itself should be patched to a level of a recent (within the past year) bi-annual patch set.

SEP

[Tim Nelson]

Initially looks like a bad tape or drive.

How long before the error occurs ? You noted that the job starts at 2pm but the log entry says 4pm.

Run it manually so you can better watch it.
fbackup -g graph.file -f /dev/rmt/Xm

The 800GB capacity of the tape is compressed.

Are you really using LTO 3 tapes ?

[Steven Schweda]

I know nothing about your tape drive, but my
tape drives can write at more than one
density. Without seeing your actual fbackup
command, it's hard to say what you're doing.

[James R. Ferguson]

Hi Carlos:

If your files are changing as you attempt to back them up, then 'fbackup' will consume additional tape.

By design, 'fbackup' will retry the backup of any file 'maxretries' times if the file has been found to have changed since the inception of its backup to tape. The copy of the file on tape is marked "bad" so that it cannot be recovered, but the tape cannot be rewound so it is wasted. By default, five (5) retries per file can occur. This can cause multiple tape reels to be required.

Regards!

...JRF...

[Carlos A. Munoz Lopez]

Ok. Sorry, I forgot to mention I'm running a HP-UX 11.23 in a rx4640 server, with latest patches from September 2006 (Feature Enablement, HW Enablement and Quality Pack).

Steven:

I think tape couldn't be bad because it's a new media, and I've also tried different tapes. I checked the tape drive using stm and there's no problem with the tape drive. About the type of media, the Ultrium 960 drive uses LTO 3 tapes 400GB native (800GB compressed) so I'm using the right media according to product specifications.

Tim:

This is the usual time, it takes about 2 hours to get the 200GB backup done.

James:

The files that are being backed up aren't changing at all. These files are the ones generated by Oracle RMAN, so DBA has to write them first on disk then on tape, because we lack of a professional backup tool (DataProtector, TSM, Veritas NetBackup, etc.) so we don't have any media manager to send the RMAN backup directly to tape. As you can notice the backup itself is not marked as bad, but it needs an additional volume to be completed sucessfully, there's no HW error reported. Since it's not an interactive backup, there's no way we can know the requirement of this additional volume because it's running in background.

This is the configuration of the backup generated by SAM:

fbackup -0 -u -g /var/sam/graphFCAa13131 -I /var/sam/log/br_index.full -c /etc/sam/br/fbackup_config -d /var/adm/fbackupfiles/d
ates -f /dev/rmt/c13t3d0BEST

Graph:
i /backup/tcbp

Index:
8192 1 /
1024 1 /backup
3072 1 /backup/tcbp
7455768576 1 /backup/tcbp/TCBP_20070402_b9ie68hi_1_1
8177778688 1 /backup/tcbp/TCBP_20070402_baie68n2_1_1
6904725504 1 /backup/tcbp/TCBP_20070402_bbie68ss_1_1
6316023808 1 /backup/tcbp/TCBP_20070402_bcie691n_1_1
8502165504 1 /backup/tcbp/TCBP_20070402_bdie696i_1_1
8594407424 1 /backup/tcbp/TCBP_20070402_beie69cb_1_1
7432765440 1 /backup/tcbp/TCBP_20070402_bfie69h7_1_1
7891558400 1 /backup/tcbp/TCBP_20070402_bgie69mc_1_1
8663023616 1 /backup/tcbp/TCBP_20070402_bhie69rr_1_1
6326288384 1 /backup/tcbp/TCBP_20070402_biie6a1b_1_1
7938162688 1 /backup/tcbp/TCBP_20070402_bjie6a5i_1_1
8412389376 1 /backup/tcbp/TCBP_20070402_bkie6abl_1_1
7858692096 1 /backup/tcbp/TCBP_20070402_blie6ai2_1_1
7458627584 1 /backup/tcbp/TCBP_20070402_bmie6ans_1_1
7907991552 1 /backup/tcbp/TCBP_20070402_bnie6atl_1_1
23305945088 1 /backup/tcbp/TCBP_20070402_boie6b3e_1_1
22568370176 1 /backup/tcbp/TCBP_20070402_bpie6ble_1_1
19602571264 1 /backup/tcbp/TCBP_20070402_bqie6c66_1_1
23489085440 1 /backup/tcbp/TCBP_20070402_brie6cmj_1_1

Config:
blocksperrecord 32
records 32
checkpointfreq 32
readerprocesses 2
maxretries 5
retrylimit 5000000
maxvoluses 100

Thanks.

[Steven Schweda]

"-f /dev/rmt/c13t3d0BEST" sounds good, but
only if the OS knows what "BEST" really means
for this tape drive. It's not clear from a
quick look at the specifications, but if the
drive can write at a lower density than the
tape allows, and if it's being told to do
that, then that could explain things.

Old DLT drives had LEDs on the front to tell
you the density while it was working. I
don't know how you could verify the density
on one of these drives. If you get desperate,
you could try some different
"/dev/rmt/c13t3d0xxx devices to see if you
can find one which will put more bits on a
tape. (This could take a while.)

[Bill Hassell]

Actually, modern tape drives only have one density. The BEST terminology is a throwback to the days of reel-to-reel tapes and simply means: if the drive has the ability to compress data, then do it. The density is exactly the same, just less data is written for a given pattern. You don't have an 800 Gb tape despite all the marketing hype to the contrary. You have a 400 Gb tape and it will only store 400 Gb -- period. If the data is compressed, the bits going to the tape will be 400 Gb but with the special patterns, repetitive data can be expanded to more than 400 Gb.

Now, the Ultrium 960 is the fastest tape drive HP currently makes and it is almost always starved for data. Your system (CPU, RAM, I/O channel, backup software, etc) cannot keep up with the tape's speed. If you have 2:1 compressible data, the tape will backup 400 Gb in an hour. So your backup should take about a half hour. But you are starving the drive and unlike earlier models, the drive can slow down to as slow as 27 Megs/sec or 97 Gb per hour (200 Gb = 2 hours rather than 30 minutes)

So the first problem is with the config file -- it is badly setup for LTO drives. Use this config file:

blocksperrecord 4096
records 64
checkpointfreq 4096
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

The Ultrium 960 is extremely fast and needs to be kept busy or it will slow down and perhaps stop/restart which can severely wear down the mechanism Try the above config file on the command line and you may find that the tape starts performing much better. NOTE: DO NOT put the tape in the same I/O channel as your disks. It will drastically slow down the backup speed.

To answer some other fbackup questions:

> normal EOT

Yes, the end of the tape has been reached. Excessive resyncs and busy files may be the problem as well as a bad config file.

> backup does not allow to append several sessions in a single tape

Absolutely correct. The tape is always rewound so the tyable of contents at the front is always accurate.

> I don't know if fbackup assign a protection period...

No. fbackup keeps a record of the tape usage but all tapes are always available and can be read or reused at any time.

[Bob_165]

I've had this happen with a DAT drive. It's a bad tape head..If your on warranty, swap it out

[Raynald Boucher]

Just a thought,

I'd first try using drive /dev/rmt/c13t3d0BESTb for compression (or /dev/rmt/xmb instead of /dev/rmt/xm).

Next, if you have Ultra320 SCSI controllers and/or A7173A PCI-X Dual Channel Host Bus Adapters, you may need to (re)configure them. See "man mptconfig"

RayB

[Carlos A. Munoz Lopez]

Thanks Bill for your recommendations, it really decreased the backup time, now it takes about 20 minutes in performing the backup, that's a lot faster than I expected. Thanks a lot. We'll see if this new configuration also help to solve the problem with the requirement for the additional volume. I'll let you know. Thanks.