topic Re: tape drive troubles in Operating System - HP-UX

tape drive troubles

Fred Martin_1 — Tue, 29 May 2007 11:53:13 GMT

I'm having trouble with a DLT 7000. I use cpio for file backups.

I issue a backup (cpio -ocuvB) to the tape, and that seems to go well.

Next, issue rewind (mt rew) before I read back for a verify, and it can't find the device.

ioscan -fn shows the device is claimed and lists the dev's etc.

# mt -f /dev/rmt/c4t6d0BEST rew
/dev/rmt/c4t6d0BEST: No such device or address
# cpio -ictBv < /dev/rmt/c4t6d0BEST
sh: /dev/rmt/c4t6d0BEST: Cannot find or open the file.
# ls -l /dev/rmt/c4t6d0BEST
crw-rw-rw- 2 bin bin 205 0x046000 May 29 11:36 /dev/rmt/c4t6d0BEST

A few weeks back the same thing happened, except that the ioscan was flaky - sometimes it would not claim the device.

So, using my HP support, and on HP's recommendation, I had the drive replaced.

The replacement worked fine a few weeks. Now it's doing essentially the same thing, except that the device always seems to be claimed.

Can anyone give me some insight or some ideas as to how to approach this?

The odd thing is that a backup (cpio -ocuvB) appears to "work", that is, it runs for the expected amount of time, and stops without error.

Subsequent rewinds or tape listings fail.

Thanks in advance,
Fred

Re: tape drive troubles

Steven E. Protter — Tue, 29 May 2007 11:57:49 GMT

Shalom Fred,

Could be another bad drive.

Check and replace cables, perhaps scsi card. These drives like being on their own exclusive scsi card, better throughput and reliability.

Also, I'd remove all drivers stape and tape drivers, rmsf, possibly re-arrange /etc/ioinit and start again.

cstm mstm xstm show no trouble?

Another idea: Bad or unsupported tape media.

SEP

Re: tape drive troubles

A. Clay Stephenson — Tue, 29 May 2007 12:44:28 GMT

I would lean towards a second failing drive but you also need to check your termination as well. Surprisingly, a poorly (or unterminated) SCSI bus will often work almost perfectly --- the worst kind of problem to have.

Here are the things that you should check:
1) Is the bus terminated in exactly two places? At the physical ends of the bus?
2) Is at least one device on the bus supplying termination power?
3) Does your total cable length exceed maximum bus length?
4) Most DLT-7000's were HVD (FWD) SCSI devices; if so, only one active DLT-7000 or 8000 is supported per bus.

If all of these conditions are met, I would replace the terminator(s) and see if the condition improves and then you are down to either the drive or the HBA --- or in very rare cases the cable --- but that is rare (but not unheard of) for intermittant problems.

Re: tape drive troubles

Fred Martin_1 — Tue, 29 May 2007 13:07:30 GMT

Thanks.

This drive is alone on the interface/bus, and is the only tape drive on the server.

I'm not thinking it's a tape issue - these are almost a year old and due for replacement but I've tried about 6 tapes, all same result.

I've never used the stm tool before, so that's been interesting. Selected the device easy enough. 'Information', 'Verify' and 'Exercise' all run clean after many iterations. Can't get 'Diagnose' to run, the interface (mstm) just beeps at me as though it's not an option that I can use.

At first opportunity I will re-seat the drive and cables, perhaps the interface card as well.

Re: tape drive troubles

Fred Martin_1 — Tue, 29 May 2007 15:07:13 GMT

This tape drive is alone on this bus (ioscan doesn't show anything else on the 0/7 hardware path). Is it safe to disconnect the drive, cables and terminator, without powering the server off? (assuming nothing is trying to access the drive)

Re: tape drive troubles

A. Clay Stephenson — Tue, 29 May 2007 16:57:49 GMT

While it is not recommended to do so and the manuals will always tell you to shutdown, I have removed/replaced/added hundreds of tapes drives on live systems without a single problem --- but perhaps "The Force" was always strong with me. Only in rare circumstances, can a production server be shutdown for such a trivial task as disk or tape drive replacement. The HBA is another story unless your hardware supports OLAR and I would never replace an HBA or NIC unless the machine and card are OLAR compatible.

Of course, you should at least power off the drive

Re: tape drive troubles

Fred Martin_1 — Wed, 30 May 2007 11:25:32 GMT

I powered off the drive, intending to re-seat cables, but instead on a whim I just turned it back on and tried another backup/verify with cpio. It worked OK.

If this trend continues I'll call HP about another tape drive.

Still, more tests are warranted.

Fred

Re: tape drive troubles

Fred Martin_1 — Wed, 30 May 2007 14:51:43 GMT

Seems I have to power-cycle the drive to get a couple of good hours of use, basically one good backup/verify, then it starts failing again as described above.

Power-cycle and it's good again for a while.

I still haven't reseated cables/terminator but that's next.

Re: tape drive troubles

Fred Martin_1 — Tue, 05 Jun 2007 09:56:13 GMT

Well, it appears that once again I have met the enemy and he is me.

This whole business appears to be caused by running out of tape space. It's asking for a second volume.

It can get about 19 GB on there, but more than that and it fails.

The drive is a DLT7000, C6375A.

The tape is an HP DLT IV, which should be OK for 40-80 GB and I'm only getting 20, it would seem.

The device I'm using is /dev/rmt/c4t6d0BEST

I have others installed:

/dev/rmt/0m
/dev/rmt/0mb
/dev/rmt/0mn
/dev/rmt/0mnb
/dev/rmt/c4t6d0BEST
/dev/rmt/c4t6d0BESTb
/dev/rmt/c4t6d0BESTn
/dev/rmt/c4t6d0BESTnb

Am I using the wrong one? I'm not sure what the suffixes stand for in the dev names.

I'm running a cpio right now, and the "Compress" and "35.0" lights are lit.

Fred

Re: tape drive troubles

Bill Hassell — Tue, 05 Jun 2007 13:09:11 GMT

The DLT 7000 has a native capacity of 35 Gb -- period. Compressed values, although widely advertised, are not meaningful without a specified data content. The drive stores exactly the same number of bits, whether compressed or not, it's just the encoding of the bits that allows repeated patterns to take less space on the tape.

ALL of the device files you listed are compressed. You decode the device files using the lssf command and look for the words: "best density available". For modern tape drives (DLT, DDS, LTO, etc) this has nothing to do with density as it did in the ancient times of 1/2" reel-to-reel.

The problem with running out of tape is very likely your measurement method for the backup. If you save tens of thousands of files, there will be a high overhead relative to the size of the files, thus contributing to a lower value. Also, if you don't keep the data moving at full speed, you'll get terrible performance. cpio (and tar and pax, etc) are simply not suitable for high speed tape drives. Try fbackup and be sure to use the config file for best performance.

Re: tape drive troubles

Fred Martin_1 — Tue, 05 Jun 2007 13:36:36 GMT

Thanks, I'm off to read the fbackup man pages and anything else I can find about it.

As an administrator someplace else, I used it, but the scripts were written by others and frankly I never took the time to see what it all was doing.

Fred

Re: tape drive troubles

Fred Martin_1 — Tue, 19 Jun 2007 14:10:09 GMT

By the way, I have this DLT7000 alone on a bus. Using cpio, how long would you expect it to take to backup about 17 GB (just the write, no verify)?

Mine seems to be running very slow; I re-seated cables but that didn't seem to make any difference.

Re: tape drive troubles

Bill Hassell — Wed, 20 Jun 2007 10:04:00 GMT

> DLT7000 alone on a bus. Using cpio, how long would you expect it to take to backup about 17 GB

cpio is a single threaded application so it has to read a record from disk, select the tape, write a record to it, wait for a status and then repeat. The old DLT7000 will likely never run a full speed, or more accurately, will run full speed for a few records, then stoip, back up, and wait for more data. tar, cpio, pax, dump, etc are not designed for modern tape drives and will be unable to handle large files (more than 10Gb).

If the drive is kept busy, you can backup about 18 Gb/hour with uncompressible data, perhaps more if the data can be compressed by the drive. Anything less and your backup program is the problem. The only built-in program that can keep the drive busy is fbackup, but only if you use a config file like this:

blocksperrecord 4096
records 64
checkpointfreq 4096
readerprocesses 6
maxretries 5
retrylimit 5000000
maxvoluses 200
filesperfsm 2000

---------------------------------------------

Example for a complete backup starting at / (root):

fbackup -i / -v -c config-file -f /dev/rmt/0m

Display the tape header with dates:

frecover -V - -f /dev/rmt/0m

Display the table of contents:

frecover -I - -f /dev/rmt/0m

Re: tape drive troubles

Fred Martin_1 — Wed, 20 Jun 2007 11:44:11 GMT

Thanks Bill.

It's taking me 5-6 hours to backup 17 GB right now, using cpio. I looked back over several years worth of records, and it used to be a little quicker (4 hours) but that's it.

The bulk of the data are Progress database files, and online packed backups of same.

At first I was thinking that there was a cable or hardware problem but I doubt that now.

I've never used fbackup but will give it a shot.

Re: tape drive troubles

Fred Martin_1 — Fri, 20 Jul 2007 07:43:39 GMT

fbackup definitely runs faster, and gets more data on a single tape, than cpio - at least with my particular data.

I have some questions about fbackup and cpio, which I'll put in a new thread.