Backup time increased after tape drive replacement

[Roy Chenwoeth]

Environment: OpenVMS 7.3-2 on Alpha, TL892 tape library with TZ89 tape drives.

Problem: We recently had a tape drive replaced after a hardware failure. Since that time our backup times have increased about 25%. Previously backups took about 8 hours, now it's running 10-11 hours.

We backing up the same amount of information from the disks - actually slightly less. The backup script has not changed, and we seemed to be using the same number of tapes.

I fairly new to OpenVMS administration. I don't see how the hardware configuation would change.

I'm looking for any ideas if someone has seen this problem before.

Thanks,

-Roy-

[Hoff]

Was your old DLT drive also a TZ89 DLT drive?

Are you re-using existing DLT (DLT-4) media?

As for the immediate checks: check for increasing fragmentation on the input disks, for I/O contention on the input or output buses, and check the process quotas for the operator username.

For testing, make sure you use fresh, new CompacTape IV DLT cartridges. Alternatively, reinitialize the cartridges in the TZ89, specifying DLT-4 TZ89 35 GB with compression (compaction) enabled.

Is the front panel selecting TZ89 35 GB and compression? (Or more specifically, is the new TZ89 set the same as the old TZ89?) For settings, you can look here:

http://intelligent-peripheral.com/manuals/ek-tz89n-ug-a01.pdf

/BLOCK=32768 minimum and 65535 is probably better (and ensure your quotas suffice), enable caching, and enable compaction. And the process quota recommendations for the operator username are in the system manager's essentials manual in the OpenVMS documentation set:

http://www.hp.com/go/openvms/faq/

There are various discussions of BACKUP over in the OpenVMS FAQ, as well:

http://www.hoffmanlabs.com/vmsfaq/

I see I have a run-on section in BACKUP (discussions on data security erase and virtual terminals don't belong there), and I'll fix that for the next edition of the FAQ.

The other obvious approach would be to ask whomever installed the drive for help.

Stephen Hoffman
HoffmanLabs

[Martin Hughes]

How long has it been since the tape drive was replaced?. I am wondering how much evidence you have after the tape drive change to determine that the problem was triggered by this event. Also, were the backups reliably taking 8 hours previously? or did it vary?.

Are you running backups to both TL892 drives in parallel?. If so, do both backups take longer than previously?.

What I would do is try and determine whether backing up to this drive takes longer generally, or whether the increase in backup duration is related to only part of the backup. For instance, if you backed up 10 disks each night to this drive, is each disk backed up taking 25% longer? or can you isolate part of the backup where you have a sudden spike?.

The reason I ask this is to try and determine whether the problem is more likely hardware related, or a performance issue such as an I/O bottleneck.

[Peter Quodling]

Tape drived replaced and it got worse...

Did someone forget to reattach scsi terminators?

q

[Wim Van den Wyngaert]

May be your drive is more sensible to bad tape or even to new tapes.

As far as I now, compression and density is remembered ON the tape (e.g. if you do a init again, these values will be used as defaults).

If you have it with all backups I would ask to replace the drive again.

Wim

[Hoff]

I'd try to isolate the difference between the old and the new environment. Swapping is certainly one approach, and would certainly isolate whether it's the drive or something upstream.

If this is an up-rev'd tape drive model from the previous DLT drive, for instance, it might not be streaming effectively if the host SCSI controller can't feed it fast enough. TZ89 is toward the upper-end of fast, and some SCSI controllers can shovel data that quickly. Termination can be an issue. Process quotas. SCSI bus errors. SCSI contention. Cartridges running at lower densities, or differences in drive settings. Even down-rev device firmware.

[Simon Fedele]

I have seen this problem before with one of our systems with TZ89 tape drives. Check that the firmware versions on your drive have not changed after they were replaced. The firmware version that we use is 2561.

VMS0_SF-> anal/sys

OpenVMS (TM) system analyzer

SDA> clue scsi/sum

SCSI Summary Configuration:
---------------------------
SPDT Port STDT SCSI-Id SCDT SCSI-Lun Device UCB Type Rev
-------------- -------------- -------------- -------- -------- ------ ----
814FDA40 PKJ0

815C3000 PKI0
8170FF40 4
817100C0 0 MKI400 81714840 TZ89 2561
817151C0 5
81715340 0 MKI500 81714D00 TZ89 2561

815BAAC0 PKH0
81853B40 0
8183F740 0 GKH0 8184A600
8170F240 2
814EBE40 0 MKH200 816E03C0 TZ89 2561
8170F540 3
8170F6C0 0 MKH300 816E7000 TZ89 2561

[Roy Chenwoeth]

Thanks to everyone for all the input and suggestions. I'll try to answer every question in the order received.


===== Holf =====

The old drive was also a TZ89.

The backup script has not changed. We do have some fragmentation, but I don't believe this it changed significantly overnight.

COMPACTION is specified in the BACKUP command. I can't get the Compression->ON on the front panel to 'stick'.

The actual sequence of repair was:

-DLT0 failed

-HP did not have one on the shelf. We don't use DLT1, so the engineer swapped DLT0 with DLT1 to get us back in business.

-The next backup job, using the former DLT1, ran 2 hours longer. I don't know for certain, but have to assume that both drives we on the same firmware version. HP replaces the (now bad) DLT1 with a new drive a day later.

-After a day or so of long backup times, HP replaces DLT0 (formally DLT1) with a new/refurbished drive. Long backup times continue.



We are using new DLT tapes.


/BLOCKSIZE=65535




==== Martin Hughes ====


The drive DLT0 was replaced twice in the last week (see above for detail). The backup time increased from 8 to 10 hours over night. The current amount of data backed is basically the same.

We only use one drive, DLT0, for backup.



==== Peter Quodling ====

I checked the drive cage, and terminators are present.



==== Wim Van den ====

We are using new tapes. I did INITIALIZE a bunch of tapes yesterday with /MEDIA_FORMAT=COMPACTION specified. It looks the backup is still on track to run two hours longer. The DENSITY of the currently mounted tape is TK89.

The tape drive has been replace two times in the last week.


==== Hoff and Simon Fedele ====

The backup script and process has not changed, so quotas and other settings are the same.

The addresses are:

DLT0, SCSI-ID: 4, LUN: 0
DLT1, SCSI-ID: 5, LUN: 0


I did the ANALYZE/SYS and the current firmware on both drives is 296D.




==== All ====

I will try open another call with HP today. If anyone has any other ideas, let me know. I do appreciate everyone taking time to respond.


Thanks,

-Roy-

[Art Wiens]

Sorry, no answers, just questions:

If you can be present while a backup is occuring, what does it "sound" like? ie. can you hear the tape drive starting/streaming/stopping often? A block size of 65K should be able to keep it going fairly steadily.

Are the front panel indicators showing you are in a "compacted" mode? 40/80 on a TZ89 right?

How about error counts? $ show error before and after a backup ... climbing?

Is it possible to try the backup on the other drive?

And a huge stretch, can you ask HP for your old drive back just to try and verify that it is or isn't the drive? (Assuming it wasn't completely dead.)

It certainly wouldn't be the first time in history that two bad spare tape drives were swapped in! I've been on both sides of the screwdriver in that scenario!

I feel your pain, see my quote ;-)

Cheers,
Art

[Roy Chenwoeth]

--- Art --

Error counts are not increasing above 0.

--- All ---

After using many of your suggestions, I finally called HP support. As I said at the top, I'm fairly new -- and I found that this machine is way behind on patch levels in all areas. HP wants to get up-to-date on patches because the drive firmware is current, but the software is not.

So, I'll have to schedule some downtime.

Thanks all.

-Roy-