DLT8000 BACKUPS TWICE AS SLOW AS TZ88(DLT4000) BACKUPS??!!

[Jessie Bunker-Maxwell]

We recently purchased a Quantum DLT8000 40/80GB external table-top tape drive(TZ-90) to replace our DEC TZ-88N-TA (DLT4000) external table-top tape drive.

We were expecting our nightly image disk backups to take about half as long with the new drive as they did with the old drive, but instead the backups are taking twice as long and I can't figure out why.

Here's the info I think is pertinent.

We have an AlphaServer 4100 running OpenVMS 7.2-1. We can't upgrade the hardware or the O/S - we need to stay where we are for too many reasons to go into here.

We do image backups nightly from five logical drives including: one 4GB Raid1, one 18GB Raid1, and three 4GB non-RAID disks.

Our old DEC TZ88N-TA has an uncompressed transfer rate of 1.5MB/sec and a compressed transfer rate of 3.0MB/sec. The new Quantum DLT8000 has a native transfer rate of 6MB/sec and a compressed transfer rate of 12MB/sec.

We have a KZPDA-AA FWSE SCSI card on our AlphaServer 4100 and I checked with HP Support to make sure that the DLT8000/TZ-90 would be compatible with that card and was told it would be.

We have always used the 20GB setting and the tape drive "Compress" function in combination with the backup utility INIT/MEDIA_FORMAT=COMPACT command to do compressed transfers on the TZ-88 to 20/40GB DLTIV tapes.

Backups using the TZ-88 tape drive complete in about 6.5 hours.

We bought new 40/80GB DLTIV tapes when we bought the DLT8000 drive.

I set the SCSI ID on the DLT8000 to 5 which is the SCSI ID setting of our TZ-88. I did a shutdown/power-off of the AlphaServer, connected the DLT8000, rebooted it and the system immediately recognized the DLT8000 as MKC500: as it had recognized our old tape drive.

My first test of the DLT8000 drive was done with the same backup commands that I have always used, which are:

init/media_format=compact MKC500: backup
mount/for/unload MKC500: backup

backup/image/list=san$bck:dra1_daily_image.lis/igno=(inter,label)/rec/nocrc dra1: mkc500:back1/sav/fast

and so on for the other four logical disk drives

In the second test, I changed the init command to this:

init/density=tk89/media_format=compact MKC500: backup

Since we are running such an old version of OpenVMS, there are no density settings specifically for the DLT8000 or the SDLT 220 or 320, so I thought I'd use TK89 as the closest possibility.

I selected both 40GB and COMPRESS on the DLT8000 drive for both backups.

I used the new 40/80GB DLTIV tapes for both tests.

Both backups took about 12 hours.

I don't know where to look for the cause of the slow backups.

Any suggestions would be appreciated.

Also included below are the valid density settings in 7.2-1.

Thanks much.

Jess

DEFAULT Default density
800 NRZI 800 bits per inch (BPI)
1600 PE 1600 BPI
6250 GRC 6250 BPI
3480 IBM 3480 HPC 39872 BPI
3490E IBM 3480 compressed
833 DLT TK50: 833 BPI
TK50 DLT TK50: 833 BPI
TK70 DLT TK70: 1250 BPI
6250 RV80 6250 BPI EQUIVALENT
NOTE: Only the keywords above are understood by
TMSCP/TUDRIVER code prior to OpenVMS Version 7.2.
TK85 DLT Tx85: 10625 BPI - Cmpt III
TK86 DLT Tx86: 10626 BPI - Cmpt III
TK87 DLT Tx87: 62500 BPI - Cmpt III
TK88 DLT Tx88: (Quantum 4000) - Cmpt IV
TK89 DLT Tx89: (Quantum 7000) - Cmpt IV
QIC All QIC drives are drive-settable only
8200 Exa-Byte 8200
8500 Exa-Byte 8500
DDS1 Digital Data Storage 1 - 2G
DDS2 Digital Data Storage 2 - 4G
DDS3 Digital Data Storage 3 - 8-10G
DDS4 Digital Data Storage 4
AIT1 Sony Advanced Intelligent Tapes

[Jon Pinkley · Accepted solution]

Jess,

The DLT8000 should be faster, as long as you can provide the data to it fast enough.

Note that if you don't specify blocksize when writing to tape, it will use 8192 byte blocks, and DLT performance will be poor.

If you don't need to be able to copy savesets from tape (using copy), then use /blocksize=65024/group=0. if you want to be able to copy the savesets using copy, you will have to use a /blocksize=32256

How long does it take to backup to the nl: device?

If backup can't keep the DLT8000 streaming, then the DLT8000 will have to go into "shoeshine" mode, and that is a real performance killer.

It is possible that the KZPDA (Fast Wide Single Ended) is a limiting factor. Make sure you have a short SCSI cable (I can't remember the limits for FAST SE, but it would be best if it was 1 Meter or shorter, although 2 Meters may work). FW SCSI should in theory be sufficient, (20 MB/sec) but that's the limit, without overhead, so 12 may be pushing the limits.

We used a KZPCA (Ultra 2 Wide LVD) adapter with our DLT8000 on an ES40 6/667 and didn't have performance problems.

The DLT8000 comes in two interface configurations, LVD/SE or HVD. You must have the LVD/SE version (or it wouldn't work with the KZPDA), so it is compatible with the KZPDA, but the KZPCA U2 W LVD would be better if the 4100 and 7.2-1 support it. But before spending the ~$150 for a used KZPCA, you need to verify that the adapter is really the bottleneck.

You should have a dedicated SCSI adapter for the DLT8000, at least there should not be any other active device in use on that adapter while you are backing up to the DLT8000.

If BACKUP can't supply data to the NL device at a sustained 12 MB/sec, then putting in a faster SCSI adapter isn't going to help (much).

Time the following operations for a baseline.

To measure the "best case" disk read speed.

(consider turning on image accounting for timing the backups)

$ show accounting ! see what it is, so you can restore
$ set accounting/enable=image

On a dedicated disk (perhaps you will need to boot from an installation CD) if so you will need to use show time, and type ahead another show time while backup is running.


$ mount/for disk:
$ set term/over
$ !backup/physical disk: nl:disk.bck/save/nocrc/group=0/block=65024
$ show time ! , uparrow, cntrl-h, space as fast as you can

That's to get the start time (if you are booted from CD)

After backup starts, type in show time into the type ahead buffer, so it will be executed as soon as backup completes.

This will give you an idea of the spiral read speed on the disk. This is close to best case, as there will be few seek operations, and they will be short seeks.

Then repeat, but use backup/image instead of backup/physical

This will be a better test of the time it takes for the read portion of an image backup.

To see how much the writing to tape slows you down, repeat the backup/physical but this time to the DLT:

$ mou/for disk:
$ init mkc500: phytst /media=compaction
$ backup/phy disk: mkc500:phytst.bck/save/nocrc/block=65024/group=0/rewind/media=compact

If anything will keep the DLT8000 streaming, that will.

With the version of BACKUP you have, the tuning recommendations are different than for recent version of backup. If you have 7.2-1 documentation, look at the system manager's essentials manual, and the system management utilities manuals (under backup) for tuning recommendations.

Summary: First try increasing /blocksize. That will make a big difference.

Jon

[Jon Pinkley]

Jess,

Unless you are really CPU starved, I would recommend using /CRC after you have finished the testing.

I would also recommend using /group=0, since with DLT, if there is an uncorrectable media read error, the drive won't let backup read the next record anyway, so the XOR redundancy that backup writes will not help you. At least that is my experience. The backup default is to write a XOR block for every 10 data blocks (/group=10). So you will use at least 10% more tape, probably more if you are using compression, since the XOR block will generally compress much less than the other data.

Jon

[Markus Waldorf_1]

Hi,

Cabling, proper scsi bus termination (active vs. passive), tape drive jumpers like parity check and termination power? Parity needs to match the scsi bus of your host. Have you tried using init/media_format=nocompact? There are usually also jumpers on the actual tape drive that allow your to disable build-in compression. If compression is under software control than you should disable device hardware compression as double compression will decrease performance.

Best regards,
Markus

[Jon Pinkley]

Markus Waldorf>>"If compression is under software control than you should disable device hardware compression as double compression will decrease performance."

I think what Markus is saying is that if the data is already compressed by some software, then hardware compression won't help, and may hurt. I.e. whether the data is being compressed by the backup application or pre-compressed by a utility like zip or gz, the drives compression will not be able to compress the already compressed data.

Note that VMS BACKUP does allow you to control the hardware compression on the drive via a backup qualifier, /MEDIA=COMPACTION. I don't believe VMS BACKUP from OpenVMS 7.2-1 had the capability of compressing data via software, V8.3 does although the help file for backup doesn't show anything about it.

$ BACKUP /DATA_FORMAT=COMPRESS ! V8.3+ software data compression

There is an LED on the front of the drive that will tell if the drive is using hardware compression or not.

Note that the same warning also applies to encrypted data, since it appears to be random data, so no patterns can be found for compression to replace with smaller tokens.

If compression was the cause of Jess's problem, it was probably because compression was working too well, and the data couldn't be supplied fast enough to keep the compressed data flowing to the write buffers at 6 MB/s.

The small default blocksize that backup uses is a killer for fast tape drives. Even 65024 becomes a limiting factor with drives like LT04, and possibly LT03 drives. HP and IBM LT03 drives can write data to tape at 80MB/sec, and keeping their input buffers full is a challenge.


See http://www.datastor.co.nz/Datastor/Promotions.nsf/4a91ca5e06d20e15cc256ebe0002290e/d954d1c5e5e6df09cc25723b00740956/$FILE/When%20to%20Choose%20LTO3%20Tape%20Drives.pdf

[Jessie Bunker-Maxwell]

Jon, thank you for all the info and suggestions. I reconnected our old tape drive earlier this week, so I now need to wait until I can do a shutdown/reboot of our AlphaServer later today in order to reconnect the DLT8000 to try the tests and changes you recommended.

I would like to have both tape drives connected simultaneously if possible but I don't think daisy-chaining them will work because the DLT8000 would have to be daisy-chained through the DLT4000 which I believe would reduce the throughput of the DLT8000 to the DLT4000 rate.

In addition to the 68-pin SCSI-3 connector, we do have a 50-pin SCSI-2 connector on our KZPDA SCSI card however, so I am going to try connecting the DLT8000 to the Alpha using that connector. This means I'm using a male-to-male 68-pin SCSI-3 to 50-pin SCSI-2 cable for that connection. If it doesn't work, that's okay but I am just hoping that it will not cause any problems or do any damage.

I will report back as soon as I have any results and thanks to you again.

Jess

[Jessie Bunker-Maxwell]

Markus, thank you for your suggestions. After the two 12-hour backups completed, I tried 10 shorter tests using different combinations of software density and software compaction enabled with the hardware compress selection but all my testing showed that when I did not select "Compress" on the tape drive itself, the backup time increased, regardless of the software settings in place. However, I will try leaving the hardware "Compress" off again when I am able to do more testing later today.

Jess

[Jon Pinkley]

Jess,

We just recently bought a used external DLT8000 off eBay. The case has two 68-pin connectors. One is connected to our KZPCA, the other has an external terminator attached (we also bought the "accessory kit").

You can daisy chain, but you will quickly use up cable length budget, and with SE, you don't have much to begin with.

I would avoid using the 50-pin connector, or any adapter/cable that converts to/from 50 pins. Unless the KZPDA is actually has two individual SCSI adapters, the second connection is connected to the same SCSI chip and SCSI bus. The HBA probably has termination on it already, and each "end" of the bus must be terminated, but if you use more than one of the connectors, the HBA will be "in the middle" of the bus, and will need to have the termination disabled. Also, when you use the 50-pin connector, it will only work with narrow devices (at least that is my understanding). Even if the DLT8000 can operate in narrow mode, you would effectively cut your transfer rate in half, since you will only be sending 8 bits per cycle instead of 16, so your Fast (10Mhz) will only transfer 10 MB/se instead of 20, and that will only make your problems worse.

Although, it isn't considered "best practice", you should be able to swap drives without shutting down/rebooting the AlphaServer. Just make sure nothing is using the drive (show dev mkc500), power off the DLT4000, disconnect the cable, and connect the DLT8000.

I would select another SCSI ID for the DLT8000 (for example 6) before connecting, then the two drives will be distinct VMS devices.

Once you reconnect the DLT8000, power the external case on. Wait for the lights to stop blinking (power on self test). If you have used a different SCSI ID, you will need to use SYSMAN to connect the device.

$ set process/priv=all ! I can't remember what privs are actually needed
$ mcr sysman io scsi_path_verify
$ mcr sysman io autoconfigure /log

If you are using the same SCSI ID for both drives, you should at least do the

$ mcr sysman io scsi_path_verify

after you change the device, so VMS will be sure to pick up the new device info.

In the mean time, do read the white paper I referenced in my last comment "When to Choose LTO-3 Tape Drives". Although it is talking about higher performance drives, the principle are the same, and I do believe that the small blocksize is likely to be the biggest contributing factor to performance degradation you experienced. I would even expect that if you use a larger blocksize with your DLT4000, you will see an improvement, although it is much easier to keep a DLT4000's buffer non-empty than a DLT8000's.

Jon

[Jessie Bunker-Maxwell]

Jon, I don't know where you are geographically, so I just wanted to try to reach you before the end of your day to let you know that your suggestion for connecting the DLT8000 to the SCSI-3 connector on our KZPDA worked like a charm - I had already reset the SCSI ID on the DLT8000 to 6 before cabling it to the SCSI-2 port in anticipation of a reboot later, so I executed the MCR sysman commands you suggested and I now see the DLT8000 as MKC600. I've printed out the LTO-3 doc and am reading it now and will try the blocksize setting first.

Jess

[Markus Waldorf_1]

Have you checked the scsi parity setting on the drive and KZPDA scsi card? The DLT8000 might be setup for use with MS-Windows. Can you run the diagnose utility to read the OpenVMS binary errorlog - it might tell you what's going on with the tape and why it's slow. Sounds like a hardware problem to me.

[Markus Waldorf_1]

The is also an old topic in the forum "slow DLT8000 and a fix please". http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=623897

" My customer has complained that when HP upgraded the firmware on their DLT8000 the backup times doubled. Hp then retro installed an older version of the firmware and things are back to normal. They have been told they need to get to the latest revision but are reluctant becuase of the speed issues."

[Jessie Bunker-Maxwell]

Jon, the backup utility blocksize qualifier definitely made a difference. I need to do more testing to determine if a different scsi card might help or if we're now getting the max throughput we can expect from this tape drive on our system, but I will be away from the office until June 11th, so will have to wait on that. Thank you so much for all your help.

Jess

[Jessie Bunker-Maxwell]

Markus, could you tell me how to check the parity setting on the scsi card and the tape drive? Jon's blocksize suggestion has made a huge difference in the backup time, but when I return to the office on June 11th, I will have more time to do additional testing and will use diag to look at what's happening. Thank you for your suggestions and pointing me to the tape drive firmware issue.

Jess

[Jon Pinkley]

Jess,

My guess is that the SCSI card probably won't make big difference. Before spending the money, you want to verify that the bottleneck is really the card. Timing a backup to the null device will give you a close approximation to the "best possible" time to read the data off the disk. There are some things that can be done to turn the authorization quotas for the username that is used for your backups, and that an affect how long it takes to backup a disk with many small files. It may be that the reading from disk is the your limiting factor, and if so, buying a faster SCSI card isn't going to speed things up significantly.

Using a large blocksize is good for several reasons, mostly reducing the per-I/O overhead.

It reduces the number of I/O operations needed, and the associated driver related setup needed for each $QIO operation.

It utilizes the SCSI bus better, as each I/O operation requires arbitration for the SCSI bus. This increases the maximum throughput possible for a given SCSI bus, in a similar way to using a larger Ethernet packet size will allow higher throughput than a smaller packet will.

Concerning DLT8000 Firmware, you can find what version is loaded using the following command:

$ pipe mcr sys$etc:scsi_info mkd600 | search* sys$pipe "Vendor Identification :","Product Identification:","Product Revision Level:"

The "Product Revision Level:" reflects the firmware loaded.

We have two DLT8000's, the one we just bought off eBay has

$! Vendor Identification : COMPAQ
$! Product Identification: DLT8000
$! Product Revision Level: 0250

Our other one has

$! Vendor Identification : COMPAQ
$! Product Identification: DLT8000
$! Product Revision Level: 0259

I am not sure what the latest firmware is. Both "seem" to work fine, although there are probably bugs fixed in newer versions.

The LTT utility can upgrade firmware, but make sure you understand the implications, and if at all possible do the upgrades when the drive/host are connected to a UPS (uninterruptible power supply).

Use the forum search for LTT for other threads discussing the tool.

Jon

[Markus Waldorf_1]

Hi, I would rather not change the default settings of the KZPDA scsi card, in particular since it seems to work fine with the old tape drive. Google for "Quantum DLT8000 Jumper Settings" should show you the link. If I remember correctly a parity mismatch will also show you ghost devices, ie. the drive device will show up several times.

Some scsi drivers will automatically slow down performance if there are errors on the bus. I would make sure to use the correct SCSI terminators and don't attach any scsi-2 or other devices on the same bus that are sync/async and have different latency like HD's. You may also want to check that none of the pins of the cables are bend, which can happen easily.
Best regards,
Markus

[Jon Pinkley]

Jess,

There is a bit more to the firmware than what I stated in my last comment.

http://www.quantum.com/ServiceandSupport/SoftwareandDocumentationDownloads/DLT8000/JumperSettingsDLT8000/Index.aspx

http://www.quantum.com/ServiceandSupport/SoftwareandDocumentationDownloads/DLT8000/PartNumbersDLT8000/Index.aspx

http://downloads.quantum.com/dlt8000/dlt8000productmanual.pdf

The product manual has the SCSI commands that are supported by teh DLT8000. Look at the data returned by the INQUIRY command for the details about the firmware.

By the way, how much of a difference in elapsed time did changing the blocksize make? Just curious.

Jon

[Jessie Bunker-Maxwell]

Jon, a backup that took 5 hours on the TZ88/DLT4000 now takes only 3 hours on the DLT8000 using the /block_size=65535 qualifier on the output. The first several attempts I made on the DLT8000 without the block_size switch resulted in a backup that took 12 hours, so this was a major improvement. I will still spend some time when I have it to try to determine if the SCSI card is playing any part in reducing throughput but I am satisfied with the performance we are getting now although it's not quite what I had expected based on a comparison of the performance specs for the two tape drives.

Jess

[Jon Pinkley]

Jess,

Thanks for the update.

From your original message starting this thread:

"Backups using the TZ-88 tape drive complete in about 6.5 hours."

From your latest message:

"a backup that took 5 hours on the TZ88/DLT4000 now takes only 3 hours on the DLT8000 using the /block_size=65535 qualifier on the output."

Where these backups of different things, or did the backup on the DLT4000 improve from 6.5 hours to 5 hours when you specified /block=65535?

Blocksize is low hanging fruit. To get more performance will require a bit more work. You have just moved from one limiting factor to another. If you are satisfied with the current performance, then it really may not be worth your time to investigate how much you can improve things. No matter what you do, you probably will not make your image backups go 4 times as fast on the DLT8000 as on the DLT4000, all other things being equal.

Jon

[Jessie Bunker-Maxwell]

Hi Jon - the difference in times is due to an additional change I made that's unrelated to throughput but has shortened the overall backup time - which is that I've removed the qualifier that writes the backup recording date on all files backed up. I didn't realize how much time that was adding to the complete backup job until recently and since we are doing image backups nightly and don't use the backup date as a selection parameter, it didn't make sense to continue it. The time saved is the difference between the 6 1/2 hours and the 5 hours.

Jess

[Jon Pinkley]

Jess,

Is this 1.5 hour difference for one disk, or for several? There must be a large number of files, and if they are relatively small, any file based (non-physical) backup will be relatively slow. That is the reason for asking how long the backups take to the NL: device, leaving everything else equal. If it takes two hours to backup to the NL: device, then backups to tape will take longer than that. Backup still processes files in directory order, so every file will involve seek times. In other words, there is overhead for every file open, and the more files involved, the more elapsed time will be spent.

A physical backup will probably come closer to the 4/1 transfer ratio of the DLT8000/DLT4000, but there are other factors that can add delays. The backup recording pass is a good example; the speed of the tape drive has absolutely no effect on the time it takes for the recording pass. Likewise, the speed of the tape drive has no effect on the time it take to read the date off the disk, compute CRC, set up the $QIOs, etc.

Jon

[Jessie Bunker-Maxwell]

We have five logical drives which are comprised of a 4GB mirror set, an 18GB mirror set, and 3 standalone 4GB drives and most of these drives are about 75% full. Our system disk and main data disk have tens of thousands of small files and hundreds of larger files although we are well within the maximum file limit on all drives. I knew I couldn't expect miracles with the new tape drive, but I also knew there was something wrong when it was so much slower than the old drive. I didn't expect to match the device specs for a DLT8000 but I was hoping for a bit more than we got. However, faster is better no matter what and we wouldn't have gotten that without your blocksize suggestion. So thank you very much again.