LTO speed problems

[PSanderson]

Hi I have an LTO 460 on a XP pro box with a raid5. The drive is connected via a 29160 and is displayed at boot time as connected at 160.

I cannot achieve better than about 12GB/hr transfer rate on the drive. I have tried the StorageWorks Tape Tools perfomance test and achive the same poor performance. I also receive the same performance when reading from the tape and discarding the output. So it is not the hard disks that are the bottleneck.

The health tab in Tape Tools says that current configuration is narrow and the recommended is wide

Is narrow likely to be enough to limit the performance?

[PSanderson]

Update

I have now tried three different U320 cables and a 39160...

I am just building a new machine to test this on.

Any ideas anyone

[PSanderson]

Further update

I have now tried tow totally different machines Both spec end P4's one with 1GB memory the other with 4 (both running XP Pro)

Two Ultrium 460's with the latest firmware

a 29160LP and a 39160

3 different SCSI cables

I have run out of things to try

Any suggestions very gratefully received.

[Frank Van Riet]

Hello,

there is some info missing in here :

- You say you are using a RAID 5 setup. How many disks? What interface ? What is the disk rpm speed?
- Do you use a RAID controller or is it software RAID...software raid is painfully slow...
- Backup speed also depends a lot on what you try to backup: a lot of small files (eg. a windows system directory) is a lot slower than a few big files (eg. movies, etc). Have you tried both ?
- what application are you using for backup?

An LTO 960 was built to be used in a large backup environment, with multiple servers sending backup streams to the tape at once..

Test that have been performed show that it is mostly the servers + disk combinations that are the real botllenecks, not the tape drive..

Regards,

Frank Van Riet
PM StorageWorks
HP Belgium

[PSanderson]

Thanks for responding Frank

The RAID is a sata raid 0 with 2 x 7200rpm disks. But this is a bit of a red herring as L&TT fail on the storage performance test which DOES NOT access the disk.

Also the drive is a 460.

Cheers
Paul

[PSanderson]

Some further info

The L&TT health check reports this

version: V31.11.2005
Firmware rev F5CD is up-to-date for Ultrium 2-SCSI as of Thu Sep 2 19:00:00 2004.
Rule 19:
The current SCSI configuration is likely to be limiting the performance of the drive.
Please check that your HBA is the correct type for the drive and that the cabling is good.
The SCSI configuration referenced is the one for which device analysis was run and/or the support ticket was pulled.
If this is not via your backup server then you may not have an issue.
Current SCSI configuration: narrow Recommended: wide
The drives hardware compression algorithm is currently switched off.
This was probably set by the backup software.
The drive's setup has not been altered.
Current SCSI transfer rate limited to: n/a Recommended: 160 MB/sec. or better
There were 19 rules checked.
Device Analysis completed, and warnings have been reported.

Although it says I have a narrow connection I am connecting to a 29160LP or a 39160 and both get the same message. The drive is the only device on the bus and there is not termination on the drive - i.e. I am using the drives active termination (the light is on)

Cheers

[Richard Bickers]

12GB/hr is 3.3MB/S - which is about 10% of what you should be getting. Ouch. Interesting that LTT shows the same performance so the bottleneck must be in the Server, HBA, driver, cabling, drive pipeline. Sounds like you're using the right kind of HBAs and cables and you mention high-spec servers. You've even tried multiple drives. Are you using latest drivers?

The narrow bus will limit the burst rate to 80MB/S which isn't recommended for the Ultrium 460 but that wouldn't drop your rate down to 3!

There are a few things to check on the system like IRQ assignments and making sure the drive is the only device on the bus - including switched off devices as they affect the negotiation. What is the negotiated rate from the health tab?

There's a more complete list of things to try at www.hp.com/support/pat. The focus is on the disks as that's where the bottleneck usually is but there's a few tips from the server standpoint.

This is very unusual. It's a very significant bottleneck and it's common throughout every combination you've tried.

I doubt it's the drive. If these drives do suffer error rate issues then the performance can drop by 10-20% - not the 90% that you're seeing. It sounds like a SCSI burst rate negotiation thing - as if it's negotiated to 5MB/S burst which is the lowest possible.

One thing to try/confirm is that you get the same data rate for different compression ratios which you can do from the Device Performance test in LTT. That would verify for sure that the bottleneck is not from the mechanical/physical/media part of the drive.

B.t.w. we're in the process of releasing the next update to LTT - 4.0 SR2 - which has slightly improved health output. Main difference is the performance estimates are more accurate for the previous tapes. If you write to the drive for a few minutes and then unload the tape you'll find a good estimate of the performance - which will no doubt tie up with what you've found so far... You should find 4.0 SR2 on the web site sometime next week.

[PSanderson]

Thanks Richard

I will have another look at that link and revisit the IRQ's.

The throughput is the same for different compression rations, I have tried 1.6, 2 and 3:1 all give 3MB/s

I don't understand the narrow issue though - are either the 29160LP or the 39160 narrow. Both are rated at 160MB/s which is what I see in the adapter BIOS scan

Interesting - On the health tab under "Drive Configuration|Interface" the SCSI host ID is listed as unknown

also Drive "Configuration|Interface|Negotiated Burst Data Rate" has unknown for DT mode, bus width, clock rate and disconnect privelages - is this right?

[Richard Bickers]

Yes, that's odd. Seems like you're getting 160 with a 160 HBA. The device analysis rules will trigger because we recommend 320 for the LTO 3 but this is way off the scale compared with 3 MB/S!

Also odd that you're getting unknowns in the health/drive/config tab. If you can do it today (I'm away next week) perhaps you could try a device performance write to the drive, unload the tape, pull and email the ticket to the ltt_team@hp.com mailbox so we can take a look.

[PSanderson]

Richard I am running the test now - I am interested in why you wish me to unload the tape at the end though - just curious.

[Richard Bickers]

Because.... strangely enough, we have more accurate performance estimates from the previous tapes loaded than the current one. That's just a 'feature' of the LTO logs which we're working on for future generations.

[PSanderson]

Done cheers.

[PSanderson]

I have still not git this working - the latest test was on a shuttle PC with a fresh install of XP Pro, 1 GB memory teh only device attached is the LTOII drive and I still get 3MB/s

I have looked at the irq's and cant seea problem but I am a little unsure of myself here.

[Simon Dewhirst]

Hi there,

I am getting exactly the same problems but only 1.86mb/sec regarless of what compression/software is set/used.

I have an Adaptec 29160LM SCSI card and the drive is the only device attached to it.

I also get error 9 and moans about the configuration being narrow, yet it's set to Wide within the cards BIOS.

Disk performance is not an issue, tested with various tools including HP's own tools and can read at 27mb/sec from the drive. Also the tests run from memory in this particular case, so it doesn't make any difference.

This has been driving me mad for days now, could someone please help?

The drive is an HP Ultrium 215 which is capable of 900mb/min. 111mb/min is not acceptable and means backup routines take approximately 13 hours.

What I will also state is that the maximum sync speed in the scsi cards bios is set to ASYN and sync negotiation is set to YES/ON.

I am running the latest drivers and firmware (even though L&TT cannot detect the firmware for some reason - moans that the script cannot detect it) and have tried all sorts of different drivers for both the card and drive. Nothing has made any difference to speed.

Thanks in advance

[Richard Bickers]

I think the common theme here is termination. These drives should auto-terminate if they are in external boxes showing a green light when active.

The HBA and drive SCSI electronics re-negotiate SCSI burst rates slower if there are SCSI errors found and they go down to asynchronous if necessary. Although the LTT support ticket doesn't decode the asynchronous mode properly the words 'unknown' in that section of the ticket are a bit of a giveaway.

You could try turning active termination off and using an external terminator if there's any doubt over the drive's active termination. I assume you're using the right cabling...

If you'd like to attach a ticket to this thread I can take a look in the logs for any SCSI parity errors which would be another sign. Just save the ticket to a directory and then zip the two files inside (*.ltt and *.dat).

I think we ended up sending Paul a cable but it still didn't fix his issue. Is that right, Paul?

This is driving me nuts too...

[Simon Dewhirst]

Hi,

Thanks for the response. I think the problem may be that I have forced ASYN on the SCSI card? As far as I understand, does this not throttle the bandwidth to a maximum of 5mb/sec and the speed degrades very quickly dependant upon the length of the cable?

Granted, the cable length is not long at all and shouldn't be affected in that sense, however if the absolute maximum is 5mb/sec, would this not suggest this is where the problem resides?

When I saved the support ticket, I only received an .ltt file (no .dat). I hope this is sufficent for what is required.

I look forward to your response.

[PSanderson]

Hi Richard

No you are right this did not solve the problem and that particular project is now complete so it has had to take a back seat.

It is left as follows.

If drive is running like a dog then remove termnation if present, or terminate if it is not.

I then get to process 1 tape (about 200GB) and when I try the next tape, without fail, speed has dropped to a slow walk.

Swap the termination and I get my 60-100GB/hr.
So to get the last job done I just alternated termination between tapes.

How weird is that !!!

I may have another set of LTO's coming in soon so I may need to revisit.

[Richard Bickers]

Simon,

You probably saved the ticket from the health option which saves it in the old .ltt format. That's a bug I'm afraid... but the ticket still has what I was looking for:

|__ Drive Configuration
||__ Drive
||__ Interface
||__ SCSI Drive ID : 15
||__ SCSI Host ID : Unknown
||__ Negotiated burst data rate : Unknown
||__ DT Mode : Unknown
||__ Bus width : Unknown
||__ Clock rate : Unknown
||__ Disconnect privilege : Unknown
||__ Req/Ack Offset : 15

This looks like asynchronous mode to me. Same as what Paul was getting.

I wasn't clear from your message whether you'd set it to be asynchronous on purpose. That's not good. It should be wide and synchronous and allow the HBA and drive to negotiate the maximum rate they can both operate at - which is 80MB/S!

This is also picked up in the health/device analysis section of the ticket.

If you can change the HBA defaults back to synchronous then that has to be a good start.

Paul, thanks for getting back - your case is still a mystery. Keep in touch though I've embarassingly run out of ideas for you... Hope things go better with your new drives.

[Simon Dewhirst]

Thanks Richard, I shall try and see what happens.

If you have a look at one of the documents on Veritas's website, they suggest the following: http://seer.support.veritas.com/docs/images/234316/2940bios3.jpg

I shall take your advice in the first instance and set the "Initiate Sync Negotiation" to "Yes" and set the "Maximum Sync Transfer Rate" to "80.0"

I will post in the next day or two on how I got on and to hopefully let everyone know it was resolved.

Thanks again.

[PSanderson]

I now have another batch of LTO tapes to deal with.

I am seeing exactly the same problem.

1. Run imaging process with no active term in place, 80GB/Hr

2. Next tape, still no term - 3GB/hr

3. restart process with an active term on the drive - 80GB/Hr

4. Next tape, leave term in place - 3GB/Hr

5. Remove term and restart - 80GB/hr

I get the same scenario when using tape tools.

Still would like to sort this out...

[Simon Dewhirst]

I managed to sort the problem in the end by updating the drivers of the SCSI card, use the latest drivers for the tape device and set "Maximum Sync Transfer Rate" to "10.0 mb/sec". I now get on average 500mb/min.