Performance with FC Ultrium 4-SCSI LTO4 drive

[RalfG]

Hi,

we have an Overland Neo 4100 Lib with 3 HP LTO-4 drives (2x SCSI, 1x FC). Writing to the SCSI drives is no problem. The data is first written to disk and then written to tape. This way I get 100-130 MB/s for a single job on the SCSI drives which is perfectly fine. Each LTO-4 drive has it's dedicated spool device which is able to deliver >130 MB/s read speed.

The FC drive makes some problems. It starts writing with ~100 MB/s but then drops to 30-50 MB/s.

Our retailer already replaced the LTO-4 drive and the FC HBA (qla 2462). Without success.

I'm a bit lost now. I don't see anything obvious that is wrong. Are there any special parameters I have to change in the FC HBAs BIOS? Anything else I should try?

Here are the parameters of the FC HBA:

--------------------------------------------------------------------------------
HBA Instance 0: QLA2462 Port 1 WWPN 21-00-00-1B-32-11-ED-6E PortID 00-00-EF
--------------------------------------------------------------------------------
Connection Options : 2 - Loop Preferred, Otherwise Point-to-Point
Data Rate : Auto
Frame Size : 2048
Hard Loop ID : 0
Loop Reset Delay (seconds) : 5
Enable Host HBA BIOS : Disabled
Enable Hard Loop ID : Disabled
Enable FC Tape Support : Enabled
Operation Mode : 0 - Interrupt for every I/O completion
Interrupt Delay Timer (100ms) : 0
Execution Throttle : 16
Login Retry Count : 8
Port Down Retry Count : 30
Enable LIP Full Login : Enabled
Link Down Timeout (seconds) : 30
Enable Target Reset : Enabled
LUNs Per Target : 128
Enable Receive Out Of Order Frame: Disabled



TIA, Ralf

[RalfG]

I really can't understand what is going wrong with this drive. As a test I created a 8 GB big temp. file with data from /dev/urandom to see if I get the nativ speed when writing to tape with dd (data from urandom is compressable any more).

And again the drive beginns with good writing speed and then drops in the second run to < 50 MB/s. But in the third run it's back with > 100 MB/s. It's just not writing with constant speed. The two SCSI drives in opposite are absolute constant at 100-120 MB/s with the same test.


$dd if=/tmp/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
42925+1 Datensätze ein
42925+1 Datensätze aus
11252613120 Bytes (11 GB) kopiert, 106,823 Sekunden, 105 MB/s

$dd if=/tmp/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
42925+1 Datensätze ein
42925+1 Datensätze aus
11252613120 Bytes (11 GB) kopiert, 236,543 Sekunden, 47,6 MB/s

$mt -f /dev/ULTRIUM-TD4-D3 rewind

$dd if=/tmp/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
42925+1 Datensätze ein
42925+1 Datensätze aus
11252613120 Bytes (11 GB) kopiert, 100,627 Sekunden, 112 MB/s

$dd if=/tmp/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
42925+1 Datensätze ein
42925+1 Datensätze aus
11252613120 Bytes (11 GB) kopiert, 235,908 Sekunden, 47,7 MB/s

$dd if=/tmp/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
42925+1 Datensätze ein
42925+1 Datensätze aus
11252613120 Bytes (11 GB) kopiert, 102,296 Sekunden, 110 MB/s

$dd if=/tmp/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
42925+1 Datensätze ein
42925+1 Datensätze aus
11252613120 Bytes (11 GB) kopiert, 236,189 Sekunden, 47,6 MB/s



I found some HBA parameters that could be tuned according to a IBM redbook.

http://www.redbooks.ibm.com/abstracts/tips0250.html?Open


But none of the changes helped.



Anything else I should change to get the right performance with the fiber channel LTO-4 drive attached to a linux host?

I'm out of ideas, the drive and the HBA were already replace.

[Rob Leadbeater]

Hi,

Can you explain exactly how the fibre is connected ?

I think the 4100 is the equivalent of the MSL5/6000 which does have some limitations with regards to LTO-4 drives...

Cheers,

Rob

[RalfG]

Well, the FC LTO-4 drive is directly connected to a Qlogic 2462 FC HBA.

The two SCSI LTO-4 drives are each connected to a SCSI HBA and the changer also has it's own SCSI HBA.

I've no general problem with LTO-4 drives, the two SCSI drives are performing perfectly fine. It's just the FC drive that shows these problems.

Today I moved the FC drive from the slave lib to the master lib (each 60 slots), no change. Then I connected the FC cable of the FC drive with a different HBA in the same server where I never had performance issues with RAID devices before. Same problem.

The strange thing is that I see the expected performance for a couple of GB but then it drops massively.

[RalfG]

$dd if=/spool/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
60655+1 Datensätze ein
60655+1 Datensätze aus
15900360704 Bytes (16 GB) kopiert, 155,939 Sekunden, 102 MB/s


The 102 MB/s look good, but....

here are some statistics from my spool device during that test (the device is a RAID0 which can deliver data with ~160 MB/s).


--dsk/sdg--
_read _writ
[...]
104M 0
121M 0
102M 0
120M 0
103M 0
121M 0
102M 0
121M 0
103M 0
118M 0
106M 0
116M 0
108M 0
116M 0
108M 0
114M 0
110M 0
113M 0
112M 0
111M 0
113M 0
109M 0
115M 0
106M 0
118M 0
103M 0
120M 0
100M 0
119M 0
103M 0
113M 0
111M 0
98M 0
114M 0
108M 0
96M 0
108M 0
109M 0
103M 0
96M 0
96M 0
97M 0
101M 0
97M 0
97M 0
96M 0
92M 0
92M 0


so far so good, but then the speed drops to < 50 MB/s.


0 0
0 0
0 0
32M 0
64M 0
32M 0
62M 0
35M 0
64M 0
32M 0
64M 0
32M 0
64M 0
47M 0
49M 0
[...]


A second test directly started after the first one shows the whole problem...

$dd if=/spool/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
60655+1 Datensätze ein
60655+1 Datensätze aus
15900360704 Bytes (16 GB) kopiert, 261,715 Sekunden, 60,8 MB/s

[Rob Leadbeater]

Can you describe the fibre connection on the back of the library...

Does it go into an NSR then out via SCSI to the drives, or is it direct fibre ?

[Curtis Ballard]

Use L&TT and pull/post a support ticket from the drive. It sounds like there may be one channel in the reverse direction that is dirty or bad. LTO runs in both directions, forward then backward. If you have a bad head element in one direction then performance will be up and down based on direction of travel.

The recommended action is to clean the drive several times if that is the case.

[RalfG]

@Rob Leadbeater

The cable is directly attached to the drive. No NSR involved.

[RalfG]

@Curtis Ballard

It took me some time to generate the support ticket. Because we are running Debian Etch Linux I had to convert the ltt RPM into a Debian packet with alien.

Can you see anything interesting in the report? We had some bad Maxell tapes (seems that a whole shipment with the same manufacture date was bad). But this didn't affects the perfromance of the two SCSI drives.

How can I force cleaning of the drive? I always thought the drive would refuse cleaning if it's not needed.

This is the Lib's information regarding cleaning:

Cleaning Information
Cleaning Needed? NO
Cleaning Required NO
Cleaning is in progress NO
Clean Expired NO

[RalfG]

I triggered 4 cleaning cycles through the changers web gui, but see still the same effekt.

Writing 16 GB from disk to tape again (/dev/sdg = /spool).


$dd if=/spool/testdata of=/dev/ULTRIUM-TD4-D3 bs=256k
60655+1 Datensätze ein
60655+1 Datensätze aus
15900360704 Bytes (16 GB) kopiert, 277,436 Sekunden, 57,3 MB/s

dstat output

--dsk/sdg--
_read _writ
1438k 28B
92M 0
118M 0
109M 0
115M 0
113M 0
112M 0
116M 0
108M 0
120M 0
105M 0
124M 0
104M 0
124M 0
104M 0
120M 0
107M 0
117M 0
112M 0
112M 0
115M 0
109M 0
117M 0
107M 0
121M 0
103M 0
121M 0
103M 0
118M 0
107M 0
109M 0
115M 0
96M 0
110M 0
111M 0
99M 0
96M 0
104M 0
106M 0
104M 0
99M 0
99M 0
96M 0
96M 0
96M 0
96M 0
32M 0
0 0
0 0
7552k 0
57M 0
36M 0
60M 0
63M 0
64M 0
56M 0
45M 0
58M 0
64M 0
61M 0
42M 0
58M 0
64M 0
64M 0
43M 0
53M 0
64M 0
64M 0
47M 0
49M 0
64M 0
64M 0
59M 0
37M 0
[......]
32M 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
0 0
34M 0
94M 0
96M 0
96M 0
96M 0
105M 0
107M 0
105M 0
100M 0
96M 0
103M 0
108M 0
106M 0
100M 0
96M 0
103M 0
108M 0
105M 0
101M 0
96M 0
103M 0
109M 0
104M 0
101M 0
96M 0
103M 0

[Curtis Ballard]

Regarding the support ticket - sorry I didn't think about you may be using an OS that didn't have a native L&TT install. You can pull the support ticket using the library web interface and selecting the drive that you want. That method doesn't require installing L&TT but you have to have L&TT to read the resulting ticket. If you just post the ticket here you don't need L&TT.

The XML file is really difficult to read, an "ltd" is the default and much preferred. If you have an ltd for posting that would be appreciated.

From the XML file I was able to pick out a couple of things. The HP LTO drives have a built in head cleaner and there are several calls to clean the head with that so it appears that the drive thinks that it needs cleaned but if you cleaned it then the problem may be more than cleaning needed.

Back 62 hours ago the drive reported a defective tape with a poorly written servo track - that may match to the batch of bad tapes you mentioned. If a tape with a poor servo track is being used some tape drives will have more trouble than others since the drive will be working at the extreme edge of it's capabilities to track the tape. It's possible that this particular drive just can't follow those servo tracks but I can't tell that from the XML file, I would need the ltd. Not being able to follow bad servo tracks wouldn't indicate a drive hardware problem since the drive would be able to follow a correctly written servo track just fine and would probably follow a servo track written badly on the other side of the specification when drives that follow this servo couldn't.

[RalfG]

I can't find a way to generate a ltd type support ticket. The hp_ltt cli tool for linux only creates lzt or xml files. As far as I can see the ltd is part of the lzt file. So I'll attach the lzt file and additionally the html report I created from the lzt with the Windows LTT tool.

Regarding the bad tapes: all were exchanged weeks ago, so there shouldn't be any left. The 3 tapes I use now for testing are from a different production date and work fine in the SCSI drives (the bad tapes had problems in all drives).


In the generated report I messages regarding cleaning but the lib tells me cleaning is not needed (today I tried again 2 write cycles starting at 000044:54:55.267456 and ending at 000045:06:25.218496):


000045:04:13.707008 Clean |MechInterface commanded to brush the head.

Am I right that this is the internal cleaning that happens automatically?

[RalfG]

and the compressed html report...

[RalfG]

I started an Assessment test. The output doesn't look very promising.... (I've also attached the RsltLog.ltt file)


|__ Test 'LTO Drive Assessment test' started on device 'HP Ultrium 4-SCSI' at address 'a0.0.0[10-/dev/sg10]'
|__ executing LTO Drive Assessment Test...
|__ LTO Drive Assessment Test, version V19.08.2008
|__ Test run: Wed Aug 5 11:00:50 2009
|__ Drive serial number: HU18081FDV
|__ checking tape load ...
|__ Data Cartridge Information:
|__ Vendor: MAXELL
|__ Format: LTO-4
|__ Serial Number: 1280077913
|__ Barcode:
|__ ANALYSIS OF HISTORICAL INFORMATION (from drive logs):
|__ analyzing device data...
|__ updating device writeback data...
|__
|__ version: V19.05.2009
|__ There were 19 rules checked.
|__ Device Analysis has checked the historical information and no problems were found.
|__ RESULTS OF TESTS ON THIS DRIVE
|__ [Please note: the operations performed by this overwrite test will NOT be
|__ reflected in the Support Ticket usage/health information.]
|__ Amount of data requiredâ test option : Default
|__ erasing ...
|__ soft unload ...
|__ loading ...
|__ Iteration number: 1 of 4
|__ writing wrap 4 ( iteration 1 / 4 )...
|__ Write margin (wrap 4): Great margin (13.3 GB written)
|__ writing bottom wrap ( iteration 1 / 4 )...
|__ Warning: Drive is showing a large variation between channels.
|__ Write margin (bottom wrap): Warning (6.8 GB written)
|__ Overall Margin: Warning (20 GB written)
|__ Wrap 29 Margin: Warning
|__ soft unload ...
|__ loading ...
|__ erasing ...
|__ The LTO Drive Assessment Test has checked the history and/or operation of the selected drive, and
|__ problems have been reported.
|__ The drive is no longer recommended for use.
|__ Please contact hp support for further assistance.
|__ Test time: 6:05
|__ Test failed. See the analysis results in the test operations log.
|__ Test failed
|__ Failed

[Curtis Ballard · Accepted solution]

It looks like one channel in the head is almost completely out requiring lots of rewrites. There are a couple of other channels where the signal quality is lower than recommended so I would agree with the assessment test - drive needs replaced.

PS - yes the ltd files are embedded in the lzt files. When saving a support ticket from a library web interface you get an ltd and when using the L&TT tool it bundles and compresses all of the ltd files from one library log set into an lzt file.

[RalfG]

The drive was replaced today. The new drive doesn't show the problem and passes the Assessment Test.

Still a bit uncertain why we have to replace these drives so often (without beeing used much yet).

Next time I'll beginn with the assessment test...

[RalfG]

drive war replaced