Tar command issue with Ultrium LTO-4 1760 on SAS bus

[JACC]

Error received in dmesg ( kernel ring buffer ) when running:
tar -cvb 256 -f /dev/st0 ./filename

st0: Block limits 1 - 16777215 bytes.
st0: Failed to read 126976 byte block with 8192 byte transfer.

The odd behavior is the tar command performs without issue as long as the blocking factor is 248 or less. Anything above fails. I can also recreate the issue using the "dd" command (see following).

dd if=/dev/zero of=/dev/st0 bs=126976 count=2 (248 x 512 Bytes - equivalent to tar command using -b 248)
2+0 records in
2+0 records out
253952 bytes (254 kB) copied, 10.3574 seconds, 24.5 kB/s


dd if=/dev/zero of=/dev/st0 bs=131072 count=2 (256 x 512 Bytes - equivalent to tar command using -b 256)
dd: writing `/dev/st0': Device or resource busy
1+0 records in
0+0 records out
0 bytes (0 B) copied, 16.0792 seconds, 0.0 kB/s




Output from mt status command:

drive type = Generic SCSI-2 tape
drive status = 1174405120
sense key error = 0
residue count = 0
file number = 0
block number = 0
Tape block size 0 bytes. Density code 0x46 (unknown).
Soft error count since last status=0
General status bits on (41010000):
BOT ONLINE IM_REP_EN



System Info:

HP Proliant DL380R06 Rack Mount

52: SCSI 200.0: 10601 Tape
[Created at scsi.1363]
UDI: /org/freedesktop/Hal/devices/pci_103c_323a_scsi_device_lun0_scsi_generic
Unique ID: Er1e.jH5QXkQpQf6
Parent ID: cuhJ.XEXEELn_ra7
SysFS ID: /class/scsi_tape/st0
SysFS BusID: 2:0:0:0
SysFS Device Link: /devices/pci0000:00/0000:00:08.0/0000:0a:00.0/host2/target2:0:0/2:0:0:0
Hardware Class: unknown
Model: "HP Ultrium 4-SCSI"
Vendor: "HP"
Device: "Ultrium 4-SCSI"
Revision: "U52D"
Driver: "cciss", "st"
Device File: /dev/st0 (/dev/sg1)
Device Files: /dev/st0, /dev/tape/by-id/scsi-3500110a0012cef1a-st, /dev/tape/by-path/pci-0000:0a:00.0-scsi-0:0:0:0-st
Device Number: char 9:0 (char 21:1)
Config Status: cfg=new, avail=yes, need=no, active=unknown
Attached to: #48 (RAID bus controller)


output from modinfo cciss:

filename: /lib/modules/2.6.16.60-0.58.1-smp/kernel/drivers/block/cciss.ko
license: GPL
description: Driver for HP Smart Array Controllers version 3.6.24-4
author: Hewlett-Packard Company
depends: scsi_mod
vermagic: 2.6.16.60-0.58.1-smp SMP gcc-4.1
parm: cciss_tape_cmds:number of commands to allocate for tape devices
(default: 6) (int)
parm: cciss_allow_hpsa:Prevent cciss driver from accessing hardware kn
own to be supported by the hpsa driver (int)




output from modinfo st:

filename: /lib/modules/2.6.16.60-0.58.1-smp/kernel/drivers/scsi/st.ko
alias: char-major-9-*
license: GPL
description: SCSI tape (st) driver
author: Kai Makisara
srcversion: 823F47591D48D44F0CEBA94
depends: scsi_mod
supported: yes
vermagic: 2.6.16.60-0.58.1-smp SMP gcc-4.1
parm: buffer_kbs:Default driver buffer size for fixed block mode (KB;
32) (int)
parm: max_sg_segs:Maximum number of scatter/gather segments to use (25
6) (int)
parm: try_direct_io:Try direct I/O between user buffer and tape drive
(1) (int)
parm: try_rdio:Try direct read i/o when possible (int)
parm: try_wdio:Try direct write i/o when possible (int)

I will be happy to supply any other additional information which may be helpful in resolving this issue. Please be patient with me, manipulating a system at this level is very knew to me.

I should also add LTT was run on the system and no issues were found.


Thanks in Advance

JACC

[Steven Schweda]

I know nothing, but it sounds like a (buffer
size?) limitation in the tape driver or in
the drive itself.

Does "man st" tell you anything? Drive
documentation?

I don't do much with tape on any GNU/Linux
system, but is a block size that big normal?


> [...] Anything above fails. [...]

"Doctor, it hurts when I do this."

"Don't do that."

[JACC]

Thanks for the prompt reply Steven.

LOL over the "don't do that".

There are a couple of things which are driving me crazy about this issue. We have 300 LTO3's in the field, 74 LTO-4 and a sprinkling of about 250 others. Granted the majority of servers running SUSE but on "Different hardware". In all cases the blocking factor for tar has been 500. So now my choice is to migrate 600 servers to a lower blocking factor which also then becomes a historical recovery issue as well or figure out why this is not working.

I have a very similar system running the HP Proliant server on SUSE with an LTO-4 on a scsi bus (NON SAS) and it works just fine.

So why all of the sudden a difference?

I agree there must be some sort of lower level setting whether it be a kernel param or driver issue I can't quite figure out. And as I said before hardware really isn't my area (i.e lack experience in lower level configuration).


Thanks

JACC

[Jimmy Vance · Accepted solution]

I sent your question to the group who works on the cciss driver, I'll let you know what they say.

[JACC]

Thanks very much Jimmy, greatly appreciated.

JACC

[Jimmy Vance]

Are you using the embedded Smart Array 410i? Is it zero memory or does it have a cache module?

[JACC]

HP DL380G6 CTO Chassis
(2) Intel Xeon Processor X5560 2.80 GHz CPU's
32GB PC3-10600R-9 2Rx4 DDR3 Memory
(2) HP 460W CS HE Gold Power Supply's
HP Smart Array P410/512MB with BBWC
HP Slim SATA DVD-RW Optical Drive
(2) HP 146GB 3G SAS 15K SFF DP ENT HDD
Hewlett-Packard ILO ADV 1-SVR INCL 1YR TS&U SW
HP SC44Ge Host Bus Adapter
3YR 4HR24X7 HW PROLIANT DL380 CAREPACK
HP StorageWorks 1U SAS Rack-Mount Kit with one LTO-4 Ultrium 1760
3YR UPG WARR ONSITE 24X7 4HR 1U TAPE ARRAY
HP StorageWorks MSA60 Storage Shelf (12 SAS or SATA LFF)
(12) HP 300GB 6G SAS 15K 3.5" DP ENT HDD
HP Smart Array P800/512 BBWC Controller
2M EXT MINI-SAS X4 SFF-8088 TO MINI-SAS X4 SFF-80

[Jimmy Vance]

Thanks for the system detail. Which controller do you have the tape drive connected to?

HP Smart Array P410/512MB with BBWC
HP Smart Array P800/512 BBWC Controller
HP SC44Ge Host Bus Adapter

[JACC]

Sorry Jimmy ! I really appreciate your help.

Do you know the command to display the info?

[Jimmy Vance]

I think the easiest way is to just look and see what controller the cable from the tape drive connected to.

The reason I ask is with the specs of the systems, just a casual observation one would think the following

The embedded
HP Smart Array P410/512MB with BBWC
is probably conencted to the
(2) HP 146GB 3G SAS 15K SFF DP ENT HDD
in the server drive bays

And the
HP StorageWorks MSA60 Storage Shelf (12 SAS or SATA LFF) with
(12) HP 300GB 6G SAS 15K 3.5" DP ENT HDD
is conencted to the
HP Smart Array P800/512 BBWC Controller

Which leaves the
HP SC44Ge Host Bus Adapter
Which would be the logical choice for the controller connected to the tape drive

Your original posts shows the tape is using the cciss driver which is for the P410 and P800 controllers. The SC44Ge use the mptlinux drivers.

[JACC]

Jimmy, I will check on Monday with our hardware installation people. Unfortunately the server is located in Oregon and I am in North Dakota. I wouldn't mind a trip to the Northwest, but I doubt management would go for it : )

I will post back on Monday.

Thx

[JACC]

Good Morning Jimmy,

I just spoke to the hardware guys and the tape drive is connected to a "Smart Array P212". Sorry for the confusion, originally I had posted based on the purchase order which evidently was done thru the hardware configurator and there were some problems. But we have verifed the tape drive is connected to the P212 via the System Management Homepage.

Details of the controller:

Firmware Version: 2.74
Product Revision: C

If any other info required please let me know.

[Steven E. Protter]

Shalom,

On HP-UX, it is best not to hang a tape drive on the smart card raid interface. Of course with HP-UX, if you do that the drive doesn't show up on ioscan at all.

I would suggest connecting the drive to a standalone scsi card that is not doing hardware raid. There may be a connector on there, but it isn't going to work well.

SEP

[JACC]

Steven,

Thanks for the response.

The controller is not participating in the raid. The tape drive is the only device connected to the stand-alone card.

I think the only card we can get to interface with the tape drives is a smart array card.

JACC

[JACC]

Additional info relating to previous post.


Excerpted from: http://h18000.www1.hp.com/products/quickspecs/10854_na/10854_na.HTML



SAS Controllers
HP SC44Ge SAS Host Bus Adapter
NOTE: Recommended SAS HBA for connecting Ultrium tape drives to G5 ProLiant servers.

HP Smart Array P212
NOTE: Recommended SAS HBA for connecting Ultrium tape drives to G6/G7 ProLiant servers.


Please visit: http://www.hp.com/go/connect for a complete list of Ultrium tape drive compatibility with ProLiant servers and other industry standard servers. 462828-B21
HP Smart Array P400/256MB Controller
405132-B21
HP Smart Array P400/512MB Controller
411064-B21
HP 8 Internal Port SAS HBA with RAID 0/1
NOTE: PCI-X interface
347786-B21

[Jimmy Vance]

Is the P212 Zero memory or 256 cache?

[JACC]

Sorry for the delay. Wheels of bureaucracy turning slowly.


The controller is:

Part Number: 462828-B21-W
Desc: HP Smart Array P212/Zero Memory Controller

[Jimmy Vance]

Here is what I've heard back so far. If I hear anything else I'll pass it on

248 blocks * 512 bytes/block = 126976 bytes.
126976 bytes / (4096 bytes/page) = 31 pages.


The P212 zero memory only supports 31 scatter gather elements per command, and thus has a maximum transfer size of 126976 bytes, as the kernel will not send down any request with a transfer size larger than

(number of scatter gather elements * PAGE_SIZE) == 126976 bytes

Adding a cache board to the P212 might bump up the number of scatter gather elements the controller will report so that a larger block size will work.

[JACC]

Jimmy,

Thanks so much for input. It is greatly appreciated. We are ordering the HP Smart Array P212/256MB Controller and I will post with the results once installed. Hopefully this will cure the ills.

Thanks Again to Everyone for your input. I update points when completed.

JACC

[Cmorrall]

I'm facing a similar problem, although I'm on Windows. I have a setup with two identical libraries connected to (what I thought) was two identical P212 SAS adapters.

When doing performance testing using L&TT I could use 256 block size on one of the libraries but I got an error on the error stating 65536 bytes was the maximum block size supported.

On closer inspection, turns out one of the P212 reports "Array Accelerator Present: True" and a Total Cache size of 144 MB but the other reports "Array Accelerator Present: False".

The suggestion that the cache size will affect what maximum block size you can configure seems a valid one.

[JACC]

Ordered the Array P212/256MB Controller. Awaiting for it to arrive at the remote site (order was messed up and wrong parts arrived had to send back and reorder). Once installed and tested. I will post the results.

Regards

[JACC]

Finally got the P212/256 controller installed. No other adjustments have been made. The same behavior exists.

##### The below command runs without issue:
tar -cvb 128 -f /dev/st0 ./someFile.log
./someFile.log (no issue)

##### The below command runs but has issues:
tar -cvb 256 -f /dev/st0 ./someFile.log
./someFile.log
tar: /dev/st0: Cannot write: Device or resource busy
tar: Error is not recoverable: exiting now

##### The below command runs without issue:
tar -cvb 248 -f /dev/st0 ./someFile.log
./someFile.log


Any suggestion at this point would be greatly appreciated. All our systems with the exception of this one(HP server) are currently running a blocking factor of 500 without issue.

Thanks in Advance

JACC

[Jimmy Vance]

My only suggestion at this point is to open a support case with HP. I'd reference this thread and get it escalated past 1st level support.

[JACC]

Thanks Jimmy,

Appreciate all your help with this. I will do as suggested and post results to this thread when resolved.


Thanks to all for your input !

JACC

[JACC]

FYI:

Just got raised to Level 2 support.

Wahoooooo!