fbackup and dlt tape drives in K-class server

[Vito Sarducci]

I have an interesting scenario i need assistance on. I have a k-class with two dlt7000 drives internal. I use a script to call my fbackup utility that ultimately uses both drives for this one backup.

When the fbackup finished writing to the first tape, the processes stays open, but does not start writing the same file to the other tape?
This process has been working fine for quite awhile, and now its sporadic as far as working some days and not working others.

I have noticed that the backup runs on 1m fine and fails on 0m. Other days it runs on 0m fine and fails on 1m?

There errors that im getting are as follows;

fbackup(1027): Backup did not complete : Reader or Writer process exit

SYSFBACKUP: tape number 1 PID 16172 completed with warnings, status 4 Run time in seconds = 6374

SYSFBACKUP ERROR: tape number 2 PID = 16173, signal 9
Run time in seconds = 13526

These are the error codes im getting;

fbackup(3205): WARNING: unable to read a volume header
fbackup(9999): I/O error
fbackup(3120): write error on a record in the index

SYSFBACKUP: tape number 1 PID 26531 completed with warnings, status 4
SYSFBACKUP ERROR: tape number 2 PID = 26532, signal 9
SYSFBACKUP ERROR: 1 backups failed!!! 1 completed

Im not sure if its a data size issue with the tapes or that the tape drives are malfunctioning? If anyone can shed some light for me, very much appreciated!

Thanks,
Chris
chrisam@rocketmail.com

[Robin Wakefield]

The fact that you say it's sporadic could indicate a hardware problem, either old tapes, dirty head. I wonder if you can try cleaning the drives and use new(er) tapes.

Anything in dmesg?

Robin.

[Vito Sarducci]

This is what is in my dmesg:

SCSI: Request Timeout -- lbolt: -2045657717, dev: cd032000
lbp->state: 2020
lbp->offset: ffffffff
lbp->uPhysScript: a80000
From most recent interrupt:
ISTAT: 22, SIST0: 04, SIST1: 00, DSTAT: 80, DSPS: 00000006
NCR chip register access history (most recent last): 2576163 accesses
1, SCRATCHA: ff000060
1, SCRATCHA0<-40
0, SCRATCHA1<-10
0, SXFER<-00
0, SCNTL3<-33
0, DSP<-00a80040
128, ISTAT: 22
2, SIST0: 04
2, SIST1: 00
2, DSTAT: 80
1, DSPS: 00000006
1, SCRATCHA: ff001040
2, DSP: 00a80058
2, SCRATCHA1<-00
0, CTEST3<-04
0, STEST3<-82
lsp: 1b152780
bp->b_dev: cd032000
scb->io_id: 3728917
scb->cdb: 0a 00 02 00 00 00
lbolt_at_timeout: -2045717817, lbolt_at_start: -2045717817
lsp->state: 10d
lbp->owner: 1b152780
scratch_lsp: 0
Pre-DSP script dump [15ed5030]:
980dff00 0000000a 78351000 00000000
0e000002 00a803c0 80000000 00000000
Script dump [15ed5050]:
9f0b0000 00000006 0a000000 00a803c8
721a0000 00000000 c0000004 00a8035c

SCSI: Abort -- lbolt: -2045657717, dev: cd032000, io_id: 3728917
SCSI TAPE: dev = 0xcd032000 I/O error during close

SCSI: Abort -- lbolt: -2044962119, dev: cd032000, io_id: 375fa2a

Im not sure if this is my tape drives or not?

I have several more questions to poke around this problem?

1. Is there a way I can test the drives during the day without interruption. I dont want to backup active files that are currently being accessed.

2. These are DLT7000's so I know there is a limitation as the the maximum amount of data that can be stored to these tapes.

3. How can I tell if compression is on?

4. The funny thing about this error is it happens sporadically in that one day its the left dlt and another day its the right dlt?

5. Is there a known HP issue regarding daisy chaining the two DLT drives?

thanks,

Chris
chrisam@rocketmail.com

[Robin Wakefield]

If you have some old logfiles, just set up a loop to back them up to the no-rewind device, that way you'll fill the tape up.

There should be a compression led indicator on the front of the drive. Also run "lssf" on the device file and make sure it's set to "best density".

dmesg indicates SCSI problems - obvious things to check are loose cables/connectors before you call HP.

Robin.

[Vito Sarducci]

Im using DLT-IV tapes in the Dlt7000 drives? I tackling this from a hardware perspective first, then going after my current fbackup process. I have 7 other HP systems doing the exact same fbackup process and none of them are having problems?

Is there an issue regarding daisy chaining the two dlt7000 devices? I heard from a source there is? Can anyone confirm this?

[A. Clay Stephenson]

Hi,

You actually have indicated the true nature of the problem. More than one DLT7000 per SCSI bus is not supported. One of the drives will occasionally be starved for data. A maximun of 2 DLT4000's/bus is ok but only 1 DLT7000.

This is a situation where it almost works - just well enough to drive you crazy. The preferred configuration is 1 DLT7000 by itself on a bus.

Regards, Clay

[Vito Sarducci]

Where can i find official documentation from HP regarding this Dlt7000/Dlt8000 daisy chain SCSI issue?

I have a concern with this because our local HP SE who has been with us for years and years configured these drives into our servers daisy chained without hesitation!

I have several other HP's K and N class servers that are running the exact identical daisy chain and they all work fine without this sporadic problem.

So if someone can point me in the right direction, i need official docs to validate my test scenario of going back to old dlt4000s on this system and testing for a week of backups to see if they fail like there doing now.

Does anyone have other solid evidence that this scenario "is not" supported by HP?

thanks,

Chris
chrisam@rocketmail.com

[A. Clay Stephenson]

Hi Chris:

If you have access to an 'HP Server Ordering Guide' or 'Guide to Ordering Hardware, Software' and Support' or an 'Hp Configuration Guide' you will find that it clearly states
'Each DLT7000 Tape Drive mechanism requires a dedicated data path to Host' and 'Two DLT 4000 Tape Drives can be supported per data path to Host'.

Regards, Clay

[Michael Francisco · Accepted solution]

I have a K260 with an HP DLT Tape Library (Model A4851A) which holds two DLT tape drives. Page 2-8 of the manual (HP Part number A4851-90000 E1197) shows the three methods of attaching the library to the host.

1. Single drive library - cable from host to bus one; terminate bus one.

2. Two drive library Daisy chained - cable from host to bus one; cable from bus one to bus two, terminate bus two.

--> There is a note indicating that Daisy chained is for DLT4000 drives only. <--

Of course, there is
3. Two drive library - bus 1 and bus 2 connected to separate SCSI cards.

There is also a note on page 2-7 which reads...

"
*number of drives in the library and drive type (only DLT4000 drives are supported as daisy-chained)
"

A pdf version of this manual can be found here:
http://www.hp.com/cposupport/manual_set/lpg28402.pdf (The pages you need are 33&34)

Hope this helps!
Michael

[Vito Sarducci]

My mistake, these drives are Quantum DLT8000's not 7000's. I noticed the firmware for the dlt8000 drives were not identical? One was at revision 15 and one was at revision 20? Would this make the difference of why the drives are sporadically causing the backup to fail?


All the help from all of you is great and very much appreciated!

[A. Clay Stephenson]

Hi Chris,

While I would like to have the firmware levels the same, the fact that you are seeing SCSI timeouts is the classic symptom of too many tape drives on a common bus. I once tried this configuration on a D-box and saw exactly what you are seeing - it almost works perfectly. The worst kind of problem to have.
Buy another controller, cable, and terminator and your problems will be over.

Regards and good luck, Clay

[Robin Wakefield]

Also make sure you have the latest patches, otherwise you won't get anywhere near the full 80Gb out of them, should you need it.

Robin.

[Vito Sarducci]

Im going to have HP come out and update the firmware so both drives are current. What bothers me about this whole thing is that i have 6 other HP Kclass and Nclass servers with two Dlt's in the same configuration of using one SCSI controller and they have not had this problem for over a year since the Dlt8000's were installed? This just doesnt make much sense?

[Celso Medina Kern]

Hi Chris,

You?ve definitively found your answer. I?ve been a HP hardware engineer for several years and DLT7000 should be never put in the same SCSI bus(Fast Wide SCSI).

My thought about your backup solution, considering you have several hosts, is why you don?t use better tool to manage them, for instance, Omniback. You would get:
- automatic media management;
- better performance;
- graphycal monitoring & scheduling;
- graphycal centralized management;
- notification & reporting;
- online backups;
- load balancing;
- more robust backups;
- easy and fast restores
- ...

PS: I am not a sales person, just an Omniback admirer.

Sorry if it is a "wet place rain" as we say in Brazil.

[Vito Sarducci]

Celso, Clay, Francesco, and Robin,

I thank you and do believe we have pin pointed the issue. ok, since i have Dlt8000's and Dlt7000's on these boxes, what avenue should I take to correct the problem?

Some of my Kclass and Nclass servers do not have another SCSI slot available? What can I do? HP installed these Drives on these systems?

What are your thoughts all?

thanks,

Chris
chrisam@rocketmail.com

[A. Clay Stephenson]

Hi Chris,

The only answers I can think of are not cheap.
When you say you have no scsi slots available does that mean you have no HSC slots in your K boxes? Can you add an Expansion I/O card to give you more HSC slots? The bad news is that an HP-PB slot will do you no good because those SCSI controllers are not quite fast enough to stream a DLT7000 or 8000; you need the HSC SCSI controllers. On the N-boxes do you mean than there are no mode PCI slots?

If you truly have no more slots then you need to look at Fibre Channel and a Fibre Channel to SCSI Multiplxer. This will not be cheap.

Another option to look at is to use OmniBack with a tape library. Hopefully you can directly
connect the hosts with the most data to tape drives in the library. The good news is that the robitics and some tape drives can be directly connected by SCSI to one host. Other tape drives within the library can be SCSI connected to another hosts and it all works well. Those hosts not directly connected can backup over the network.

If it were me and I could get the budget, OB2 and a tape library would be my way to go. Life will be a lot simpler.

The other answer and again not cheap is SAN.

Regards, Clay

[A. Clay Stephenson]

Hi again Chris,

One other thought: Since these drives were installed by HP, I would call my HP rep and see what he might be willing to do. You might be able to at least get a trade-in credit on some of your drives and a really good deal on OB2.

[Vito Sarducci]

Im going to explore that option as we speak with my boss. Im not sure what we can get out of HP, but if the firmware update does not fix this problem, then HP will have to anty up and come up with a game plan to curb this possible problem with all my systems.

HP likes those types of bottlenecks. Makes them $$$$

thanks clay,

Chris
chrisam@rocketmail.com

[Celso Medina Kern]

Hi Chris,

Clay has given you a very good hardware roadmap for your backup solution. Unfortunately there is no cheap exit.

Omniback in a shared library configuration is heaven for your backup solution. Even better if you have Storage Area Network.

Thinking about the current problem, I want to give you any other possibility I could see, even it seems to be stupid:
- make your DLT drives slower, unsetting hardware compression on both drives;
- make fbackup slower, reducing blocks_per_record and file_reader processes in fbackup config file;
- if you do not have problems with backup window, choose less i/o intensive backups to run concurrently, or do not run concurrent backups using the same scsi bus at all. In this case, with Omniback, you could set 2 drives to work in cascade mode, what means you would have the media in second drive automatically continuing your backup without manual media change)

If you choose Omniback to make backups with the current daisy chainned DLTs and concurrent backups, the problem tend to become worse since Omniback reader and writer processes are more powerfull than fbackup.

Another possibility is to analyze your K-Class I/O to see if there is a way two replace cards by new ones that occupy less slots, allowing you to include a second F/W scsi card. Even not being the best solution, maybe you could put one drive in HP-PB bus, the probability of not streaming DLT because HP-PB botleneck depends on what other cards are in the same HP-PB, but your fbackup would never hang.

If you like i could take a look in your `ioscan -fnC ext_bus` and `model` commands output to double check hardware possibilities.

I looked at www.hp.com and the prices in case you want to install an HSC expander is:
A3186A K4XX/K360 HP-HSCslotexpans $4,080.00
A2969A 20 MB FWD SCSI-2 interface $1,320.00

If replacing technology is in question, I strongly recommend LTO/Ultrium drives, may be worth replace 2 DLTs by 1 Ultrium. I recently installed a brand new system N4000/FC60/Ultrium/Omniback that backup 87GB in 50 minutes using just 1 drive. It is operating system and Oracle datafiles backup basically. The same data in K570/Autoraid/DLT7000/fbackup used to take 4 hours.

Best regards