StoreEver Tape Storage
cancel
Showing results for 
Search instead for 
Did you mean: 

LTO3 drive cannot save 533 GB

Jose M. del Rio
Frequent Advisor

LTO3 drive cannot save 533 GB

Hi:
I'm trying to save about 530 GB to our LTO 960 drive.
The cartridge is LTO3 (400-800 GB).
The fbackup process exits with error 2 (normal EOT):


fbackup(1004): session begins on Sat Dec 24 17:30:00 2005
fbackup(3203): volume 1 has been used 2 time(s)
fbackup(3024): writing volume 1 to the output file /dev/rmt/0m
fbackup(3003): normal EOT
fbackup(3310): enter '^[yY]' when volume 2 is ready on /dev/rmt/0m, or '^[nN]' to discontinue:
fbackup(3004): writer aborting
fbackup(1002): Backup did not complete : Reader or Writer process exit


Although some part of the data is already compressed in disk, I don't think this is the problem for 2 reasons:

1) so far we have been saving successfully about 520 GB

fbackup(1004): session begins on Mon Dec 5 18:00:00 2005
fbackup(3203): volume 1 has been used 2 time(s)
fbackup(3024): writing volume 1 to the output file /dev/rmt/0m
fbackup(3055): total file blocks read for backup: 1112888644
fbackup(3056): total blocks written to output file /dev/rmt/0m: 1113024555

2) querying the the drives's compression logs (using TapeTools) I can see the drive is indeed compressing at about 2:1

- Sequential Access Counters
- Bytes Written from App :0x0000008238af8a88
- Bytes Written to Tape :0x00000042bafcdf24
- Bytes Read from Tape :0x000000000014a804
- Bytes Read to App :0x0000000000289525
- Cleaning Required :False
- Tape Usage Log
- Thread Count :5
- Total Datasets Written :0x00000000000fd0ba
- Total Write Retries :0
- Total Unrecovered Write Errors :0
- Total Suspended Writes :21
- Total Fatal Suspended Writes :0
- Total Datasets Read :0x0000000000000141
- Total Read Retries :0
- Total Unrecovered Read Errors :0
- Tape Capacity Log
- Main Partition Remaining Capacity :16525 (MB)
- Main Partition Maximum Capacity :400308 (MB)
- Performance Log
- Direction Changes per 100MB :4
- Data Rate After Compression :1.4 MB/s
- Maximum Data Rate :264 MB/s
- Data Rate Before Compression :2.5 MB/s
- Native Data Rate : 80 MB/S
- Data Compression Log
- Read Compression Percentage :196
- Write Compression Percentage :195
- MB to Host :2
- Bytes to Host :562469
- MB from Tape :1
- Bytes from Tape :305156
- MB from Host :533386
- Bytes from Host :1018504
- MB to Tape :273327
- Bytes to Tape :843556


If I understand correctly these data, 533 GB are read from disk and sent to the drive, which compresses them to 273 GB.
Thus:
1) why 273 GB written to tape occupy almost 400 GB?
2) why the "Remaining Capacity" is about 16 GB?

Thanks.
Regards.
14 REPLIES
Luk Vandenbussche
Honored Contributor

Re: LTO3 drive cannot save 533 GB

Hi,

Try is there is a difference if you use the device file /dev/rmt/0mb
Jose M. del Rio
Frequent Advisor

Re: LTO3 drive cannot save 533 GB

Hi Luk,
thanks for the suggestion.
Tried with 0mb.
Same result.

- Tape Capacity Log
- Main Partition Remaining Capacity :16075 (MB)
- Main Partition Maximum Capacity :400308 (MB)
- Performance Log
- Direction Changes per 100MB :4
- Data Rate After Compression :2.1 MB/s
- Maximum Data Rate :264 MB/s
- Data Rate Before Compression :4 MB/s
- Native Data Rate : 80 MB/S
- Data Compression Log
- Read Compression Percentage :1212
- Write Compression Percentage :194
- MB to Host :0
- Bytes to Host :6160
- MB from Tape :0
- Bytes from Tape :508
- MB from Host :533423
- Bytes from Host :918688
- MB to Tape :274202
- Bytes to Tape :986860
Marino Meloni_1
Honored Contributor

Re: LTO3 drive cannot save 533 GB

I do not know Unix, so I cannot know if you already do this, but do not appear to me: the compression is highly dependent of the source data, and if you are using data in your server, you will not always have the same compression ratio available.
The value you get from the drive may be are some kind of statistic or projections of the capacity....
I would suggest to use LTT system perfomance tool, using the restore test in order to create a defined amount of data with the same and known compression ratio, and then run the test and see how many data can be stored on the device
marino
Richard Bickers
Trusted Contributor

Re: LTO3 drive cannot save 533 GB

The logs clearly show a compression ratio of 2:1 and also that around 273GB have been written to tape (natively). That certainly shouldn't fill up a 400GB (native) tape!

Only things I can think of-
1) You've got shocking error rate - that can increase the amount of tape needed and reduce performance to the same extent. This sort of level is only possible if you've a head down which is very unlikely. I assume the drive is fairly new and in a good environment. It would also be picked up in the device analysis section of the LTT support ticket.
2) You're appending to the first set of data and therefore running out of tape on the second. This would be my guess but I'm not up on Unix to know how to specify this.

I notice the transfer rates are also very low. 2.5MB/s compressing to 1.4MB/s. This drive supports 80MB/s native! We have a whole section on performance at www.hp.com/support/pat if you want some tips. Main starting point is to use a 'proper' backup application and a well set up disk system (lots of spindles and RAID) capable of delivering data at much higher rates. You can use LTT system performace test (back-up pre-test) to measure the speed of you disks independently.

Anyway, whatever your performance, you should be able to get 400GB native onto a 400GB tape. The 16GB capacity remaining is there for around 5% margin. If, for instance, you do have a drop of error rate (normally less than 5%) then you'll still be able to get 400GB onto the tape. This is also useful for copying tapes - if you had no margin then you couldn't copy from one tape onto a tape that has slightly less error rate margin. Does that make sense?
It's more interesting when it's gone wrong
Jose M. del Rio
Frequent Advisor

Re: LTO3 drive cannot save 533 GB

Thanks Marino and Richard.

Marino:
We did run the sysperf test but for different reasons. It is used to test the read and write transfer rate of the file system. Nothing to do with the tape capacity itself. I don't get your point.


In fact, we have run all the TapeTools tests and everything is OK except this warning:
- The current SCSI configuration is likely to be limiting the performance
of the drive.
- Please check that your HBA is the correct type for the drive and that
the cabling is good.
- The SCSI configuration referenced is the one for which device analysis
was run and/or the support ticket was pulled.
- If this is not via your backup server then you may not have an issue.
- Current SCSI transfer rate limited to: 160 MB/sec. Recommended: 320
MB/sec. or better

Richard:
1) where do you get the shocking error rate from? I can't see anything related in the output I enclosed
2) we are not appending data the second time as fbackup takes the whole tape for itself
3) with respect to transfer rates (1.4 - 2.5 MB/s), I'm not sure how it is computed. The fact is we are saving 533 GB native (273 GB compressed) in about 9 hours. That is 16 MB/s (8 MB/s). It is still a very low rate but I think the bottleneck is the file system because the sysperf test (backup option) measured a read rate of 20 MB/s. We are using ATA disks in an EMC Clariion CX700 disk array. We are testing these days to increase the number of fbackup-reader processes.
4) thanks for the hints about the 5% margin and the URL (I found it very useful).

Jose M. del Rio
Frequent Advisor

Re: LTO3 drive cannot save 533 GB

Sorry, I think I did something wrong.
I assigned at the same time points to the answers and pressed "Submit points".
For some reason the 5 pts. to Richard were recorded as 0 pts.
Sorry.
Richard Bickers
Trusted Contributor

Re: LTO3 drive cannot save 533 GB

Hi Jose,

This is very odd. I can't see why you can't get more than 273GB onto a 400GB tape (all numbers native). I mentioned the shocking error rate as that does have a dramatic impact on capacity - but it would have to be shocking. If you attach an LTT ticket we can take a look. I doubt it's this but can check quickly from a ticket. Most likely reasoning would be contamination - what kind of environment are you operating in? These drives are good so I'd suspect some external influence if the error rate was dropping (actually increasing...).

Are you sure you're using LTO3 media (yellow cartridge?). Dumb question but LTO2 media (dark red) does give you 200GB native with a bit to spare...

We can tell all sorts of things from a ticket. If you can pull one after you've done your backup that would be best but a ticket from the drive with the tape in will give us most of what we need. Please use LTT 4.0. Use 'save as' and a name you can refer to. That will get you two files that start with your name 'xxxx_header.ltt' and 'xxxx_some_drive_ref.dat' in /logs. Please attach both if you can, if not then the it's .dat file has all the data in it.

Your performance figures seem odd too. The data suggests 1.4MB/s and you're getting 8! I'm puzzled...
It's more interesting when it's gone wrong
Jose M. del Rio
Frequent Advisor

Re: LTO3 drive cannot save 533 GB

Hi Richard,
- we are not working on a polluted atmosphere
- our tapes are pretty black ;-) but, yes, they are LTO3
- we have run LTT Read/Write test with the Whole Tape options and it has correctly written 400 GB. So we suspect it could be something related to compression or some kind of fbackup limitation.
- with respect to compression, we would like to run a LTT test to write, let's say, 550 GB with compression but, as far as we are aware:
1) the Read/Write test only writes native (1:1) data
2) the Data Compression test writes only a few data (< 10 min.).
Do you know a way to do it?
- in relation to an fbackup issue, we have just launched the same backup with dd instead of fbackup. We'll see the results.

Thanks for your offer to examine our support ticket. I'll be out of the office for some days but I'll try to upload one before leaving.
Best regards.
Marino Meloni_1
Honored Contributor

Re: LTO3 drive cannot save 533 GB

Hello Jose

Those black cartridges does not looks as HP cartridges, did you try with the HP one that come with the device?


To do the test (As I suggested in my previous post)
select "Sys Perf"
Select "Restore Pre-Test" tabs
her you can select the path, the compression ratio ( selecting 2:1 for example and writing down 750 GB data, you should be able to write them on a 400 GB native cartridge (LTO3) without problem

marino
Richard Bickers
Trusted Contributor

Re: LTO3 drive cannot save 533 GB

Hi Jose,

It looks to me like the compression is working OK. You're giving it 533GB and it's compressing down to 273GB. The mystery to me is why it's hitting EOT (end of tape) at that point.

It's true that's not HP media you're using as HP LTO3 media is yellow. Should work OK with other vendors though.

Marino's method should work well for creating known compressible data on your disk which you can then back up onto tape.

It would be nice to eliminate the application from the issue. The only other method is to use the Device Performance test but that only goes to 16GB max. It will at least show you compression is working for that amount. Sounds like we should increase that value in future...

We might learn more from the ticket. At least we'll know how well your drive has been writing data.
It's more interesting when it's gone wrong
Richard Bickers
Trusted Contributor

Re: LTO3 drive cannot save 533 GB

One more thought... On Windows we recommend customers disable RSM (Removable Storage Manager) which polls the drive every few seconds and which, in turn, causes a flush of data to tape. In most cases this impacts performance because each one requires a subsequent append but if performance is already low it's actually capacity that takes the hit due to the wasted dataset space from each flush.

If there's something similar going on with your system it could have a serious effect on capacity. Can you check if there are any 'background' services that might be talking to the drive?
It's more interesting when it's gone wrong
Jose M. del Rio
Frequent Advisor

Re: LTO3 drive cannot save 533 GB

It' definitely an fbackup issue.
We have tried with dd and everything has gone OK:
- Tape Capacity Log
- Main Partition Remaining Capacity :120182 (MB)
- Main Partition Maximum Capacity :400308 (MB)
It seems fbackup is wasting a looot of space.
We will be playing with fbackup parameters (blocksperrecord, checkpointfreq...).
Or it is maybe related to "fast search marks"...
Jose M. del Rio
Frequent Advisor

Re: LTO3 drive cannot save 533 GB

I forgot to mention we have been saving daily about 520 GB to the tape with no problem.
Our data grown recently to 533 GB and that was when the problem showed up.
So we suspect it's a matter of size.
Richard Bickers
Trusted Contributor

Re: LTO3 drive cannot save 533 GB

Glad you found the culprit. You can't really beat a 'proper' application for effective backup - both for capacity and performance - but that does mean spending money.

If you're interested, the ones HP recommends can be found from our compatibility web pages www.hp.com/go/connect. Follow the link on tape backup and then software compatibility.

For HP-UX it lists:
- CA BrightStor ArcServe 11.x
- HP Data Protector 5.5
- Native TAR, CPIO (wasn't expecting to see these!)
- Symantec Netbackup 5.1
- Legato Networker 7.2
- HP IgniteUX 2 stage boot from tape

If you get to the bottom of the fbackup issue it would be great to post it back out to the forum. You can't be the only one... Thanks.
It's more interesting when it's gone wrong