StoreEver Tape Storage
1748103 Members
4675 Online
108758 Solutions
New Discussion юеВ

Mounting LTFS failed after data was written (unmount interrupted)

 
Wuxia
Member

Mounting LTFS failed after data was written (unmount interrupted)

I have problem mounting an LTFS LTO6 tape which was just recorded on an Ultrium 6650. After recording my data I deleted the original files (big mistake) and unmounted the tape. Before the tape was successfully unmounted though there was a short power outage which restarted my computer. I then tried to mount the tape but all I got was:

78b2 LTFS14000I LTFS starting, QUANTUMLTFS Standalone version 2.2.0, log level 2
78b2 LTFS14058I LTFS Format Specification version 2.2.0
78b2 LTFS14104I Launched by "ltfs /mnt/ltfs"
78b2 LTFS14105I This binary is built for Linux (x86_64)
78b2 LTFS14106I GCC version is 4.4.7 20120313 (Red Hat 4.4.7-16)
78b2 LTFS17087I Kernel version: Linux version 2.6.32-504.30.3.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-11) (GCC) ) #1 SMP Wed Jul 15 10:13:09 UTC 2015 x86_64
78b2 LTFS17089I Distribution: CentOS release 6.6 (Final)
78b2 LTFS17089I Distribution: CentOS release 6.6 (Final)
78b2 LTFS17089I Distribution: LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
78b2 LTFS17089I Distribution: CentOS release 6.6 (Final)
78b2 LTFS14063I Sync type is "time", Sync time is 300 sec
78b2 LTFS17085I Plugin: Loading "ltotape" driver
78b2 LTFS17085I Plugin: Loading "unified" iosched
78b2 LTFS20013I Drive type is HP LTO6, serial number is xxxxxxxxxx, firmware revision is O39D
78b2 LTFS17160I Maximum device block size is 1048576
78b2 LTFS11005I Mounting the volume
78b2 LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 0
78b2 LTFS11170E Failed to read label (-1012) from partition 0
78b2 LTFS11009E Cannot read volume: failed to read partition labels.
78b2 LTFS10023I LTFS volume information:
78b2 LTFS10031I Volume Name     : (null)
78b2 LTFS10024I Volser(Barcode) :
78b2 LTFS10025I Volume UUID     :
78b2 LTFS10026I Format Time     : 1970-01-01 02:00:00.000000000 EET
78b2 LTFS10027I Block Size      : 524288
78b2 LTFS10028I Compression     : Disabled
78b2 LTFS10029I Index Partition : ID = #000, SCSI Partition = 0, Total Capacity = 0 MiB, Available Space = 0 MiB
78b2 LTFS10030I Data Partition  : ID = #000, SCSI Partition = 0, Total Capacity = 0 MiB, Available Space = 0 MiB
78b2 LTFS14013E Cannot mount the volume

 

I tried ltfsck with various options but it just said the same:

LTFS17089I Distribution: CentOS release 6.6 (Final)
LTFS17089I Distribution: CentOS release 6.6 (Final)
LTFS17089I Distribution: LSB_VERSION=base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
LTFS17089I Distribution: CentOS release 6.6 (Final)
LTFS17085I Plugin: Loading "ltotape" driver
LTFS20013I Drive type is HP LTO6, serial number is xxxxxxxxxx, firmware revision is O39D
LTFS17160I Maximum device block size is 1048576
LTFS16014I Checking LTFS file system on '/dev/st0'
LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 0
LTFS11170E Failed to read label (-1012) from partition 0
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS16080E Cannot check volume (8)

 

I'm pretty sure the information is there and there is even an index written at the end if only I have a way to format the media without destroying the data and then run ltfsck that will probably get my data back but I can't figure out how to do it. Is there a way back for my lost data? Thanks.

7 REPLIES 7
Wuxia
Member

Re: Mounting LTFS failed after data was written (unmount interrupted)

After some thought probably the best thing for me to do is to try to dump the tape to file (with something like dd) and try to extract the files one by one using photorec - all files on tape are types that photorec should understand. I also have a list of all the files because wrote a list locally for reference before the unmounting.

Whatever happend to the tape just make me doubt the usefulnes of LTFS as archiving solution - it should definetly be more robust than this. I simply don't undertand how is this possible.

1. If the recording is finished and only the index partition is not updated it still should be able to restore the files by using the index at the end of the data partition.

2. If the Inxex partition is damaged how it was possible the tape to be mounted and written to in the first place?

I'm not an LTFS expert so any insights on this will be greatly appreciated.

Wuxia
Member

Re: Mounting LTFS failed after data was written (unmount interrupted)

I'm kind of talking to myself here but anyway. I tried to dump the data partition but I cannot go beyond the EOD marker of the bad index partition. Is there any way to position the tape at the data partition?

Curtis_Ballard
HPE Pro

Re: Mounting LTFS failed after data was written (unmount interrupted)

LTFS tapes use two partitions and all commands are performed on the "active" partition. 

mt has a 'setpartition' command that will let you change the active partition.  If the tape is correctly formated for LTFS then you should have two different partitions labeled 0 and 1.  A dd following a load will only access partition 0.  You should be able to use mt and the setpartition command to switch to partition 1 and then dd will be able to read that partition.

It looks like you are using the Quantum LTFS software.  Have you tried contacting Quantum or possibly using another vendor's LTFS software?  Since LTFS is an open format the LTFS software from any source should be able to read tapes written using any other LTFS software.  It is possible that the recovery functions in a different vendor's software may be more robust.


I work for HPE

Accept or Kudo

Wuxia
Member

Re: Mounting LTFS failed after data was written (unmount interrupted)


@Curtis_Ballard wrote:

LTFS tapes use two partitions and all commands are performed on the "active" partition. 

mt has a 'setpartition' command that will let you change the active partition.  If the tape is correctly formated for LTFS then you should have two different partitions labeled 0 and 1.  A dd following a load will only access partition 0.  You should be able to use mt and the setpartition command to switch to partition 1 and then dd will be able to read that partition.


I've tried setpartition, but it didn't work both on the bad tape an on a good ltfs tape:

 

# mt -f /dev/nst0 setpartition 1
/dev/nst0: Invalid argument

I've also tried with setpartition 0,  partseek 0 0, partkseek 0 1. Same error. What am I doing wrong here?

@Curtis_Ballard wrote:

LTFS tapes use two partitions and all commands are performed on the "active" partition. 

It looks like you are using the Quantum LTFS software.  Have you tried contacting Quantum or possibly using another vendor's LTFS software?  Since LTFS is an open format the LTFS software from any source should be able to read tapes written using any other LTFS software.  It is possible that the recovery functions in a different vendor's software may be more robust.

Not yet. I'll try that, thanks.

 

BTW if I dd the bad index partition record by record  the first file I get is empty, then the next one is the actual xml index and the third one is empty. If I do the same on a working LTFS tape I get two extra files before that: first the tape label and then a short xml which are missing from the bad tape so they must have been overwritten somehow which is the reason the tape can not be mounted. May be I could try to rebuild the index by writing the missing  label and xml file and then add the actual xml index from the same tape manually. Will that work? Something like:

  1. New label (read from a working tape)
  2. New partition description xml (read from a working tape)
  3. Empty file
  4. Index xml (read from the same tape)
  5. Empty file

But that will probably be the last thing to try as it seems dangerous.

ChrisMartin
Occasional Advisor

Re: Mounting LTFS failed after data was written (unmount interrupted)

From the description it does sound like the index partition (partition 0) has been overwritten, probably with an index.  As you have correctly surmised, the index partition should start with an 80-byte label, then a filemark, an XML volume label, two filemarks, then a copy of the XML index.  When the repair tool ltfsck starts, it looks for the label to confirm that this is an LTFS volume.  If it doesn't find one then it takes the only "safe" default behaviour and stops, since to start making assumptions at this stage could cause more damage...  I don't think in this instance that trying software from a different vendor would help, because as far as I know this functionality is pretty common across most vendors.

It is possible to recreate the index partition but the process is very manual and I don't believe there are any publically-available utilities to help.  It's really important to only write to the index partition, since a write to the start of the data partition would destroy the data as well...  Copying data from the tape before starting the repair attempt is always a wise move (but apologies, I'm not sure how to do that using only standard tools like dd).  The LTFS format specification (available publically at http://www.snia.org/ltfs) gives details of what is expected in the labels etc.  The key thing is probably to get the first few records correctly onto tape; then the repair utility ltfsck should be able to fix up the actual index for you once it recognizes partition 0 as belonging to an LTFS volume.

_eMaX_
Occasional Visitor

Re: Mounting LTFS failed after data was written (unmount interrupted)

I've exactly that problem, too. I can format ltfs tapes as much as I want, but I can never read them...

 

 root@monster юВ░ /usr/local/lib юВ░ ltfsck /dev/nst0                                                                                                [2023-12-23 18:47:31]
LTFS16000I Starting ltfsck, HPE StoreOpen Software version 3.5.0, log level 2
LTFS16088I Launched by "ltfsck /dev/nst0"
LTFS16089I This binary is built for Linux (x86_64)
LTFS16090I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-11)
LTFS17087I Kernel version: Linux version 6.5.0-14-generic (buildd@lcy02-amd64-031) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 i386
LTFS17089I Distribution: PRETTY_NAME="Ubuntu 23.10"
LTFS17089I Distribution: DISTRIB_ID=Ubuntu
LTFS17085I Plugin: Loading "ltotape" driver
LTFS20013I Drive type is HP LTO6, serial number is **removed**
LTFS17160I Maximum device block size is 524288
LTFS17157I Changing the drive setting to write-anywhere mode
LTFS16014I Checking LTFS file system on '/dev/nst0'
LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 4096
LTFS11170E Failed to read label (-1012) from partition 0
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS16080E Cannot check volume (8)
LTFS20076I Triggering drive diagnostic dump
LTFS20096I Diagnostic dump complete
 тЬШ  root@monster юВ░ /usr/local/lib юВ░ ltfsck /dev/nst1                                                                                              [2023-12-23 18:47:47]
LTFS16000I Starting ltfsck, HPE StoreOpen Software version 3.5.0, log level 2
LTFS16088I Launched by "ltfsck /dev/nst1"
LTFS16089I This binary is built for Linux (x86_64)
LTFS16090I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-11)
LTFS17087I Kernel version: Linux version 6.5.0-14-generic (buildd@lcy02-amd64-031) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 i386
LTFS17089I Distribution: PRETTY_NAME="Ubuntu 23.10"
LTFS17089I Distribution: DISTRIB_ID=Ubuntu
LTFS17085I Plugin: Loading "ltotape" driver
LTFS20013I Drive type is HPE LTO9, serial number is **removed**
LTFS17160I Maximum device block size is 524288
LTFS17157I Changing the drive setting to write-anywhere mode
LTFS16014I Checking LTFS file system on '/dev/nst1'
LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 4096
LTFS11170E Failed to read label (-1012) from partition 0
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS16080E Cannot check volume (8)
LTFS20076I Triggering drive diagnostic dump
LTFS20096I Diagnostic dump complete

 

[Moderator edit: Erased the confidential info]

 

_eMaX_
Occasional Visitor

Re: Mounting LTFS failed after data was written (unmount interrupted)

Here's a more complete workflow. You see that I first format the drive, then unload and reload it and then try to do an ltfsck - which always fails. This happens both with my LTO6 as wel as my LTO9 drive.  Using tar to write to those same drives works with no problem. Changing the blocksize all the way down to 64 k doesn't help either.

 

 тЬШ  root@monster юВ░ /usr/local/lib юВ░ mkltfs --device=/dev/st0 --tape-serial="600001" --volume-name="600001"                                        [2023-12-23 19:15:38]
LTFS15000I Starting mkltfs, HPE StoreOpen Software version 3.5.0, log level 2
LTFS15041I Launched by "mkltfs --device=/dev/st0 --tape-serial=600001 --volume-name=600001"
LTFS15042I This binary is built for Linux (x86_64)
LTFS15043I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-11)
LTFS17087I Kernel version: Linux version 6.5.0-14-generic (buildd@lcy02-amd64-031) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 i386
LTFS17089I Distribution: PRETTY_NAME="Ubuntu 23.10"
LTFS17089I Distribution: DISTRIB_ID=Ubuntu
LTFS15003I Formatting device '/dev/st0'
LTFS15004I LTFS volume blocksize: 524288
LTFS15005I Index partition placement policy: None

LTFS17085I Plugin: Loading "ltotape" driver
LTFS20013I Drive type is HP LTO6, serial number is **confidential info erased**
LTFS17160I Maximum device block size is 524288
LTFS17235I Formatting the media will place drive in write-anywhere mode
LTFS17157I Changing the drive setting to write-anywhere mode
LTFS15049I Checking the medium (mount)
LTFS15010I Creating data partition b on SCSI partition 1
LTFS15011I Creating index partition a on SCSI partition 0
LTFS17165I Resetting the medium's capacity proportion
LTFS11097I Partitioning the medium
LTFS20107I Replacing existing MAM barcode BPE384L6                         with user-specified 600001
LTFS11100I Writing label to partition b
LTFS11278I Writing index to partition b
LTFS11100I Writing label to partition a
LTFS11278I Writing index to partition a
LTFS15013I Volume UUID is: 77dd822f-85e9-435c-8d9c-72dfda7c6b63

LTFS15019I Volume capacity is 2350 GB
LTFS20076I Triggering drive diagnostic dump
LTFS20096I Diagnostic dump complete
LTFS15024I Medium formatted successfully
  root@monster юВ░ /usr/local/lib юВ░ mtx -f /dev/sg7 unload 17 0                                                                                     [2023-12-23 19:17:18]
Unloading drive 0 into Storage Element 17...done
  root@monster юВ░ /usr/local/lib юВ░ mtx -f /dev/sg7 load 17 0                                                                                       [2023-12-23 19:18:15]
Loading media from Storage Element 17 into drive 0...done
  root@monster юВ░ /usr/local/lib юВ░ ltfsck /dev/st0                                                                                                 [2023-12-23 19:19:11]
LTFS16000I Starting ltfsck, HPE StoreOpen Software version 3.5.0, log level 2
LTFS16088I Launched by "ltfsck /dev/st0"
LTFS16089I This binary is built for Linux (x86_64)
LTFS16090I GCC version is 4.8.5 20150623 (Red Hat 4.8.5-11)
LTFS17087I Kernel version: Linux version 6.5.0-14-generic (buildd@lcy02-amd64-031) (x86_64-linux-gnu-gcc-13 (Ubuntu 13.2.0-4ubuntu3) 13.2.0, GNU ld (GNU Binutils for Ubuntu) 2.41) #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Nov 14 14:59:49 UTC 2023 i386
LTFS17089I Distribution: PRETTY_NAME="Ubuntu 23.10"
LTFS17089I Distribution: DISTRIB_ID=Ubuntu
LTFS17085I Plugin: Loading "ltotape" driver
LTFS20013I Drive type is HP LTO6, serial number is  **confidential info erased**
LTFS17160I Maximum device block size is 524288
LTFS17157I Changing the drive setting to write-anywhere mode
LTFS16014I Checking LTFS file system on '/dev/st0'
LTFS11175E Cannot read ANSI label: expected 80 bytes, but received 4096
LTFS11170E Failed to read label (-1012) from partition 0
LTFS11009E Cannot read volume: failed to read partition labels.
LTFS16080E Cannot check volume (8)
LTFS20076I Triggering drive diagnostic dump
LTFS20096I Diagnostic dump complete
 тЬШ  root@monster юВ░ /usr/local/lib юВ░