HPUX Ingite Recovery Tape Problems

[Serializer]

Hello,

I have got problems with accessing the tape drive.

We have replaced the old tape by a brand new one. There was no "initialization/format" done on the tape, necessary?

Here I attach some outputs. For me it looks like the resp. device file is corrupted. Anyone could help me diagnose and heal it?

I have heard about "insf" to rebuild it, but I am a bit scared to run it on a critical system. Is there a way to remake just the device file which is used (unix tape devices).

UNAME:

--

uname -a:

HP-UX server01 B.11.11 U 9000/800 256805121 unlimited-user license

--

IGNITE (here we realized there is a problem):

--

/opt/ignite/bin/make_tape_recovery -I -x inc_entire=vg00

Error:

ERROR: Failed to open tape device /dev/rmt/0mn:No such device or address
(errno = 6).

--

DEVICES:

--

bash-3.1# pwd
/dev/rmt
bash-3.1# ls -l
total 0
crw-rw-rw- 2 bin bin 205 0x035000 Apr 5 2006 0m
crw-rw-rw- 2 bin bin 205 0x035080 Apr 5 2006 0mb
crw-rw-rw- 2 bin bin 205 0x035040 Sep 25 19:02 0mn
crw-rw-rw- 2 bin bin 205 0x0350c0 Apr 5 2006 0mnb
crw-rw-rw- 2 bin bin 205 0x035000 Apr 5 2006 c3t5d0BEST
crw-rw-rw- 2 bin bin 205 0x035080 Apr 5 2006 c3t5d0BESTb
crw-rw-rw- 2 bin bin 205 0x035040 Sep 25 19:02 c3t5d0BESTn
crw-rw-rw- 2 bin bin 205 0x0350c0 Apr 5 2006 c3t5d0BESTnb
crw-rw-rw- 1 bin bin 205 0x035001 Apr 5 2006 c3t5d0DDS
crw-rw-rw- 1 bin bin 205 0x035081 Apr 5 2006 c3t5d0DDSb
crw-rw-rw- 1 bin bin 205 0x035041 Apr 5 2006 c3t5d0DDSn
crw-rw-rw- 1 bin bin 205 0x0350c1 Apr 5 2006 c3t5d0DDSnb
crw-r--r-- 1 bin bin 205 0xfffffe Apr 5 2006 stape_config

--

MT:

--

bash-3.1# mt -f /dev/rmt/0mn stat
Drive: HP C7438A
Format:
Status: [0]
File: 0
Block: 3

--

MT (more):

--

bash-3.1# mt -f /dev/rmt/0mn rew
/dev/rmt/0mn: No such device or address
bash-3.1# mt -f /dev/rmt/0mn fsf
/dev/rmt/0mn: No such device or address
bash-3.1# mt -f /dev/rmt/0mn bsf
/dev/rmt/0mn: No such device or address

--

TAR:

--

bash-3.1# tar cvf /dev/rmt/0mn /tmp/1/
tar: cannot open /dev/rmt/0mn

bash-3.1# tar tvf /dev/rmt/0mn /tmp/1/
tar: cannot open /dev/rmt/0mn

--

IOSCAN:

--

bash-3.1# ioscan -fnCtape
Class I H/W Path Driver S/W State H/W Type Description
=====================================================================
tape 0 0/1/1/1.5.0 stape CLAIMED DEVICE HP C7438A
/dev/rmt/0m /dev/rmt/c3t5d0BEST /dev/rmt/c3t5d0DDS
/dev/rmt/0mb /dev/rmt/c3t5d0BESTb /dev/rmt/c3t5d0DDSb
/dev/rmt/0mn /dev/rmt/c3t5d0BESTn /dev/rmt/c3t5d0DDSn
/dev/rmt/0mnb /dev/rmt/c3t5d0BESTnb /dev/rmt/c3t5d0DDSnb

--

regards,

Serializer

[RAJD1]

>> Error:

ERROR: Failed to open tape device /dev/rmt/0mn:No such device or address
(errno = 6).

- Check if there is already a tape loaded or not.

# mt -t /dev/rmt/0mn status

It should say "BOT Online" ...

-It is showing " Status: [0]"  so seems there is no tape loaded.

- You can go to the datacenter and take a look on the tapedrive , LED status, after the tape is loaded. 

Cheers,

RajD.

[Matti_Kurkela]

> I have heard about "insf" to rebuild it, but I am a bit scared to run it on a critical system.

"insf" without options does not "rebuild" anything that already exists. It only adds new device nodes for hardware/drivers that don't already have them.

If you want to rebuild the existing device nodes, the "-e" option of the insf command  is needed.

> Is there a way to remake just the device file which is used (unix tape devices).

Yes, there is. With additional options, you can restrict "insf -e" to remake the device nodes for a particular device or device class only.

For example, this remakes tape device nodes only:

insf -e -Ctape

This remakes only the device with the specified hardware path:

insf -e -H 0 0/1/1/1.5.0

[Serializer]

Helloo,

at first thanks Rajd1 and Matti.

I will have to try this.

Fortunately we have another identical HPUX running on the site, so I could use it for reference with this result:

HPUX In-Question (bad):

--

bash-3.1# mt -f /dev/rmt/0mn stat
Drive: HP C7438A
Format:
Status: [0]
File: 0
Block: 3

--

HPUX Reference (good):

--

# mt -f /dev/rmt/0mn stat
Drive: HP C7438A
Format:
Status: [81114700] EOF online compression immediate-report-mode
File: 2
Block: 0

--

What sucks is that, the crew tells me, there _is_ a tape in the drive, although referring to the reply before it looks like the drive was emtpy.

I will give a try to the "insf". Hopefully this wont trigger any nukes ;)

regards,

Serializer

[Bill Hassell]

>> What sucks is that, the crew tells me, there _is_ a tape in the drive, although referring to the reply before it looks like the drive was empty.

Does the crew know what a DDS tape looks like when it is loaded? When you pop out the tape, only about 3/8" of the tape cartridge sticks out. A properly loaded tape will be about 1/4" inside the front slot *and* there will be a green ready light. The previous status with [0] means there is no tape loaded. NOTE: someone can push a DDS4 tape into a DDS3 tape drive and it will look like it is loading. But wait for about a minute and it will pop out automatically because it is incompatible.

Now the status:

>> Status: [81114700] EOF online compression immediate-report-mode

indicates that the has written 2 files, then written an End-Of-File marker and is sitting somewhere in the middle of the tape. *BUT* if you try to read the tape, it will return an end of file error. SInce it is unusual to have the tape sitting in the middle somewhere, are you trying to save tape by writing additional backups to the same tape? BAD IDEA. Saving tape will eventually lead to a disaster - either someone loses the pierce of paper that lists the date and time and number of files, of someone forgets to space the tape forward and writes over the first file. NOTE: unlike reel-to-reel tapes from 20 years ago, modern streaming tape drives require very complex sync tracks, so the EOT marker is a firmware End-of-Tape mark written after the last data record has been written. It is virtually impossible to position past this EOT marker.

I would suggest sitting in front of the tape drive so you can see what is happening. Start with status checks and an empty drive, then load a tape and repeat. Then position the tape by wirting something simple like a tar backup of /tmp. Use the ATT rewind device file 0m and check the status, then use mt--rewind to return to BOT and write the tar backup again using 0mn, the ATT no-rewind option. And most important, read the man page for mt, specifically the details about the bizarre ATT options versus the sane Berkeley options.

For something as importrant as an Ignite backup, write a script that verifies the tape is ready and correctly positioned by issuning mt--rewind. Then run make_tape_recovery and do not use the mt--offline option when  make_tape_recovery returns a bad status.

[Serializer]

Bill,

thanks for you message. The situation is quite bizarr, unfortunately I cant be physically at the drives now. As said we run both HPUX in a cluster, each one has its own tape drive. 

Today I checked the Ignite backup logs, first I wonder if the tape on HPUX1 needs to be initialized, AFAIK its not necessary, its a tape. On the HPUX 2 it complains about missing files...

HPUX 1 (primary):

--

WARNING: list_expander returned a warning; please check the recovery.log file
for details.
ERROR: /usr/bin/dd if=/var/tmp/uxinstlf.recovery of=/dev/rmt/0mn bs=2k failed
to write LIF onto tape.
ERROR: Failed to generate LIF on tape .

* Creating local directories for configuration files and archive.

======= 10/13/12 20:00:33 METDST Started /opt/ignite/bin/make_tape_recovery.
(Sat Oct 13 20:00:33 METDST 2012)
@(#) Ignite-UX Revision C.6.7.79
@(#) net_recovery (opt) $Revision: 10.778 $

* Testing for necessary pax patch.
* Passed pax tests.
* Checking Versions of Ignite-UX filesets

* Creating System Configuration.
* /opt/ignite/bin/save_config -f
/var/opt/ignite/recovery/2012-10-13,20:00/system_cfg vg00
* Backing Up Volume Group /dev/vg00
* /usr/sbin/vgcfgbackup /dev/vg00
* Creating Map Files for Volume Group /dev/vg00
* /usr/sbin/vgexport -s -p -m /etc/lvmconf/vg00.mapfile /dev/vg00

* Creating Control Configuration.
* Creating Archive File List
* Creating Archive Configuration

* /opt/ignite/lbin/make_arch_config -c
/var/opt/ignite/recovery/2012-10-13,20:00/archive_cfg -g
/var/opt/ignite/recovery/2012-10-13,20:00/flist -n 2012-10-13,20:00 -r
pa -b 64 -d Recovery\ Archive -t 1 -i 3 -m t
* Saving the information about archive to
/var/opt/ignite/recovery/previews
* Creating The Tape Archive
4+0 records in
3+1 records out
I/O error

--

and HPUX 2 (backup):

--

* Cleaning up old configuration file directories
* Creating local directories for configuration files and archive.

======= 10/13/12 21:00:10 METDST Started /opt/ignite/bin/make_tape_recovery.
(Sat Oct 13 21:00:10 METDST 2012)
@(#) Ignite-UX Revision C.6.7.79
@(#) net_recovery (opt) $Revision: 10.778 $

* Testing for necessary pax patch.
* Passed pax tests.
* Checking Versions of Ignite-UX filesets

* Creating System Configuration.
* /opt/ignite/bin/save_config -f
/var/opt/ignite/recovery/2012-10-13,21:00/system_cfg vg00
* Backing Up Volume Group /dev/vg00
* /usr/sbin/vgcfgbackup /dev/vg00
* Creating Map Files for Volume Group /dev/vg00
* /usr/sbin/vgexport -s -p -m /etc/lvmconf/vg00.mapfile /dev/vg00

* Creating Control Configuration.
* Creating Archive File List
* Creating Archive Configuration

* /opt/ignite/lbin/make_arch_config -c
/var/opt/ignite/recovery/2012-10-13,21:00/archive_cfg -g
/var/opt/ignite/recovery/2012-10-13,21:00/flist -n 2012-10-13,21:00 -r
pa -b 64 -d Recovery\ Archive -t 1 -i 1 -m t
* Saving the information about archive to
/var/opt/ignite/recovery/previews
* Creating The Tape Archive
50424+0 records in
50424+0 records out
* /opt/ignite/data/scripts/make_sys_image -c n -d /dev/rmt/0mn -t n -s
local -n 2012-10-13,21:00 -m t -w
/var/opt/ignite/recovery/2012-10-13,21:00/recovery.log -u -R -g
/var/opt/ignite/recovery/2012-10-13,21:00/flist -a 32861890

* Preparing to create a system archive.

* Archiving contents of itlims02 via tar image to local device/dev/rmt/0mn.
pax: dr_sync/data/archlogs.tar.gz : No such file or directory
WARNING: The pax command returned a non-zero exit status (exit status 1).
* Creation of system archive complete.


======= 10/13/12 23:06:11 METDST make_tape_recovery completed with warnings

--

MT says this:

HPUX1:

--

bash-3.1# mt -f /dev/rmt/0mn status
Drive: HP C7438A
Format:
Status: [0]
File: 0
Block: 3

--

HPUX2:

--

root@itlims02 [/opt/adm/soe/log]
#mt -f /dev/rmt/0mn status
Drive: HP C7438A
Format:
Status: [0]
File: 2
Block: -1

--

[Bill Hassell]

>> I wonder if the tape on HPUX1 needs to be initialized

There is no such operation as tape initialization for modern tapes.

>> * Creating The Tape Archive
>> 4+0 records in
>> 3+1 records out
>> I/O error

The tape is bad or the tape drive is defective and needs to be replaced. Check syslog and you should see some nasty messages from EMS reporting the defective tape.

>> HPUX1:

>> # mt -f /dev/rmt/0mn status

>> Drive: HP C7438A
>> Format:
>> Status: [0]
>> File: 0
>> Block: 3

The status [0] indicates that there is no tape loaded or no status can be retrieved. The block count cannot be more than 0 with no tape, so there is definitely something wrong with the tape drive. Have you inserted a cleaning tape recently?

>> HPUX2:

>> pax: dr_sync/data/archlogs.tar.gz : No such file or directory

Actually, quite normal. A file that was present when the backup started has been removed, not uncommon on a busy system. The end of the information says:

>> make_tape_recovery completed with warnings

which is OK. Warnings do not prevent the archive from being finished.

>> #mt -f /dev/rmt/0mn status
>> Drive: HP C7438A
>> Format:
>> Status: [0]
>> File: 2
>> Block: -1

This is normal at the end of the backup. When the tape archive (pax) finished, the tape drive wrote an End-Ff-File marker and left the tape positioned past the EOF (the no-rewind option of the 0mn devicefile). Because this position is essentially undefined, the status is [0]. FIle 2 was the last file written and the block number is undefined. If the tape is then rewound, the drive automatically writes an EOf-Data marker which prevents accidental (future) unspooling of the tape when trying to read past the last valid file marker.