Image backup not found on tape

[Mike Smith_33]

I am helping a site recover from a flood. Upon arrival I discovered that none of the tapes had image backups for two of the system disks that were in the backup schedule on the tapes. After much work, I have the two nodes that were lost recovered to new hardware so I turned my attention to the backups. Cluster is a 3 node lan cluster with each node having its own system disk. They were only running backups from one node which had the other two system disks mounted. I reviewed the backup job and I did not see a problem with the command. No warnings are given during the backup, in fact it seems to indicate that the disks are backed up.

Last night I specifically set the job off myself and watched it exit with no errors. I loaded the tape and did a listing out to a file, all disks were there except for the two system disks mscp mounted from the other nodes. The backup command occurs after the tape is initted:


backup/nocrc/block=65534/ignore=(label,inter)/image /journal=journaldir:filename/media=compaction diskname: tapename:diskname.bck/save


I logged on to one of the nodes whose system disk was not backing up. I ran a simple image backup of its system disk to its tapedrive and listed the saveset and the files were there.

What in the world am I missing? System is ds15a running VMS 7.3-2. System is only up to update v9.0. Thanks in advance for any help.

[labadie_1]

Hello

you say the backup is on tapename:diskname.bck/save

Is tapename a logical pointing to nl: ?

I notice backup/nocrc
While recent tapes handle the crc, older tapes do not deal with this, but I do not think this may be the problem.

A backup/list of the tape shows what ? an empty save-set, no save-set at all ?

[Robert Gezelter]

Mike,

I would second the comment to check that logical and I would expand it to I would like to see the entire logfile.

Even though /NOCRC saves some processing, I have seen it uncover problems so I do NOT generally recommend /NOCRC.

- Bob Gezelter, http://www.rlgsc.com

[Mike Smith_33]

Tapename is just what i put in or purposes of the question. It is really mkb600:

A listing of savesets shows all the other savesets except the two remote system disks. They are just not there. I did a listing so I could see every file on the tape just to be sure.

The /nocrc works for the other drives so I dont think that is it.

Just now I used the tape drive on one of the systems that is not showing up on the tape to backup the system drive of one of the other cluster members. Just a quick and dirty backup job and it worked with no problem. I put a /log on the backup job and saw every file going to tape. I did the listing out to a file and all the files are there.

[Wim Van den Wyngaert]

Are you sure the backup command gets executed ? Do you have a log of it ? No "exit" before the backup command to disable them ?

Wim

[Mike Smith_33]

$ show logical mkb600 yields nothing
$ show device mkb600 gives

nodename$mkb600:

[Mike Smith_33]

backup job consists of a list of disk and a subroutine to do the actual backup. It actually goes through one by one. One of these is first in the list, the other is third. No exits and I actually watched it run. What I want to try next is to run the job with the /log on the backup command.

[labadie_1]

How long does it last ? 5 seconds or a more reasonable 20 minutes ?

May be you can check in SDA the open channels, and if the tape is used ?
sda> sh proc/id=xxx/chan

and a basic
sh dev mkb600:/fu
does show the item
Operations completed
increasing during the backup ?

[Wim Van den Wyngaert]

Was the journal file created ? Anything in it ? May be errors were redirected with set mes or define sys$error ?

Wim

[Mike Smith_33]

The journal file shows both missing disk and their files. I am not seeing any errors or problems listed?

[Volker Halle]

Mike,

some accidental /REWIND or $ INIT MKB400: between the BACKUP commands ? Is image accounting turned on ? If so, look at the image accounting records for BACKUP.EXE ? Does the no. of image accounting records for BACKUP.EXE match the no. of disks backed up ? Any unexpected short execution time and/or exit status ?

Volker.

[Hoff]

Please post the DCL. All of it.

Please identify the specific MKA600: device.

I'd probably only use /NOCRC on the backups I didn't ever want to restore, particularly if this is (as I suspect) a DAT/DDS. DLT is rather more reliable than DAT/DDS, and I might consider turning off CRC. (DAT/DDS is very nearly write-once archival media, in my experience.)

Those file system interlocks being ignored are there to prevent silent data corruptions in the output BACKUP saveset or output disk, too.

Ignoring the label processing is fairly benign in some environments, but can also lead to cases where the media is overwritten.

There are cases of write-only tapes (and I've met a few of these over the years), a situation which has lead me to use /VERIFY. If you need seriously faster and are skipping /VERIFY to stay in your window and you don't have an Ultrium-class (fast) tape, you might want to go disk-to-disk-to-tape.

[Mike Smith_33]

I am making progress. I have run this same backup again and gotten the remote system disks savesets to show up. The tapes are sdlt. I have found a tape from before the flood, the disks are not on it but logfiles from that time are there. I will post more once I check a few more things.

[Mike Smith_33]

Thanks for all the replies so far, I will get points in for all once i get this straight.

I was able to recover old logs from 6 weeks or so ago and it shed great light on the issue. Both those disk were getting the following error:

%backup-f-procindex, error processing index file on $3$dka0:, RVN 1
-system-f-volinv, volume is not software enabled


I need to go look this up now. That is the update for now.

[Robert Gezelter]

Mike,

An -F- error, that would likely have terminated the backup. Also check for any definitions of the symbol BACKUP (perhaps /RECORD is on?).

BACKUP/IMAGE should work without the volume being write enabled.

- Bob Gezelter, http://www.rlgsc.com

[Mike Smith_33]

I will check the backup for the /record but I am 98% sure that is not there. What we think happened at this point is that at some point the system disk maybe through a shutdown or something were dismounted but did not leave this node cleanly resulting in a mount verify state on the drive. In fact I lost two hours early this morning when that exact thing happened on one of the nodes. At this point we believe that is what happened.

If anyone thinks this is far fetched, let me know.

[Jon Pinkley]

RE:"%backup-f-procindex, error processing index file on $3$dka0:, RVN 1
-system-f-volinv, volume is not software enabled"

RVN 1 indicates the backup thought that $3$dka0: was part of a bound volume set. For normal volumes the relative volume number (RVN) will be 0. I am quite certain that bound volume sets are not valid as system disks, so is the client sure that these were really system disks?

The message just seems really odd for a backup of a system disk.

Can you post the "sanitized" version of the backup command that was used? (you are evidently protecting the clients identity, which is fine, but please give us enough information help).

Jon

[Volker Halle]

Mike,

system-f-volinv, volume is not software enabled

is a CLEAR indication, that that disk was NOT correctly MOUNTED at the time the BACKUP ran. Most likely, the disk was in MntVerifyTimeout, because the MSCP-Serving node had been down for more then MVTIMEOUT seconds and that disk had not been dismounted and re-mounted after the other node came back.

Volker.

[Jan van den Ende · Accepted solution]

Mike,

>>>
In fact I lost two hours early this morning when that exact thing happened on one of the nodes.
<<<

Sometimes, the drive gets into MOUNT_VERIFY and does not respond to any commands before the timeout expires.

If that was.is the case, next time you CAN speed up things be enring
SYSGEN
USE ACTIVE
SHOW MVTIMEOUT (>>> and remember the CURRENT value
SET MVTIMEOUT
WRITE ACTIVE

In case of MSCP served, probably needed on all involved nodes

Now MOUNT verify times out

SYSGEN reset MVTIMEOUTto remembered value (on all changed nodes)

It is quicker han waiting!

hth

Proost.

Have one on me.

jpe

[Mike Smith_33]

Sorry it took me so long to respond, I was on a plane all day yesterday. I have requested a copy of the backup com procedure so if they send me that I will sanitize and post.

Thanks for the last few posts which have been extremely helpful. I definitely believe it is mount related.

[Jeremy Begg]

Hi Mike,

Just a quick suggestion for you to establish the underlying problem, based on my own experience. If you can modify the backup job, try the following:

- at the start of the job, do a
$ SHOW DEV D
to list all disk devices

- immediately prior to the BACKUP command, do a
$ SHOW DEV/FULL 'device'
where 'device' is the device about to be backed up.

This way you will know exactly the state of your disks when BACKUP is running.

Regards,
Jeremy Begg

[Ian P]

Hi all, Not here ofteen but...

IMHO it is not wise to keep system disks mounted on any systems other than the owners. This is one example of what can occur, another is the inability to mount the system disk at boot time because of an "another volume with the same name" type error, probably only occurs when shadowed.

Suggestion, only mount the remote disks when you actually need them, put some F$GETDVI's in place to verify the status of those disks both before and after mounting and raise alarms if there appears to be a problem. And dismount them when done.

[Jan van den Ende]

@ Ian P,

>>>
another is the inability to mount the system disk at boot time because of an "another volume with the same name" type error,
<<<

Excuse me, but if you EVER encounter such error, it is HIGH TIME to review your config, or more precise, your boot procedures!

__EACH__ device should be, and if it has ever been in the config, it _IS_, uniquely named.
If your procedures use the fully qualified names ie, using $ALLCLASS$ or nodename$diskname syntax (as thy should in a cluster environment), then this error will NOT occur!

This looks more like a mount-verify remote-mounted type of situation.

Mike: maybe you should modify the backup procedure to verify that each disk _IS_ present, available, and mounted at the node running the Backup procedure. Available through simple F$GETDVI functions.

hth

Proost.

Have one on me.

jpe

[Ian P]

Excuuuuse me, Jan, but with all due respect you have forgotten that device names can and do change during the boot process. If your system disk is shadowed, it starts as a DK, DG or whatever and ends up as a DSA device when the shadow device is formed.

So you mount system A's system disk on system B, then system A reboots it attempts to mount EG $1$dka0 and fails because there is a DSA device with the same volume label. A similar scenario can occur if you use SCSI device naming.

Cheers, Ian

[Mike Smith_33]

I finally got an email from the site with the backup script. I have sanitized it of any site information. I am about to close this but I remember a couple of requests for the backup script.