Has backup/image/ignore=interlock become useless?

[Jon Pinkley]

Correction in terminology:

I said "In other words, backup copies blocks, not records, and uses low level access (the ACP interface) to read (non-saveset) files instead of RMS."

That should be: "In other words, backup copies blocks, not records, and uses low level access (the ACP interface IO$M_ACCESS) to open (non-saveset) files instead of RMS file opens."

And while I was entering this, I see Volker has provided the answer to why only local writers are detected (and why a file open/close for update on anther node is detected, as it (under most conditions) changes the revision date.

Jon

[Volker Halle]

Bill,

if you've really used /VERIFY on the BACKUP command (as shown in your example, which probably is NOT from the batch .LOG file itself) and you have not received VERIFYERR errors, then consider, that the verification process seems to compare the files from the saveset against the files from the source disk having been saved and not vice-versa.

Volker.

[Jon Pinkley]

Expanding on what Volker said about verification:

Verification was probably originally meant to provide a way to know that there were no uncorrectable errors introduced during the copy process, and to allow a check of the complete end-to-end data path between the source disk and saveset. Remember that BACKUP was written before tape drives with ECC were available. In my opinion, doing a backup/listing of the saveset provides a reasonably good verification that the saveset is readable, although it isn't a confirmation that it the data is identical to what is on the disk. But the only way for it to be identical, is if the disk hasn't changed.

Verification is really only useful on a static disk. Otherwise you will get verification errors on any file which has changed between the time the file was written to the saveset, and when it is read from the saveset in the verification pass. The verification pass starts after the whole saveset has been written, and in the case of a saveset on tape, after the tape has been repositioned to the start of the saveset. This repositioning isn't optimized (at least that is my experience with 7.3-2); it rewinds the tape, then checks each saveset until it finds the correct one. In other words, if there are many savesets on the tape, it takes progressively longer to reposition the tape as the saveset number on tape gets higher. This shouldn't be necessary with modern tape drives that can report position, and skip to a position on the tape, but there is no TMSCP command or even $QIO function to do that. It can be done with a $QIO DIAGNOSE SCSI pass-through, but I am not expecting that this functionality will ever be added to BACKUP.

Summary: /VERIFY is expensive from a time point of view, and unless the source disk is static, it doesn't provide much useful information.

Note that the same "verification errors" would happen if you used convert/share before your backup, a second convert/share after backup, and a differences on the two copies. If the files had changed during the interval between the two copies, differences will report that the files are different. This doesn't mean the first copy was "bad", only that the file is no longer the same as it was at a previous point in time. I.e. if you do get verification errors, that is not necessarily an indication that what is on the tape is inconsistent, only that it is different than what is on the disk at the time the verification pass re-read the disk.

Jon

[AEFAEF]

I find /VERIFY more useful than that. I always use it when making tapes. If nothing else, it tells you that there is something readable on the tape!

Re ECC: I now use /GROUP=0 because of that, but I still use /VERIFY for all save-to-tape operations.

Re BACKUP/LIST: Someone once posted on cov how he trusts BACKUP/LIST for determining save-set integrity. Excuse me a minute while I try to find it . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Working . . . . . . . . . . . . . OK, I'm back. (Yeah, I know -- silly me.)

I couldn't find the original by Jerry Leichter, but here is the url for my quote of it. In it, he deliberately corrupts a save set by changing one byte in it and shows how BACKUP/LIST picks it up. (If you prefer, just search comp.os.vms for RECORD BOUNDARY BACKUP RUTGERS and read my only post in the thread, or maybe you'll find the original! Google groups appears not to have it. The title of the thread is "LD062 Install Question".)

http://groups.google.com/group/comp.os.vms/browse_frm/thread/7ae42920ab4a3a19/cd52ae44c694abe7?lnk=gst&q=record+boundary+backup+rutgers#cd52ae44c694abe7

And sure, there'll be some differences causing verification errors, but you can see if they look reasonable. No, that doesn't guarantee that the save operation went flawlessly, but if will reveal any major problems. It's a quick (okay, not so quick!) spot check.

And I've had situations where the verify pass resulted in a (fatal!) parity error part way through the save set. I'd call that useful information!

And if the CRC values are checked during the verification pass, and no bad values are found -- that's also a good sign. (I would assume that CRC values are checked, but I don't know for sure. Anyone?)

I've had enough problems with tapes that I always use /VERIFY. In fact, when it comes to causing trouble, printers are just hopeless tape-drive wannabees.

Summary: I always use /VERIFY when saving data to tape.

AEFAEF

[comarow]

I'm unclear. Are you trying to create a disaster recovery tape?

Then arrange some down time and boot
stand alone backup, (still supported on
5.5-2) and you'll get a clean, supported
backup.

You can then backup/image/VERIFY for your own
security.

[Wim Van den Wyngaert]

I did a test with a disk under usage. Files wre being created in reverse (1000, 999, ....).

The directory was backuped with more entries than when the backup started. The entries themselves were not backuped. Thus anal/dis says "invalid file identification in directory".

Anal/dis also found that some file that are/were present were not in the directory ("not found in dir"). This because they moved to another (new) block).

So, make sure to do anal/dis after the restore.

fwiw

Wim

[AEFAEF]

Wim,

Somewhere in the docs or perhaps ECO release notes there is exactly that advice: Run ANAL/DISK after a BACKUP restore operation.

AEF

[Richard W Hunt]

Too many years ago I worked on some utilities since obsolete because DEC (yes, that long ago) and some 3rd-party vendors eventually provided what I needed.

I learned that when you bypass the file system and "roll your own" QIO ops, you also don't track the contents of the DIRID cache or HDRCACHE. ANAL/DISK is capable of doing this, and the file system knows what is in the cache, too. However, if something changes in a directory, it changes in the cache first. THEN it gets written back to the disk based on write-behind settings. But with an unpredictable and untrackable delay.

As to a comment I read in passing, EMC always tells you to quiesce anything you are trying to replicate. They even have white papers on the subject. (Don't have the URL for you, haven't used my PowerLink account in ages.) But the comments about using your apps to perform backup is spot-on. We had a devil of a time getting ORACLE backups that meant anything at all until we started used RMAN and forced log-file switches afterwards. Then we could use the RMAN file and the stored log files to recover our files elsewhere.

The idea of doing a shadow-split also doesn't work if you have controller-based shadowing, which is often what you have for HSG or EMC fiber-attached storage. Bottom line, it is nearly impossible to quiesce a disk enough to make a really good backup.

I believe this can be traced back to the lack of a file-system primitive that could be issued against a device, perhaps even as a diagnostic-class command requiring beaucoup privilege, to flush all pending QIO write-behind buffers for a given disk. Granted, probably almost impossible to do, and in a clustered environment, even closer to impossible than in a stand-alone case.

My approach, which clearly would not work for the system disk, was to dismount and remount the volume I was about to replicate, and then live with the fact that I was taking a point-in-time snapshot. But at least I knew WHEN the point in time had occurred.

[Ian Miller.]

According to Kirby then the caches can be flushed by taking out a write lock on the appropriate F11B$c lock. This could be done by opening BITMAP.SYS and INDEXF.SYS for write.

Perhaps pausing the writes then SET VOLUME/REBUILD/FORCE might do it.

[comarow]

I went back and reviewed this. It seems you
have a shadow set for your system.
Backing up any in use drive can lead to problems. If a directory is altered, you can lose files in the directory.

But you have the PERFECT Solution.

Dismount one of the shadow disks. Mount it privately and create the backup save set.
It will be supported, works great, and will
give you great peace of mind.

If you are afraid of losing your one and only disk, you can shadow up an extra disk and use that. In any case, you still have the dismounted copy of your system disk.

What is the problem with a supported, simple
solution?

Bob Comarow

[Ian Miller.]

The supported process for backups using shadowed disks is described in the fine documentation

http://h71000.www7.hp.com/doc/732final/aa-pvxmj-te/00/00/70-con.html