Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Using Backup/Encrypt - Saveset Corruption with no warning.

 
SOLVED
Go to solution
The Brit
Honored Contributor

Using Backup/Encrypt - Saveset Corruption with no warning.

About 3 weeks ago we implimented encryption on our daily backups using the native OpenVMS Encryption option (OpenVMS V8.3-1H1).

The backup job in question run nightly after quiesing our app (so all files are closed). The job creates 8 savesets. At the time that the savesets are created, each has a "check file" included, this is used for verification. When the Backup is finished, the Verification job is started, and it opens each saveset and extracts the check file. This is basically to verify that the saveset is readable.

The backup and verify jobs have been running fine, and unchanged, since we implimented the encryption, that is until Tuesday night.

On Tuesday night, the backup job ran fine, gave no errors either in the code, or on the tapedrives. When the subsequent verify job executed, it ran fine for the first 4 savesets (extracting the check file as it should). On the 5th saveset it returned the following error,

%BACKUP-F-ENCBSRNOT, internal error. Backup Summary Record required for decryption not found

The job then blew off. After manually checking, savesets 6, 7 & 8 were fine, so the problem seemed to be only with saveset 5.

Note: The scripts had run fine up to this time, and ran fine last night.

The help for this message says;

ENCBSRNOT, internal error. BACKUP Summary, Record required for
decryption not found

Facility: BACKUP, BACKUP/ENCRYPT Command

Explanation: The BACKUP Summary Record that contains the data key under
which the save set is encrypted was not found at the start
of the save set. It is possible that the save set was not
encrypted.

User Action: None. The save set is probably corrupted and cannot be
restored. If the save set was in fact not encrypted, it can
be recovered by omitting the /ENCRYPT qualifier.

When I checked to see if the saveset was in fact encrypted, backup told me that it was, and that I had to use the /Encrypt qualifier.

I tried accessing the saveset manually, i.e. from an interactive session, using Backup/list and the full backup restore command, both with the /encrypt=... qualifier include and excluded. With it included I got the same error as above. With it excluded I got the appropriate "This is an encrypted saveset.." message.

I am concerned that there was no indication in the original Backup Script that this saveset was both corrupted and unrecoverable. If I had not had my verification job in place I would have had no indication that the saveset was bad, and would have happily sent the tape offsite thinking it was a valid backup.

I have attached both the backup command, and the verification command. However keep in mind that these scripts are not inherently bad, having been use consistently and successfully since encryption was implemented, including last night.

I have reported this event to HP but I was interested if anyone had seen this before and could shed light on possible causes.

thanks \

Dave.
22 REPLIES 22
Craig A
Valued Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave

Are there any good reasons why you are using /NOCRC?

Also, why are you you not using /SAVE after ND06.BCK

Personally, I'd be tempted to run a full suite of tests with /CRC/SAVE and possibly with /VERIFY to see what happens.

HTH

Craig
The Brit
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Craig,

/NOCRC ?? No good reason for or against. It was on the command when I took over administration of these scripts. Not really relevant however since it has always been there, does not appear to influence (that I can see) the encryption problem that I observed.

/Save ?? This is unnecessary since it is the default when writing to tape!

Dave
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

This was a bug in BACKUP utility. This was reported to engineering and has been fixed. This problem occurs while restoring the encrypted saveset, even though the encrypted savesetâ s contents are intact. BACKUP saveset stores the encryption key (which is encrypted) in its BSR (BACKUP summary record). During restore, BACKUP compares this value with the value specified in the BACKUP command. If this comparison fails, then BACKUP reports the %BACKUP-F-ENCBSRNOT. BACKUP utility had a bug which causes BACKUP report the above error in some conditions based on the HEX value of the key. This has been identified and fixed.

Please provide the image identification information of BACKUP.EXE and BACKUPSHR.EXE.

$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) SYS$COMMON:[SYSEXE]BACKUP.EXE;
$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) SYS$COMMON:[SYSLIB]BACKUPSHR.EXE;

Let me check, are the BACKUP images in your machine older and does not contain the fix.

Regards,
Ketan
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

There is a fix available for this issue in V8.3 both in Alpha and IA64. The fix part of VMS83I_BACKUP-V0300 and VMS83A_BACKUP-V0400 kits.

=================
5.2.2 BACKUP Fails While Listing Or Restoring The Encrypted SaveSet

5.2.2.1 Problem Description:

BACKUP fails with below error message while listing or
restoring the encrypted saveset.

%BACKUP-F-ENCBSRNOT, internal error. Backup Summary Record required for decryption not found
=================

You can verify the fix by restoring the saveset(which reports error on V8.3-1h1) on V8.3 Alpha or IA64 after installing the above kits. Please let me know the image idtification information of BACKUP images in V8.3-1h1 and I will check whether they have the this fix in them.


Regards,
Ketan
The Brit
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Ketan,

I have attached the image information you requested.

Regarding your comments, I should say that my systems are patched pretty much up to date (at least up to ~4 weeks ago), so unless this "fix" came out recently, I would expect it to be in. Alternatively there is a condition (which I found) which is not covered by the fix.

Dave.
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

Did you get a chance to verify the fix on V8.3 Alpha or IA64 machine? Does the BACKUP able to restore or list the saveset (which reports error on V8.3-1h1) on V8.3 after installing the above mentioned kit?

Regards,
Ketan
P Muralidhar Kini
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

>> There is a fix available for this issue in V8.3 both in Alpha and IA64.
Looks like the problem that you have reported is a known problem which is
fixed. The patch with fix is released for openVMS V83 Alpha/I64.

>> (OpenVMS V8.3-1H1).
The system that you are using is OpenVMS V8.3-1H1. If the problem is same
then all you need may be a Backup patch for OpenVMS V8.3-1H1.

>> You can verify the fix by restoring the saveset(which reports error on
>> V8.3-1h1) on V8.3 Alpha or IA64 after installing the above kits.
If you have other nodes in the cluster then you can try out the tests suggested
by ketan. This should help you to confirm that the problem that you are facing
is indeed the known problem.

>> I have reported this event to HP
Good. So if the problem is the same one, then its a matter of you getting a new
backup image for OpenVMS V8.3-1H1.

Regards,
Murali
Let There Be Rock - AC/DC
P Muralidhar Kini
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

>> If you have other nodes in the cluster
Oops. To make it clear, i meant any other openVMS V83 node in the cluster.

Regards,
Murali
Let There Be Rock - AC/DC
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

The fix for this issue on V8.3-1h1 is part of below images.

$
$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) BACKUP.EXE;1
BACKUP.EXE;1
OpenVMS IA64
Image
"BACKUP"
"V8.3-1H1"
"XBOR-0090070000"
30-NOV-2007 14:24:47.18

$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) BACKUPSHR.EXE;1
BACKUPSHR.EXE;1
OpenVMS IA64
Image
"BACKUPSHR"
"V8.3-1H1"
"XBOR-0090070000"
30-NOV-2007 14:23:56.75
$
$


Regards,
Ketan
P Muralidhar Kini
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

Images in your V83-1H1 system -
>> Image name: BACKUP
>> Link Date/Time: 30-AUG-2007 10:53:53.65

>> Image name: BACKUPSHR
>> Link Date/Time: 30-AUG-2007 10:53:06.56

V83-1H1 Images with fix -
>> BACKUP.EXE;1
>> 30-NOV-2007 14:24:47.18

>> BACKUPSHR.EXE;1
>> 30-NOV-2007 14:23:56.75

Looks like you dont have the fix image installed in your system.
You need to install the above mentioned BACKUP images and that should solve the problem.

Regards,
Murali
Let There Be Rock - AC/DC
Shriniketan Bhagwat
Trusted Contributor
Solution

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

Please install the VMS831H1I_BACKUP-V0100 kit. This kit contains the latest BACKUP image of 18-SEP-2009.

=============================
SYSEXE]BACKUP.EXE (new image)

Image Identification Information

Image name: "BACKUP"
Image file identification: "V8.3-1H1"
Image build identification: "0090090011"
linker identification: "Linker T02-28"
Link Date/Time: 18-SEP-2009 17:52:21.95
Overall Image Checksum: 53FA54FC

[SYSLIB]BACKUPSHR.EXE (new image)

Image Identification Information

Image name: "BACKUPSHR"
Image file identification: "V8.3-1H1"
Image build identification: "0090090011"
linker identification: "Linker T02-28"
Link Date/Time: 18-SEP-2009 17:50:06.60
Overall Image Checksum: D638B667
=============================


This kit contains the fix but the fix is not documented in the ECO Cover Letter. Please install the kit and let me know if it resolves the problem.


Regards,
Ketan
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

Cause for this issue.

During the save operation, BACKUP copies the datakey which is encrypted in BSR(BACKUP summary record) for the encrypted savesets. While listing or restoring the encrypted savesets, BACKUP first searches for datakey stored in BSR. When it finds this datakey, it compares this value with -1 (.i.e. Hexadecimal FF) to check whether it had found a key. There are chances that this datakey may contain the value "FF" which was generated during encryption, resulting in BACKUP reporting ENCBSRNOT error message. This does not cause any data loss or corruption. Even though the BACKUP fails during restore, data in the saveset is intact. Nothing to worry about the data.

Regards,
Ketan
Craig A
Valued Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave

I didn't know /SAVE_SET was the default for tape drives and I've been working with VMS since 1987. Yikes! You learn somethign new every day.

I always add use it in the BACKUP command when writing to tapes - for completeness.

Ketan said:

>This does not cause any data loss or >corruption. Even though the BACKUP fails >during restore, data in the saveset is >intact. Nothing to worry about the data.

Was this your experience? i.e. Did you manage to restore data from the saveset?

Craig
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi Dave,

I have verified the source code listing of the BACKUP images which were built on 18-SEP-2009 (which are part of VMS831H1I_BACKUP-V0100 kit). The source code listing contains the modification which fixes the issue. You will be all set after installing VMS831H1I_BACKUP-V0100 kit.

Regards,
Ketan
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi,

>> Was this your experience? i.e. Did you manage to restore data from the saveset?

My update was with respect to this particular issue which is discussed in this thread. Yes I have seen other customers facing the same issue and resolved it after using this fix.


Regards,
Ketan
Hoff
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Personally, I always specify /SAVE_SET on the BACKUP parameter that I think contains the saveset, and regardless of what else is going on, as I'd rather not have a surprise.

Backup is wonky enough, and if I mess up with a BACKUP or a MOUNT command (or for those occasional cases where BACKUP goes into the weeds), having /SAVE_SET on the saveset is an extra measure of don't-gronk-my-disks.
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi,

/SAVE_SET qualifier is not mandatory for tape operations. It could be save operation or listing from the tape or restore from tape. It is good /no harm in using it. I personally use this qualifier every time where the saveset is involved.

Regards,
Ketan
Robert Gezelter
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Dave,

I agree with Hoff on the use of /SAVE_SET. It is easy to use the default when explicitly referencing a tape drive. It is more dangerous to do when using a logical name or symbol substitution.

I have seen mishaps when people salvaged code without sufficient care and then walked into problems. For that reason, I always recommend explicitly using the qualifier.

On /CRC, it is admittedly not as useful as it was on open-reel tape, but overall, I tend to use /CRC and the redundancy groups. I have seen cases where a saveset was corrupted as it was copied across a network, and the /CRC and redundancy group identified and corrected the problem.

I originally encountered this danger shortly after BACKUP was released, when I was copying files between a VAX-11/780 and a PDP-11/34 over a DMC-11-based DECnet link. The DMC-11 had a timing problem that would occasionally drop bytes caused by a mishandling of Bus Grant Late on the VAX-11/780 UNIBUS. Getting the "unrecoverable error, data recovered" message on a network file access was an unexpected experience.

- Bob Gezelter, http://www.rlgsc.com
P Muralidhar Kini
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi,

>> I didn't know /SAVE_SET was the default for tape drives
Yes, /SAVE_SET is the default for the tape drives.
But then i always prefer to use /SAVE_SET when dealing with savesets.

Regards,
Murali
Let There Be Rock - AC/DC
Shriniketan Bhagwat
Trusted Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hi,

/CRC qualifier instructs BACKUP to calculate and include a cyclic redundancy check (CRC) value for each block of data written to the backup file. RESTORE, upon seeing the presence of this check, re-calculates the CRC value and ensures that the new value matches the one written to the backup file. If the two values do not match, RESTORE reports a CRC error and then attempts to recover the block using standard XOR recovery methods. CRC has a significantly higher probability of detecting errors than the checks normally used on tapes and disks. Additionally, CRC detects any corruption which may occur as the data is transferred to or from the computer's memory. There is significant overhead involved in calculating the CRC. Using the /CRC qualifier may degrade the performance of BACKUP.

It is good to use both /CRC and /GROUP qualifier with BACKUP.

Regards,
Ketan
Hoff
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

LTO is rated at less than 10^17 bits read uncorrectable, and less than 10^27 bits read undetectable, and incorporates redundant writes and read-after-write support for automatically detecting write errors, and implements data redundancy.

The ECC schemes used are (240,234,7) and (64, 54, 11) Reed-Solomon, which are pretty good ECC.

CRCs are implemented to detect any compression errors, as well.

And yes, you're expecting BACKUP to use and to recover from what is usually compressed data, just to keep things truly interesting.

And for comparison purposes, there are empirical studies showing three to six hard errors per terabytes on hard disk drives. The approach used there is RAID. That's a rate of 3 to 6 hard errors found in 10^12 bits of disk.

Somewhat playing the devil's advocate, I'm not entirely convinced that the BACKUP implementations of CRC and XOR are effective (or even needed) with modern storage, and that writing duplicate tapes might not be the better overall approach for the likely failures.

Put another way, there's the ubiquitous belt-and-suspenders approach that is sometimes blindly applied by various sites, but there's also the "are you even going to get anything back from the tape to even try a BACKUP CRC/XOR recovery?" question? Sure, you might catch a bad device or bad tape, but are you even going to get data back from that widget, presuming the read-after-write doesn't catch the error, or the storage media degrades or later becomes damaged.

I've not seen a study in this area.

Put another way, just because we've always used a particular approach or solution doesn't necessarily mean it's still required.
The Brit
Honored Contributor

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Thanks for the discussion guys,

Normally when I apply an update patch, I compare the contents with the master list and manually add any which are not included in the update patch, to my list of patches to apply. I dont know how I missed this patch (BACKUP V0100). But I will take care of it since it doesn't require a reboot.

I was not able to verify the backup (on a different system) because I re-inited the tape to create the replacement backup. (this was less complex at our location that modifying the cataloging system to change the tape barcode).

Thanks again

Dave