topic Using Backup/Encrypt - Saveset Corruption with no warning. in Operating System - OpenVMS

Using Backup/Encrypt - Saveset Corruption with no warning.

The Brit — Thu, 17 Jun 2010 12:47:37 GMT

About 3 weeks ago we implimented encryption on our daily backups using the native OpenVMS Encryption option (OpenVMS V8.3-1H1).

The backup job in question run nightly after quiesing our app (so all files are closed). The job creates 8 savesets. At the time that the savesets are created, each has a "check file" included, this is used for verification. When the Backup is finished, the Verification job is started, and it opens each saveset and extracts the check file. This is basically to verify that the saveset is readable.

The backup and verify jobs have been running fine, and unchanged, since we implimented the encryption, that is until Tuesday night.

On Tuesday night, the backup job ran fine, gave no errors either in the code, or on the tapedrives. When the subsequent verify job executed, it ran fine for the first 4 savesets (extracting the check file as it should). On the 5th saveset it returned the following error,

%BACKUP-F-ENCBSRNOT, internal error. Backup Summary Record required for decryption not found

The job then blew off. After manually checking, savesets 6, 7 & 8 were fine, so the problem seemed to be only with saveset 5.

Note: The scripts had run fine up to this time, and ran fine last night.

The help for this message says;

ENCBSRNOT, internal error. BACKUP Summary, Record required for
decryption not found

Facility: BACKUP, BACKUP/ENCRYPT Command

Explanation: The BACKUP Summary Record that contains the data key under
which the save set is encrypted was not found at the start
of the save set. It is possible that the save set was not
encrypted.

User Action: None. The save set is probably corrupted and cannot be
restored. If the save set was in fact not encrypted, it can
be recovered by omitting the /ENCRYPT qualifier.

When I checked to see if the saveset was in fact encrypted, backup told me that it was, and that I had to use the /Encrypt qualifier.

I tried accessing the saveset manually, i.e. from an interactive session, using Backup/list and the full backup restore command, both with the /encrypt=... qualifier include and excluded. With it included I got the same error as above. With it excluded I got the appropriate "This is an encrypted saveset.." message.

I am concerned that there was no indication in the original Backup Script that this saveset was both corrupted and unrecoverable. If I had not had my verification job in place I would have had no indication that the saveset was bad, and would have happily sent the tape offsite thinking it was a valid backup.

I have attached both the backup command, and the verification command. However keep in mind that these scripts are not inherently bad, having been use consistently and successfully since encryption was implemented, including last night.

I have reported this event to HP but I was interested if anyone had seen this before and could shed light on possible causes.

thanks \

Dave.

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Craig A — Thu, 17 Jun 2010 14:44:04 GMT

Hi Dave

Are there any good reasons why you are using /NOCRC?

Also, why are you you not using /SAVE after ND06.BCK

Personally, I'd be tempted to run a full suite of tests with /CRC/SAVE and possibly with /VERIFY to see what happens.

HTH

Craig

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

The Brit — Thu, 17 Jun 2010 15:35:51 GMT

Hi Craig,

/NOCRC ?? No good reason for or against. It was on the command when I took over administration of these scripts. Not really relevant however since it has always been there, does not appear to influence (that I can see) the encryption problem that I observed.

/Save ?? This is unnecessary since it is the default when writing to tape!

Dave

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Thu, 17 Jun 2010 15:47:04 GMT

Hi Dave,

This was a bug in BACKUP utility. This was reported to engineering and has been fixed. This problem occurs while restoring the encrypted saveset, even though the encrypted savesetâ s contents are intact. BACKUP saveset stores the encryption key (which is encrypted) in its BSR (BACKUP summary record). During restore, BACKUP compares this value with the value specified in the BACKUP command. If this comparison fails, then BACKUP reports the %BACKUP-F-ENCBSRNOT. BACKUP utility had a bug which causes BACKUP report the above error in some conditions based on the HEX value of the key. This has been identified and fixed.

Please provide the image identification information of BACKUP.EXE and BACKUPSHR.EXE.

$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) SYS$COMMON:[SYSEXE]BACKUP.EXE;
$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) SYS$COMMON:[SYSLIB]BACKUPSHR.EXE;

Let me check, are the BACKUP images in your machine older and does not contain the fix.

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Thu, 17 Jun 2010 16:01:11 GMT

Hi Dave,

There is a fix available for this issue in V8.3 both in Alpha and IA64. The fix part of VMS83I_BACKUP-V0300 and VMS83A_BACKUP-V0400 kits.

=================
5.2.2 BACKUP Fails While Listing Or Restoring The Encrypted SaveSet

5.2.2.1 Problem Description:

BACKUP fails with below error message while listing or
restoring the encrypted saveset.

%BACKUP-F-ENCBSRNOT, internal error. Backup Summary Record required for decryption not found
=================

You can verify the fix by restoring the saveset(which reports error on V8.3-1h1) on V8.3 Alpha or IA64 after installing the above kits. Please let me know the image idtification information of BACKUP images in V8.3-1h1 and I will check whether they have the this fix in them.

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

The Brit — Thu, 17 Jun 2010 16:08:56 GMT

Hi Ketan,

I have attached the image information you requested.

Regarding your comments, I should say that my systems are patched pretty much up to date (at least up to ~4 weeks ago), so unless this "fix" came out recently, I would expect it to be in. Alternatively there is a condition (which I found) which is not covered by the fix.

Dave.

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Fri, 18 Jun 2010 02:24:44 GMT

Hi Dave,

Did you get a chance to verify the fix on V8.3 Alpha or IA64 machine? Does the BACKUP able to restore or list the saveset (which reports error on V8.3-1h1) on V8.3 after installing the above mentioned kit?

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

P Muralidhar Kini — Fri, 18 Jun 2010 03:30:43 GMT

Hi Dave,

>> There is a fix available for this issue in V8.3 both in Alpha and IA64.
Looks like the problem that you have reported is a known problem which is
fixed. The patch with fix is released for openVMS V83 Alpha/I64.

>> (OpenVMS V8.3-1H1).
The system that you are using is OpenVMS V8.3-1H1. If the problem is same
then all you need may be a Backup patch for OpenVMS V8.3-1H1.

>> You can verify the fix by restoring the saveset(which reports error on
>> V8.3-1h1) on V8.3 Alpha or IA64 after installing the above kits.
If you have other nodes in the cluster then you can try out the tests suggested
by ketan. This should help you to confirm that the problem that you are facing
is indeed the known problem.

>> I have reported this event to HP
Good. So if the problem is the same one, then its a matter of you getting a new
backup image for OpenVMS V8.3-1H1.

Regards,
Murali

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

P Muralidhar Kini — Fri, 18 Jun 2010 03:34:24 GMT

Hi Dave,

>> If you have other nodes in the cluster
Oops. To make it clear, i meant any other openVMS V83 node in the cluster.

Regards,
Murali

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Fri, 18 Jun 2010 03:41:43 GMT

Hi Dave,

The fix for this issue on V8.3-1h1 is part of below images.

$
$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) BACKUP.EXE;1
BACKUP.EXE;1
OpenVMS IA64
Image
"BACKUP"
"V8.3-1H1"
"XBOR-0090070000"
30-NOV-2007 14:24:47.18

$ ANALYZE/IMAGE/SELECT=(ARCH,FILE,NAME,IDENT,BUILD,LINK) BACKUPSHR.EXE;1
BACKUPSHR.EXE;1
OpenVMS IA64
Image
"BACKUPSHR"
"V8.3-1H1"
"XBOR-0090070000"
30-NOV-2007 14:23:56.75
$
$

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

P Muralidhar Kini — Fri, 18 Jun 2010 04:07:26 GMT

Hi Dave,

Images in your V83-1H1 system -
>> Image name: BACKUP
>> Link Date/Time: 30-AUG-2007 10:53:53.65

>> Image name: BACKUPSHR
>> Link Date/Time: 30-AUG-2007 10:53:06.56

V83-1H1 Images with fix -
>> BACKUP.EXE;1
>> 30-NOV-2007 14:24:47.18

>> BACKUPSHR.EXE;1
>> 30-NOV-2007 14:23:56.75

Looks like you dont have the fix image installed in your system.
You need to install the above mentioned BACKUP images and that should solve the problem.

Regards,
Murali

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Fri, 18 Jun 2010 04:11:06 GMT

Hi Dave,

Please install the VMS831H1I_BACKUP-V0100 kit. This kit contains the latest BACKUP image of 18-SEP-2009.

=============================
SYSEXE]BACKUP.EXE (new image)

Image Identification Information

Image name: "BACKUP"
Image file identification: "V8.3-1H1"
Image build identification: "0090090011"
linker identification: "Linker T02-28"
Link Date/Time: 18-SEP-2009 17:52:21.95
Overall Image Checksum: 53FA54FC

[SYSLIB]BACKUPSHR.EXE (new image)

Image Identification Information

Image name: "BACKUPSHR"
Image file identification: "V8.3-1H1"
Image build identification: "0090090011"
linker identification: "Linker T02-28"
Link Date/Time: 18-SEP-2009 17:50:06.60
Overall Image Checksum: D638B667
=============================

This kit contains the fix but the fix is not documented in the ECO Cover Letter. Please install the kit and let me know if it resolves the problem.

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Fri, 18 Jun 2010 04:25:30 GMT

Hi Dave,

Cause for this issue.

During the save operation, BACKUP copies the datakey which is encrypted in BSR(BACKUP summary record) for the encrypted savesets. While listing or restoring the encrypted savesets, BACKUP first searches for datakey stored in BSR. When it finds this datakey, it compares this value with -1 (.i.e. Hexadecimal FF) to check whether it had found a key. There are chances that this datakey may contain the value "FF" which was generated during encryption, resulting in BACKUP reporting ENCBSRNOT error message. This does not cause any data loss or corruption. Even though the BACKUP fails during restore, data in the saveset is intact. Nothing to worry about the data.

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Craig A — Fri, 18 Jun 2010 10:24:40 GMT

Hi Dave

I didn't know /SAVE_SET was the default for tape drives and I've been working with VMS since 1987. Yikes! You learn somethign new every day.

I always add use it in the BACKUP command when writing to tapes - for completeness.

Ketan said:

>This does not cause any data loss or >corruption. Even though the BACKUP fails >during restore, data in the saveset is >intact. Nothing to worry about the data.

Was this your experience? i.e. Did you manage to restore data from the saveset?

Craig

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Fri, 18 Jun 2010 10:31:41 GMT

Hi Dave,

I have verified the source code listing of the BACKUP images which were built on 18-SEP-2009 (which are part of VMS831H1I_BACKUP-V0100 kit). The source code listing contains the modification which fixes the issue. You will be all set after installing VMS831H1I_BACKUP-V0100 kit.

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Fri, 18 Jun 2010 10:35:04 GMT

Hi,

>> Was this your experience? i.e. Did you manage to restore data from the saveset?

My update was with respect to this particular issue which is discussed in this thread. Yes I have seen other customers facing the same issue and resolved it after using this fix.

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Hoff — Fri, 18 Jun 2010 12:17:14 GMT

Personally, I always specify /SAVE_SET on the BACKUP parameter that I think contains the saveset, and regardless of what else is going on, as I'd rather not have a surprise.

Backup is wonky enough, and if I mess up with a BACKUP or a MOUNT command (or for those occasional cases where BACKUP goes into the weeds), having /SAVE_SET on the saveset is an extra measure of don't-gronk-my-disks.

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Shriniketan Bhagwat — Fri, 18 Jun 2010 12:40:53 GMT

Hi,

/SAVE_SET qualifier is not mandatory for tape operations. It could be save operation or listing from the tape or restore from tape. It is good /no harm in using it. I personally use this qualifier every time where the saveset is involved.

Regards,
Ketan

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

Robert Gezelter — Fri, 18 Jun 2010 12:48:50 GMT

Dave,

I agree with Hoff on the use of /SAVE_SET. It is easy to use the default when explicitly referencing a tape drive. It is more dangerous to do when using a logical name or symbol substitution.

I have seen mishaps when people salvaged code without sufficient care and then walked into problems. For that reason, I always recommend explicitly using the qualifier.

On /CRC, it is admittedly not as useful as it was on open-reel tape, but overall, I tend to use /CRC and the redundancy groups. I have seen cases where a saveset was corrupted as it was copied across a network, and the /CRC and redundancy group identified and corrected the problem.

I originally encountered this danger shortly after BACKUP was released, when I was copying files between a VAX-11/780 and a PDP-11/34 over a DMC-11-based DECnet link. The DMC-11 had a timing problem that would occasionally drop bytes caused by a mishandling of Bus Grant Late on the VAX-11/780 UNIBUS. Getting the "unrecoverable error, data recovered" message on a network file access was an unexpected experience.

- Bob Gezelter, http://www.rlgsc.com

Re: Using Backup/Encrypt - Saveset Corruption with no warning.

P Muralidhar Kini — Fri, 18 Jun 2010 13:04:14 GMT

Hi,

>> I didn't know /SAVE_SET was the default for tape drives
Yes, /SAVE_SET is the default for the tape drives.
But then i always prefer to use /SAVE_SET when dealing with savesets.

Regards,
Murali