topic Re: Backup/(no)crc/group in Operating System - OpenVMS

Backup/(no)crc/group

Guillou_2 — Tue, 14 Sep 2004 02:46:13 GMT

Hi,

I use a tz86 for my backup. The save is done with /crc and with /group=0.

I would like to have some precision about these parameters and in particular when /group takes the null value.

Thanks in advance for your help

regards

Steph

Re: Backup/(no)crc/group

Uwe Zessin — Tue, 14 Sep 2004 03:19:34 GMT

Hello Steph,

/CRC is an end-to-end check that verifies the whole datapath and the consistency of the save-set (when used with /VERIFY or during a restore).

/GROUP is used for redundancy when a tape block cannot be read[*]. It works with an XOR mechanism similar to RAID-5.

[*] The problem is that many modern tape drives use error and correction mechanism on their own, which is good, but if the tape drive cannot correct a media error it appears to have lost synchronisation and will not let the user get beyond this bad spot.

So any redundancy created by OpenVMS BACKUP cannot be used, because the tape drive will not give it more data to recover.

Re: Backup/(no)crc/group

faris_3 — Tue, 14 Sep 2004 03:56:46 GMT

Integrity Features Of The VMS BACKUP Utility

Copyright (c) Digital Equipment Corporation 1991. All rights reserved

COMPONENT: BACKUP Utility OP/SYS: VMS

SOURCE: Digital Customer Support Center

VERSION INFORMATION:

Information Applies To: VMS, All Versions

OVERVIEW:

The BACKUP Utility is used to save and/or copy data. When saving
files, BACKUP is designed to maintain the data so it can be restored
and used reliably.

This article describes the features of BACKUP that allow it to write
data to a save set and recover that data for later use.

Subjects covered in this article are:

o The BACKUP qualifiers /CRC, /GROUP_SIZE and /IGNORE=INTERLOCK

o Media errors

o Files marked NOBACKUP

/CRC QUALIFIER:

A CRC (Cyclic Redundancy Check) is a mathematical calculation
done bit by bit on data transferred from one location to
another. Some modern tape drives automatically generate a CRC
(Cyclic Redundancy Check) when writing data. This provides a
more robust error detection capability than older model drives.

Data copied to tape or disk must pass through various
controllers and busses and potentially back again. The CRC
generated by a tape or disk drive only checks the data after the
media has received it.

The CRC generated by BACKUP allows it to detect a failure when
any of the components that the data passes through fails to
transmit it correctly. Without the CRC generated by BACKUP,
there is no way to detect that the data was written or read
incorrectly. Any data corrupted or lost due to a failure along
the transmission path cannot be recovered. Therefore, the use of
the /NOCRC qualifier is not recommended.

/GROUP_SIZE QUALIFIER:

When BACKUP writes to a saveset, it organizes the data in groups
of blocks. The size of a block of data can be determined with
the /BLOCK_SIZE qualifier. The number of blocks in a group can
be determined with the /GROUP_SIZE qualifier.

The BACKUP Utility maintains a certain amount of redundant user
data within a group of blocks. If a file has to be restored,
this redundant data can be used in place of a block within the
group that is unreadable.

If the default group size of 10 is used, then a recovery block is
written for every 10 blocks of data stored in the saveset. If
one block within that 10 is unreadable because of media or other
errors, it can be recovered from the redundant data.

If a group size of 100 (BACKUP's maximum value) is specified,
only errors in one block of that 100 can be recovered. If two or
more blocks within a group of 100 contain errors, data will be
lost and cannot recovered.

A group size of zero means that no redundancy information is
maintained in the saveset. Therefore, no replacement data is
available for unreadable blocks.

/IGNORE=INTERLOCK QUALIFIER:

This qualifier allows BACKUP to save a file that is currently
opened for write or exclusive access. This can increase
application uptime while still allowing data to be saved.
However, /IGNORE=INTERLOCK should be used with caution.

When using /IGNORE=INTERLOCK, BACKUP only saves the data that
actually exists in a file on disk at the time of processing.
Data that is in a user or system buffer in memory will not be
saved. Data that is being modified while BACKUP is reading the
file may not be saved in a consistent state.

For example, an RMS file is opened for write by several
processes, each with their own local buffers. Modified data is
stored in these buffers until they are flushed and the changes
are written to the file on disk. These changes may not have been
written to the file at the time BACKUP saves it. It is also
possible that only a partial update of the file on disk has
occurred.

If RMS index information about the location of the data within
the file has not been written to disk, the data that is saved
will not be accessible after a restore because there is no
pointer to it.

Since user data and RMS index information is written separately,
the index structure could point to nonexistent records after a
restore.

MEDIA ERRORS:

Since the BACKUP Utility is designed to save as much data as
possible, when media problems occur (such as PARITY and
FORCEDERROR errors), BACKUP will continue to write data to its
output destination. When restoring files that encountered media
errors while being saved, there is no warning message that a
particular file may have questionable data. Therefore, the
various output that logs BACKUP save operations (Batch Log file,
Backup Log file, etc.) should be examined. Any file that
encountered media errors should be noted.

FILES MARKED NOBACKUP:

The BACKUP Utility can save the attributes of a file, including
its storage allocation, without actually saving the contents of
the file. The attributes of large files such as the system PAGE
and SWAP files can be saved without wasting space in an output
saveset. When a file is marked NOBACKUP, only the file header
(which contains the file attributes), is written by BACKUP to the
output saveset. None of the contents of the file are saved.

A file marked NOBACKUP appears to be in the correct form after
being restored. None of the data previously existing within the
file is available. The only files that should be marked NOBACKUP
are ones where the data is not needed.

For volumes that contain essential user data, it may be advisable
to add the NOBACKUP option to the /IGNORE qualifier
(/IGNORE=(NOBACKUP)). This will prevent the accidental loss of
data if a file was inadvertently marked NOBACKUP. The BACKUP
Utility issues an informational message and lists the files it
encounters with the NOBACKUP attribute. By referencing this
list, the DCL command "SET FILE/BACKUP" can be used to restore
the BACKUP attribute for future BACKUP operations.

NOTE: When using more than one option for the /IGNORE qualifier,
enclose all options within the parentheses, separated by
commas. If the same qualifier is used more than once in
a command line, only the last iteration is used. In the
following example, only the NOBACKUP option will be used:

BACKUP/IGNORE=INTERLOCK/LOG/IGNORE=NOBACKUP input output

Re: Backup/(no)crc/group

Wim Van den Wyngaert — Tue, 14 Sep 2004 04:36:00 GMT

Check http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=658938

As I understand it you are correct because the tape drive is doing raid with the tape.

Wim

Re: Backup/(no)crc/group

Uwe Zessin — Tue, 14 Sep 2004 05:49:15 GMT

ECC is not the same as XOR block recovery (e.g. RAID-5).

Re: Backup/(no)crc/group

Wim Van den Wyngaert — Tue, 14 Sep 2004 15:21:59 GMT

Uwe,

What is ECC in your opinion ?
It "seemed" to be the same as the /group t.i. some kind of raid 5.

Wim

Re: Backup/(no)crc/group

Guenther Froehlin — Tue, 14 Sep 2004 15:59:05 GMT

With /GROUP=n BACKUP calculates an extra redundancy block for every n save set blocks. This is done by an XOR function basically an add operation:

For example with /GROUP=3 the XOR block based on a byte operation looks like this:

B1 B2 B3 X
1. 12 80 43 -> = (12 + 80 + 43 ) mod 256 = 135
2. 93 85 87 -> = 265 mod 256 = 9

Let's assume block B2 fails to read correctly the reconstruction looks like this:

B2 1. = 135 - 12 - 43 = 80
B2 2. = 9 - 93 - 87 = -171 + 256 = 85

This should give you the idea. Now the XOR is done for all bytes of all buffers in an XOR group. That takes some CPU time. Plus it takes an extra save set block. With the default of /GROUP=10 you get a 10% overhead in save set blocks.

Given my experience with tape drive and tape media technology over the last couple of years this BACKUP feature became obsolete. My experience is: If BACKUP cannot read a certain block from a tape the rest of the tape is unreadable as well. Therefore I always use /GROUP=0.

The CRC is a bit different. First, the 128-bytes block header is always protected with a CRC and the 4-bytes CRC is stored within the block header. Within the block header is another 4-bytes CRC for the remainder of the save set block. With /CRC BACKUP calculates this CRC value and stores it in the reserved space in the block header. So using /NOCRC does not save any space, just some CPU time (bad on old VAX systems but not an issue for Alphas).

Now the good thing with the /CRC is it puts a "seal" on the save set block before it leaves BACKUP's control in memory. And BACKUP checks that "seal" when the data is back in memory. Any other piece (hardware, firmware, software) messing around with a save set block can now be detected by BACKUP. In my eyes: Worth every nanosecond!

/Guenther

Re: Backup/(no)crc/group

Antoniov. — Wed, 15 Sep 2004 12:26:03 GMT

My experience (actually use only DAT tapes) is like Guenter: if device can't read a record, cannot read furthermore the tape.

Antonio Vigliotti

Re: Backup/(no)crc/group

Wim Van den Wyngaert — Thu, 23 Sep 2004 08:35:25 GMT

I used /group=0 instead of the default of 10.
Result : backup takes 70 minutes instead of 80 on 1 GS160 and 120 minutes instead of 145 on another GS160 (both DLT8000).

So almost 20% faster. Also cpu usage is about the same % lower.

Wim

Re: Backup/(no)crc/group

Wim Van den Wyngaert — Fri, 24 Sep 2004 01:40:55 GMT

May be good to know : /group=0 for save sets on disk saves about 10% in size. So, very usable for transferring applications (faster).

Wim