Rdb database corruption

[Navipa]

Hi,

our environment; Charon-VAX 4000 model 108 and VMS v6.1, Rdb Version: Rdb/VMS V4.1-0.

One of our Charon VAX (SysB, test server) runs with old Rdb data. To update it, we recently copied (win-level) all the .vdisks except OS disk from production server to SysB, configured (charon config), updated systrartup and define logicals and made it run successfully. Rdb works fine from SQL> level, but our application fails RDSBUGCHK. From the application exception, I understand it creates DMP while it fails deleting a record from table1. 

$ RMU/VERIFY xxxxx

%RMU-W-ABMBITERR, inconsistency between area bit map page 2 and spam page 93741
%RMU-W-PAGPAGRAN, area uniform_01, page 94831
page number out of range
expected: 94831, found: 0
%RMU-W-PAGBADARE, area uniform_01, page 94831
%RMU-W-PAGTADZER, area uniform_01, page 127531
contains zero time stamp

And I queried the same table and re-created the problem (dump) and the query says ...

SQL> select * from table1 where ccno = 'A1111111' ';

%RDB-F-IO_ERROR, input or output error
-RDMS-F-CANTREADDBS, error reading pages 4:53788-53788
-RDMS-F-CHECKSUM, checksum error - computed 570202C8, page contained 53A3588A

AND

SQL> select desc1 from table1 where ccno='A1111111';
%RDB-E-NO_RECORD, access by dbkey failed because dbkey is no longer associated with a record
-RDMS-F-NODBK, 84:90164:2 does not point to a data record

AND

There was another erro about Index on eno key, So I dropped that index1, but as this table1 has storage MAP created init, I could not able to re-create that index1. 

So I FTPed old copy of RMU/BACKUP file, and restored the particular Area uniform_01 successfully using 

RMU/RESTORE/NEW_VERSION/AREA /ONLINE $2$DIA6:[000000]MART.RBF uniform_01

but even after restoring, I get the following SQL error

SQL> select * from table1 limit to 10 rows;
%RDB-F-DB_CORRUPT, database filename is corrupt
-RDMS-F-AREA_INCONSIST, storage area $disk3:[martdb]UNIFORM_AREA_01.RDA;3 is inconsistent

Then I tried to restoring all the entire DB, I get the folowing...

$RMU/RESTORE /NEW_VERSION /LOG $2$DIA6:[000000]MART.RBF
%RMU-I-RESTXT_04, Thread 1 uses devices $2$DIA6:
%RMU-I-RESTXT_00, Restored root file $disk1:[martdb]:[PROD]SOABARHK.RDB;2
%RMU-I-LOGRESSST, restored storage area $disk8:[martdb]DEFAULT_AREA.RDA;2
%RMU-I-LOGRESSST, restored storage area $disk1:[martdb]UNIFORM_AREA_01.RDA;2
%RMU-I-LOGRESSST, restored storage area $disk2:[martdb]UNIFORM_AREA_02.RDA;2
%RMU-I-LOGRESSST, restored storage area $disk3:[martdb]UNIFORM_AREA_03.RDA;3
%RMU-I-LOGRESSST, restored storage area $disk5:[martdb]MIXED_AREA_01.RDA;2
%RMU-I-LOGRESSST, restored storage area $disk6:[martdb]MIXED_AREA_02.RDA;2
%RMU-I-LOGRESSST, restored storage area $disk7:[martdb]MIXED_AREA_03.RDA;2
%RMU-F-BACFILCOR, Backup file is corrupt
%RMU-F-FATALERR, fatal error on RESTORE

Remember I did not use RMU/RESTORE, I just copied the entire sets of vdisks along with Rdb and application inti, But SQL and Application worked for many of the other tables. 

Now the question is that Can I delete the entire database on SysB and restore with another new production backup copy? I can't create duplicate database as I don't have disk space.

This is the backup and restore commands I use...is it ok?

RMU/BACKUP/ONLINE/LOG $MDB RDBBCK1:[mart]MART.RBF

RMU/RESTORE/NEW_VERSION/NOCDD_INTEGRATE/LOG $3$DIA6:[BACKUP]MART.RBF

Any other option such as IMPORT, EXPORT or UNLOAD?.

Thanks

Navipa

[Hoff]

Some key details of the steps involved in the sequence used here are unclear to me, but if the copies were made from the Microsoft Windows level — and particularly if the OpenVMS environment and the database is open and active running — those transfers won't be consistent.    (If the transfers were made while OpenVMS was shut down, then the transfers should be consistent.)

As for this general sequence (and if OpenVMS was shut down), then it's distinctly possible that not copying the OpenVMS system image got you into this, as Rdb itself requires tailoring of the system environment, and it's quite possible that the replacement OpenVMS environment you're testing with is misconfigured or incompatible.  System parameters or Rdb patch levels or other details can be fairly subtle, unfortunately.

Follow the directions for the Rdb upgrade, and only make database backups and perform transfers of the database contents with the RMU tool.   For a database transfer, use RMU to perform a database backup, then transfer the backup to the other (and previously configured) OpenVMS environment, and use RMU to restore the database backup.

Like the copy of the disk images from the Windows level, using OpenVMS and the OpenVMS BACKUP utility and a live copy of an Rdb database can fail for similar reasons — the database is active, so the storage is not necessarily consistent, and the copy may or may not be functional.

This RMU backup and restoration sequence should already be available (somewhere) in your existing and local OpenVMS environment, as it's how a database backup is performed and how that backup is restored — after a database or disk or server failure, for instance.   There are usually site-local DCL command procedures to this end.   If you do not have these backups running at intervals via a batch job or some other site-local job scheduler — or are otherwise unfamiliar with the basics of managing Rdb backups and restorations — then please check the Oracle Rdb documentation for details on performing this sequence. 

It's also possible that there are corruptions in the source database and it's worth verifying that, but I'd start these transfers with proper RMU backups and RMU restores for the data transfers.  Not the vdisk-level shenanigans.    In short, don't take short cuts.   If you've not already done so, please read the associated Rdb docs, and follow the directions, and use the provided tools. 

Typical sequence for these cases: Get an RMU backup, and shut down.  Either get an OpenVMS VAX standalone BACKUP /IMAGE copies of all disks to some other disks, or clone all of the disk images.  All disks.  These backups are your path back out, should the upgrade fail.   Apply the VAXBACK ECO kit for V6.1 — this is a prerequisite for the upgrade.  Then perform the OpenVMS upgrade to V7.3, and to whatever version of Rdb is current for OpenVMS VAX V7.3, and follow the Rdb directions for converting the database format following the upgrade.  The VAXBACK ECO kit is necessary for V6.1, prior to the upgrade.   There may well be an intermediate upgrade necessary for Rdb, but I've not checked for that.   Though likely not strictly necessary, I'd rebuild the Rdb-using applications, just to pick up any changes that might be applicable to those.

[Navipa]

Wow!, Great details!, Thanks Hoff.

I just now understand that our team has copied all the vdisks while production server and RDB is up and running. They did not shutdown the VMS. As I asked earlier,  Can I delete the entire database (corrupted in many ways) on SysB and restore with another new production backup copy? I can't create duplicate database as I don't have disk space. Because I don't see any other issues in the SysB other than RDB

Thanks

Navipa 

[Hoff]

Disk space?  You're on an emulator.   Instantiate the necessary additional disk space.  Create one or more backing files within the emulator or within the Windows environment, and connect those files to the emulator as virtual disks. 

If the underlying Microsoft Windows box does not have sufficient disk space to allow this, then add or replace the existing storage with bigger disks, and use that to add more storage into the emulation.

FWIW, this and various of your previous postings here in aggregate imply that you're getting incomplete or incorrect information and little or no or possibly even bad advice from whomever is providing your server support, beyond what are comparatively basic server management operational errors.  Accordingly, I'd infer that the OpenVMS VAX production environment and the data here may well be at some risk. 

[Navipa · Accepted solution]

Thank Hoff.

No need to worry, that was test server and I have taken production RMU backup and copied (windows level) the backup to the test server and restored and it works perfect.