Rdb SQL IMPORT fails with " %RMS-W-RTB, 8224 byte record too large for user's buffer"

[Navipa]

Hi,
I have Rdb/VMS V4.1-0 on CharonVAX OpenVMS 6.1. Our DB has 4 uniform and 3 mixed areas and spreaded on 4 vdisks of each 4 GB. And we have replaced one of the 4GB vdisk with 8GB and it application and DB runs fine more than year without any issue. As Stromasys said that version might support vdisk of size larger than 8GB, I have created a vdisk of size larger than 8GB such as 8.5, 9, 10, 12 & 16GB.

Backup and EXPORT work fine with any size of vdisks, including the size larger than 8GB. But the IMPORT fails with the following error message...
"IMPORTing table tsty
IMPORTing table tloo
IMPORTing table tccd
%SQL-F-IOERROR, an unexpected I/O error occurred
%RMS-W-RTB, 8224 byte record too large for user's buffer
%SQL-F-RESABORT, terminating the IMPORT operation.

I have attached a txt with the details of the IMPORT command I used, Import job failure log, & "$dir/full fname" and other txt file with UAF and SYSGEN parameters details.

I can't EXPORT our DB into 8GB as our DB is size larger than 8GB, but I tried with different UAF values and vdisks size, of larger than 8GB.

Is there any limitation of using vdisk of size larger than 8GB for IMPORT?
but I wonder how could the Backup and EXPORT works with any vdisks of size larger than 8GB.

Any idea? and suggestions?
Thanks Navipa

[Hoff]

The metadata associated with the sequential export-import files here are probably corrupt, or the files are corrupt.   This is typical of RMS files being moved across a network.  

While it might be possible to reset attributes, it's also possible the files have gotten hosed during the transfer. 

To protect the files during transfer, get a current version of zip — zip version 3.0, no earlier — and unzip — unzip version 6.0, no earlier — and then zip the source files with the "-V" option.   Quote the "-V" to prevent the case from changing.   Re-transfer the files.   Try again.  

Older versions of zip and unzip do not support archive sizes past about two gibibytes; 2 GiB.   You're way past that.

Some zip background. 

Here's the zip and unzip source code.

I don't know of pre-built versions of these tools off-hand — the various places these tools have been posted around the 'net are uniformly down-revision.   Sometimes badly.    Though some of which are good enough to get the current versions of zip and unzip unpacked, and then built.

(The versions of these and many of the other tools posted at HPE site are archiac, and the HPE folks are unresponsive.   The version of zip that HPE uses for kitting patches is positively ancient, and with more than a few known problems.)

You can also use BACKUP to protect the files, though that will get corrupted.   It's feasible to repair the most common BACKUP saveset corruptions.

It's also possible that there's some other problem here with this fossil version of Oracle Rdb, but what you're seemingly concerned — vdisks, quotas, etc — is not likely relevant here.

[Steven Schweda]

> To protect the files during transfer, get a current version of zip
> [...]

   Do we have any evidence that ZIP was used to package/move any files
here?

> I don't know of pre-built versions of these tools off-hand [...]

   Mr. Goatley has generally built some for major releases.  What's
available should be at:

      ftp://ftp.info-zip.org/pub/infozip/vms/

   For the more adventuresome, newer (beta) source kits should be
available at:

      ftp://ftp.info-zip.org/pub/infozip/beta/

And the Info-ZIP team has been known to build special kits on request
for folks who lack a compiler.

> You can also use BACKUP to protect the files, though that will get
> corrupted. [...]

   Well, _can_ get corrupted.

   In any case, it might help to get a more complete description of
exactly which data were moved how.  I'd expect any all-VMS method
(BACKUP, COPY) to cause no trouble, but any non-VMS method involving a
network opens up more potential for trouble.

[abrsvc]

Maybe its me, but I don't see any Zip or Unzip involved here.  I see backup and export being used to get the files to the larger disk.  This should work as expected as there is nothing to "change" the data outside of RDB.  I would expect that these utilities work regardless of the disk size unless there is a limitation of the OS.

Can you better describe what was actually attempted?  Exact commands would be best.   Maybe the sequence used was in error.

Dan

[Hoff]

The suggested addition of the zip command is to protect the file and its metadata during whatever sort of network transfer was likely used here.   It's not the cause.  

Rule of thumb with BACKUP: savesets get corrupted by ftp until and unless they don't, and I'd prefer to describe how to fix the corruptions rather than to let inexperienced folks run into it fresh and have to ask for more help. 

The pre-built VAX unzip posted at Process is not current.  Based on strings and grep, it's 5.52.   Good enough to unpack the current unzip source archive version (and then build that), but that 5.52 version is not compatible with larger zip archives.    For grins, I checked the Alpha unzip image posted there, and it too is 5.52; old.   (I haven't reported that to Hunter, having just had somebody bump into that yesterday.)

[Steven Schweda]

> The pre-built VAX unzip posted at Process is not current.

   True, but...

> Based on strings and grep, it's 5.52. Good enough to unpack the
> current unzip source archive version (and then build that), but that
> 5.52 version is not compatible with larger zip archives.

   What it's good enough for is to unpack the unz60xv.zip and
zip30xv.zip kits in the same directory, which should contain more modern
pre-built executables (with large-file support for the Alpha and IA64
executables).  The "readme" there is out-of-date, and doesn't make this
as clear as it could/should.

   But in all this Zip+UnZip detail, I forgot that this is a VAX
(-equivalent) system, and the Info-ZIP programs (all versions) lack
large-file support on VAX (because the C RTL does).  So Zip and UnZip
may be of little use on VAX for files of this size.  As usual,
everything's complicated.

[Navipa]

Thanks Dan,

I have modified my posting and it has the information you as for, and I attached two txt documant which has the details about DB IMPORT command and the error logs.

I was using 8GB Vdisk to take the RDB backup and for DB Reclaim 9DB Export and Import), it ws working fine, but now my DB size incrased bigger than 8GB, so I created new Vdisk of size 9GB (tested with 10 and 12 GB also) to do the same BACKUP and DB Reclaim. With new larger Vdisk, Backup works, but DB Restore fails (error give below) & DB Export works, but DBIMport fails after Importing few of the tables. The error are given again below...

DB IMPORT Error
IMPORTing relation tsty
IMPORTing relation tloo
IMPORTing relation tccd
%RDO-E-IOERROR, an unexpected I/O error occurred
%RMS-W-RTB, 8224 byte record too large for user's buffer
%RDO-F-RESABORT, terminating the IMPORT operation

DB Restore Error
%RMU-I-LOGRESSST, restored storage area DBA_01:[PROD]UNIFORM_AREA_01.RDA;1
%RMU-I-LOGRESSST, restored storage area DBA_02:[PROD]UNIFORM_AREA_02.RDA;1
%RMU-I-LOGRESSST, restored storage area DBA_03:[PROD]MIXED_AREA_03.RDA;1
%RMU-I-LOGRESSST, restored storage area DBA_04:[PROD]MIXED_AREA_04.RDA;1
%RMU-E-READERR, error reading $DISK1:[BACKUP]TELDB_RDB.RBF;1
-RMU-E-BLOCKCRC, software block CRC error
%RMU-W-BADPTLARE, invalid larea for uniform format data page 380468
%RMU-W-BADPTLAR2, SPAM larea_dbid: 73, page larea_dbid: 19529
%RMU-E-INVRECTYP, invalid record type in backup file
%RMU-F-FATALERR, fatal error on RESTORE

[Hoff]

Get formal help.  I know you won't though, and I know that the managers involved here have forced these constraints onto this configuration — it's a fossil version of OpenVMS VAX, it's VAX, it's a fossil version of Oracle Rdb, little or no staff support and no escalation support, etc — and I'd suspect that your managers have made their project constraints into your problem.   I also know this migration has been going on for vastly longer than it should have — getting a user-mode application from a real VAX to an emulator takes a few days for the basic data transfer, barring emulator bugs or device-specific dependencies and some logical names, or some really slow data links.   So...  

Figure out how to protect the files during transfer.   If zip won't do large files, then use BACKUP.    Expect to have to reset the file attributes when the BACKUP arrives on the destination system.   There are other potential transfer paths — DECnet being one — but those can require more knowledge and potentially more troubleshooting, and I'd tend to avoid that here.

Much like your focus on quotas — in the absence of quota-specific errors — it ain't the Rdb import that's the problem here.   Yes, that import is blowing up.   But the problem is almost certainly upstream from Rdb.  

Assuming the commands used to export the data are valid, how are the exports being transferred from the source host to the destination host.  Whatever you're doing here for the file transfer is what is screwing up the import.   Protect The RMS Files containing the database export.   Use a BACKUP saveset to do that.   If all the export files are in the directory dev:[dir], then use the command:

BACKUP dev:[dir]*.* FOO.BCK/SAVE

Transfer FOO.BCK to the emulator, using whatever corrupting path is in use now.

Reset the attributes on the BACKUP saveset file per the steps and tools in the previously-linked article.   Then restore the saveset:

BACKUP FOO.BCK/SAVE otherdev:[otherdir]

Then populate the database.

[Hein van den Heuvel]

>> Backup and EXPORT work fine with any size of vdisks, including the size larger than 8GB. But the IMPORT

How can you say that? I think there is better than 50% odds that the corruption was established during the export. Creating the backup/export failed, even failing to report that it failed, because it did not realize it.

The import tool worked fine. If ran into a corrupted RMS record and reported that without silently corrupting the database so the import program worked abolutely fine, but the import operation had to fail (because the fiel was bad). Can we agree on that?

That number 8224 ( decimal. 0x2020) is NOT a random number. Come'on you guys, you've seen it before. 8224 is the decimal value for a string of 2 spaces. So RMS ran into pure data where it expected a record length.

Both the Import and export tool are probably fine. Why? Unless RDB at that time used it's own mimic of RMS file writes, the RDB tools would not give one hoot about the file size. They just asks RMS to write (PUT) and read (GET) recrods and RMS had better do the rigth thing. That may have failed. In fact it must have failed because the corruption is in bytes which RDB tools cannot touch.. As long as we are fingerpointing, RMS is unlikely to be in the wrong as well. It is just the messenger and likely fell victim to the file-system.

The next step in the investigation would be to find out which record was bad cq which was the last good one. You can try $ ANALYZE/RMS/CHECK on the file, or write a quick program to look for a record longer than MRS (or LRL) = 1024.

Is there any Network or ZIP involved here? Or are the replies mentioing that just noise?

What openVMS version. 6.1? 5.5-2? You may well have run into an VMS bug @ 8GB.  

[edit: I now see VAX VMS 6.1 in first line of main topic... I somehow missed that. Hein]

This is all wild speculation, but if this problem started to occur repeatably once disks were larger than 8GB, your only workaround MIGHT be to use 7.99GB disks bound together in a volume set to give 16GB of space.

Just thinking out loud,

Hein.

[Hein van den Heuvel]

For jucks I adapted a little program I had to (RMS) read a file an trap records which are too long.

It reports the number of records succesfully read, and the RFA (VBN,Byte-offset) for the last record read.

You may want to try it agains your file. Grab the attachment, save to OpenVMS as RTB.MAR compile and link.

Once it reports the RTB error, use DUMP/RECORD=(COUNT=2, START=xxx)  on were XXX = the record count to see what's going on according to RMS. Or, use DUMP/BLOCK=(COUNT=2,START=yyy) where YYY is the VBN part of the RFA. to see the gory details.

In the example below, I took the program source which has a line of 106 bytes and changed the file attribute to a maximum recordsize of 100, which the program than uses as user buffer size, causing a failure reading the 106 byte record. In the example, the dump just shows all is normal... if I had not mucked with the MRS (or LRL)

$ macro rtb
$ link rtb
$ mcr sys$login:rtb tmp.tmp
112 Records, Last=(7,20), 2839 Bytes. Avg=25, LRL=106 @ 19, SRL=0 @ 1
%SYSTEM-S-NORMAL, normal successful completion
$ set file/att=mrs=100 tmp.tmp  ! Force incorrect attribute the file.
$ mcr sys$login:rtb tmp.tmp
18 Records, Last=(2,96), 567 Bytes. Avg=31, LRL=68 @ 16, SRL=0 @ 1
%RMS-W-RTB, 106 byte record too large for user's buffer
$ dump/rec=(start=18,count=2) tmp.tmp
Dump of file TMP.TMP on  8-MAR-2016 00:19:11.33
File ID (60411,8,0)   End of file block 7 / Allocated 9

Record number 18 (00000012), 0 (0000) bytes, RFA(0002,0000,0060)

Record number 19 (00000013), 106 (006A) bytes, RFA(0002,0000,0062)

 20202020 3A4C4F52 544E4F43 5F4F4146 FAO_CONTROL:     000000
 52204C55 21222020 44494353 412E2020   .ASCID  "!UL R 000010
 5521283D 7473614C 202C7364 726F6365 ecords, Last=(!U 000020
 74794220 51554021 202C2957 55212C4C L,!UW), !@UQ Byt 000030
 4C524C20 2C4C5521 3D677641 202E7365 es. Avg=!UL, LRL 000040
 3D4C5253 202C4C55 21204020 4C55213D =!UL @ !UL, SRL= 000050
              224C 55212040 204C5521 !UL @ !UL"...... 000060

[Volker Halle]

Navipa,

what kind of disks are you using with the CHARON-VAX emulator ? DKA or DUA disks ?

There were patches for DKDRIVER (SCSI) to support disk sizes > 8.6 GB for OpenVMS Alpha in 1999. So the same problem might have applied to OpenVMS VAX V6.1.

Volker.

[Hein van den Heuvel]

Ah yes, it's all coming back. Some issue with just supporting 24 bits for block numbers in the SCSI protocol.

0x01000000 = 16777216 

16777215 blocks of 512 bytes = 8589934080 = 8 GiB (gibibyte) = 8.6 GB

If that's the bug, then a bound volume set will get the OP goign for a few more years.

MOUNT/SYS/BIND=bigdisk  dka200,dka300 label2,label3

Search for 8GB in : http://www.hoffmanlabs.com/vmsfaq/vmsfaq.pdf

Hein

[Dennis Handly]

>supporting 24 bits for block numbers in the SCSI protocol.

That's probably READ6/WRITE6.  READ10/WRITE10 go to 32 bit LBA.

[Navipa]

Hi Dennis,

I don't understand your answers, can you please give in details.

[Hein van den Heuvel]

Dennis is referering to, providing details for my comment : "Some issue with just supporting 24 bits for block numbers in the SCSI protocol."

For you Navipa it is more important to look at my reply about bound volumes. Did you see that, did you understand that?

It is likely to solve this issue without requireing upgrades or updates.

You do understand that this is a OpenVMS bug/limitation, and NOT an RDB bug right?

Of course you should still try to 'sell' becoming more up-to-date (or 'less ancient') and get the patches in places.

Cheers,

Hein.

[Dennis Handly]

>Dennis is referring to, providing details for my comment

That's correct.  Those are the names of the SCSI opcodes that have that limited LBA values.

[Navipa]

Wow!!!!!!!!!!, there are many responses!!!!!!!!!, I am sorry. As I was too hurry to resolve or finding some other solution, I decided to go with Image backup options as a temporary solution.  Anyway I need to resolve the issue as Image Backup is the temporary solution.

Let me go through all your responses and suggestions and I respond you soon.

[Navipa]

Hi Dan,
As I mentioned initially and in the attached document, I am trying to RMU/Backup and /or EXPORT/IMPORT. RMU/Backup works but RMU restore doesn't, the sameway the EXPORT works, but IMPORT doesn't.

DB files are spread into 4 disks, the total RBF file's size is larger than 8GB. I have created vdisk of size 8.9GB, 9.5GB, 10 GB adn 12GB to store the RBF and RBR saveset files. I guess 8GB might be the maximum suported disk size, But I am not sure about the maximum supported size. Please see the attached files which show the error I receive while RMU/Restore and IMPORT.

[Navipa]

Dear Holf and Stewen,

I did not use any ZIP or UNZIP with this procedure. Even I never FTPed any zip or unzip file and restore into this server. I m doing everything within the local server.

Yes I uderstand the problem with ZIP and UNZIP, I had this sometime back and resolved with your earlier suggestion, that was different server to which I brought the RBF file frm the repmote server.

[Navipa]

Hi Holf,

I am Exporting and Importing within the local DB, so no question of FTP. Please see the attachment which has file attributes, import log and export detail. for the import log and errors. 

[Navipa]

Hi Volker,
....We use DSSI disks, and VAX4108 comes with 2 DSSI controllers and not necessarily load any driver and virtual tape MUA and MUB.

Config file contents....
...
.....
load TQK50 MUA address = 017774500
set MUA containter[0]="d:\tel_tapes\tape_mua0.mtd"
load TQK50 MUB container=017760404
set MUB container[0] ="d:\tel_tapes\tape_mub0.mtd"

#Configure DSSI disk, vax4108 comes with 2 DSSI controllers
set paa mscp_allocation_class[0]=2 scs_node_name[0]="DISK0"
set paa container[0]="d:\tel_vdisks\dia0.vdisk"
....
.....
# I have configured 12 vdisks and it works totally.

Volker,
Mainly I would like to resolve the vdisk issue (must), but I have 2nd question for the future, seems I have 2 virt.tapes MUA and MUB.  But I never used virt.tapes for backup, as I don't know how to define its size and its max size, is it possible to have 12 GB or 16 GB tapes and can I avoid the above said error by replaing vdisk with vtapes?

[Navipa]

Thanks Hein.
I am unable to understand your suggestions clearly... Do you want to me mount source or destination volume? can you please give some details on this bound volume. seems I can make it work for few years without patching, .....

[Hein van den Heuvel · Accepted solution]

Read my replies again, and read slowly. And open up the OpenVMS manuals

1) RMU/Backup works but RMU restore doesn't, the sameway the EXPORT works, but IMPORT doesn't.

Clearly the import commands does not succceed, but it is NOT broken, OpenVMS is to blame.

2) As I wrote, the ancient OpenVMS version you work with has a maximum (scsi) disk size of : 16777215 blocks of 512 bytes = 8589934080 = 8 GiB (gibibyte) = 8.6 GB  - What's not clear about that?

3) If a file larger than the about 16.7 M blocks is needed, as it is, then you can use a BOUND VOLUME SET to combine multiple physical disks. You create one with a command similar to MOUNT/SYS/BIND=bigdisk  dka200,dka300 label2,label3  after an INIT DKA200 label2; INIT DKA300 label3. - This structure can hold the target of the export / source for the import

>> seems I can make it work for few years without patching, .....

Yes, I think so.

Hein

[abrsvc]

re-wording and summarizing what Hein posted:

An individual SCSI disk limitation of 8GB disk supported by this OpenVMS version is the problem here.  Using bound volume sets avoids the problem by allowing for a "disk" larger than that limit.  The volume set will appear as a singular disk to the application but is handled as a group of smaller disks at the OS and hardware levels (in concept) which will prevent you from hitting the limit mentioned before.

Look up volume sets in the documentation.  For your specific case, I would bind at least 3 disks together to create a set of 24GB total.  That should give you some breathing space.

Dan

[Navipa]

Thanks Hein and Dan (abrsvc).
You are great!!! that bound volume set I created using 2x 8GB (16777215) resolved one issue which is RMU/RESTORE, but still I have a issue with IMPORT, but this time with different error (below). I have created a new post with that error. Expecting your assistance to resolve that IMPORT issues either with BOUND volume or some other way as my DB size keeps growing without resolving this EXPORT/IMPORT issues.

IMPORTing table tccd
%SQL-F-NODATRES, remaining data for this relation will be ignored
%RDB-F-SYS_REQUEST, error from system services request
-RDMS-F-FILACCERR, error extending file SYS$SYSROOT:[RDM$RUJ]teldb$00B0712182776E40.RUJ;1
-SYSTEM-F-BADPARAM, bad parameter value
%RDB-F-SYS_REQUEST, error from system services request
-RDMS-F-FILACCERR, error extending file SYS$SYSROOT:[RDM$RUJ]teldb$00B0712182776E40.RUJ;1
-SYSTEM-F-BADPARAM, bad parameter value

IMPORTing table tcop
%SQL-F-NORELRES, unable to import table tcop
%RDB-F-SYS_REQUEST, error from system services request
-RDB-E-NO_META_UPDATE, metadata update failed
-RDMS-F-FILACCERR, error extending file SYS$SYSROOT:[RDM$RUJ]teldb$00B0712182776E40.RUJ;1
-SYSTEM-F-BADPARAM, bad parameter value
%RDMS-I-BUGCHKDMP, generating bugcheck dump file SYS$SYSROOT:[SYSMGR]RDSBUGCHK.DMP;27

Thanks
Navipa