Problem with an index file on VMS

[Christopher Blackburn]

Hi,
Has anyone experienced any problems with RMS index files on OpenVMS 8.1,8.3 ?
I have an index file with two indices, the first is an alphabetic key of 12 characters and the second is an alphabetic key of 9 characters.
The index file is created with an FDL file and works fine for some time but at some point the second index gets out of sync with the first. Thus if one access the file using the first index a number of records are returned and when the second index is used a different number of records is returned.
I can't tell what operation causes the index to get corrupted i.e. Insert record, Update record or Delete record. Using ANALYSE/RMS uncovers no errors in the structure of the index file but does show the difference in the two indices.

Many thanks in advance for any suggestions.

Chris.

[Wim Van den Wyngaert]

Just a guess : key values that are NULL are not returned when you read. Did you specify a null field option in the FDL ? Posting the fdl would be nice.

Wim

[Christopher Blackburn]

I have attached the FLD used to create the file. I will try to investigate if a NULL key value has caused the problem.

[Wim Van den Wyngaert]

You don't seem to have null keys. But is this the fdl that was used to create the file ? anal/rms/fdl would be more sure.

How did you read the file ?

Wim

[Hoff]

"...but does show the difference in the two indices...": Can you post up that report?

Ok; this looks to be an old application, and one that's been ported forward to OpenVMS Alpha or OpenVMS I64?

The only 8.1 around was on OpenVMS I64 during its pre-production. I'll assume this application is not running OpenVMS I64 V8.1, as there were a number of problems with that release. It was most definitely not "production quality" code.

I'll assume current ECOs for whatever version of OpenVMS this is. If not, do get there.

Is there the possibility there is a partitioned cluster operating? Anybody been "getting creative" with VOTES and/or EXPECTED_VOTES? That'll stomp on files.

Are there any disk errors being logged? That's less likely, given the re-occurrences, but flaky hardware can haunt a configuration.

How is the file being created and operated? fdl$create? CREATE/FDL? What application? Local? Has the application been reviewed for coding errors? (That written, having different record counts on two keys should not normally happen, even with a coding bug.)

I would definitely review the code used to perform the key-related processing. If something in the code or in RMS is stomping on a key, some other code might not find what it expects and mis-report. (But I'd not expect that to arise from your reported ANALYZE/RMS pass.)

My first step would be to periodically scan the two keys using external standalone software. Key scans are simple hunks of code, and you can quickly catch and flag these errors. If you do have two different record counts (and no NULL KEY setting), that points toward a nasty problem in RMS.

The posted (old?) FDL makes the file look pretty simple; there's no real oddness apparent in the FDL. And no NULL KEY there.

Hein's rms_tune_check tool from the Freeware (directory rms_tools) would be a good option, too.

And yes, ANALYZE/RMS/FDL or such, and post the FDL.

With straight text keys, you can use a couple of DCL loops to read off the keys, or a smattering of C or other language code. (Examples of both can be found on the Freeware, and in the Ask James support database, and elsewhere.)

Stephen Hoffman
HoffmanLabs LLC

[Christopher Blackburn]

Wim,

I have used ANALYSE/RMS/FDL on the file and attached the results. The original FDL file was used to create the file. On our operational system we tidy the file up every six moths by reading the records from the index file sequentially and re-writing them to a new index file created with the FDL.
Reading from the file is performed using FORTRAN READ statements with the KEYID and KEYGE arguments. Subsequent reads are performed by using a simple READ statement.

Many thanks for your time.

[Wim Van den Wyngaert]

Probably garbage in the index and your Fortran is reading good stuff only.

Here's a test.com. Execute is with p1= your file and p2=0. Then again with p2=1.

$ open/read/share x 'p1'
$ i=0
$r:
$ read/end=e/index='p2' x x_rec
$ i=i+1
$ goto r
$e:
$ write sys$output "Via index 'p2' : ''i' records"
$ close x


Wim

[Christopher Blackburn]

Hoff,

Software can be used in a cluster but has failed on a single node. The index file is only used by a single process on a single node. The index file is created from DCL with a CREATE/FDL statement but from then on it is written to and read from VMS software written in FORTRAN. You are correct the code is old and has been working on previous versions of VMS without problems. I have user ANALYZE/RMS/CHECK to check consistency of file and it looks OK.

[Christopher Blackburn]

Wim,
When I run the DCL command file on the file it returns the same number of records for index 0 and index 1. I know when I step through the code in DEBUG it reads 6 keys from key0 and 0 keys from key1.
It must be the way we are reading the key values from the file. The primary index is read from FORTRAN using a READ statement with KEYID=0 and KEYGE=' ' and the secondary index is a READ statement with KEYID=1 and KEYGE=ICHAR(0).

Many thanks for your time - Chris.

[Hein van den Heuvel]

Chris,

Focus on the code. Trust RMS in this.

The code being used is broken.
100% sure, garantueed, no ifs or buts.

The used code is not correct. it probaly never was but you had the bad luck that the problem went unnoticed.

Potential problem areas....

1) The code fails to deal with duplicates correctly. There is a large number of duplicates when reading according to key #1.

2) The alternate key, while declared as string actually contains binary data and the the resulting sort order only now comes to bite the application in the rear.

If need be, reduce the code to a bare minimum and post as a text attachment.

Good luck!
Hein van den Heuvel ( at gmail dot com )

[Hein van den Heuvel]

In RMS, at the lowest level you can CONNECT an access stream to a file an specify a KEY ID. After that a simple read will return the first record by that key. That's what Wim's DCL example ends up doing.

When a KEYGE is also used then of course RMS does not start at tyhe begin, but at the place indicated by the contents of the current key buffer.

IF, and only if, that key buffer contains binary zero value will can you be sure that the KGE returns the first record.

So reason number 3, and the most likely reason, is that the read by key 0 left a value in the key buffer which is used as that start and which is not lower than the lowest actual key.

And reason number 4 is that the code does not really do that keyge on hte alternate key, but only switches key thus continuing on at whatever the place the in the alternate key structure the last record in the promary key structure had.

They way to analyze this is to use the DEBUGGER (or PRINT statements) to looks at the actual alternate key values returned to the progam for the first one or two record returned, and for the last record read by primary.

Please also note that an average of 30 duplicates as seen now is not problem at all, but can be in indication of potential problems in the future should the quantity of data grow to millions.


Cheers,
Hein.

[Hein van den Heuvel]

Oh... I think I see it.


You indiocate to pass: ICHAR(0).

That's not valid is it?
ICHAR accepts a single characted string and returns an integer, but you pass it an integer and sugegst to return a char.

You want to use ACHAR with returns a char and accepts an integer!

Hein.

[Christopher Blackburn]

Hein,
Sorry my mistake. The actual code reads from the file with a key greater than or equal to a string which is set to a char 0 (NULL). This was originaly intended to read all of the key values out of the second index as the valid keys stored in the file will be greater than ASCII zero.
This appears to work most of the time but the index file becomes corrupted at some point and the second index no longer seems to work. If the file is read sequentially and each record read is written back into a new file (created with CREATE/FDL) the indices are then OK for a while until the secondary index becomes corrupt again.

Chris.

[Wim Van den Wyngaert]

Your file is not corrupted.
If you re-create the file and fill it with the records of the original file, you end up with an identical file (behaviour of duplicates within a key value excluded).

IMO there's something wrong with your story.

Or your read-write is doing more than read-write.

"but the index file becomes corrupted at some point and the second index no longer seems to work"
-> what do you mean with this ? Is there a relation between the 2 indexes (e.g. composed key) ?

Wim

[Christopher Blackburn]

Wim,
I've just amended the DCL script to read from the file using index=0, key=" ", match=ge followed by normal reads for the remaining records. All 6 come out of the file and in the correct order.
When I re-run but using index=1, key=" ", match=ge or match=le no records are returned. All I get is %RMS, Record not found.
It still appears that the second index has become corrupt at some point. If I re-create the file by doing a CREATE/FDL and read all original records in sequence and write into the newly created file the two indices are OK again.

Chris.

[Doug Phillips]

From everything that's been said so far, I wonder if the CREATE is failing during the 2ndary key phase. Are you doing the "reorg's" via a command file or interactively?
Do you have any logs of the CREATE? I've had similar "problems" due to insufficient disk space or some other resource.

[Hein van den Heuvel]

>Wim,
>I've just amended the DCL script to read
> from the file using index=0, key=" ",
> match=ge followed by normal reads for the
> remaining records. All 6 come out of the
> file and in the correct order.

I'm a little confused by just 6 records when ANALYZE/RMS suggests 2630. Oh well.

> When I re-run but using index=1, key=" ", match=ge or match=le no records are returned. All I get is %RMS, Record not found.

So apparently there are no records starting with a single space character or higher valued character. Maybe the records start with a NULL?

>> It still appears that the second index has become corrupt at some point.

I see no reason, and certainly no prove, to call it corrupt. It's just behaving different from what you expect.

You may want to go in with ANAL/RMS/INT
DOWN
DOWN
DOWN KEY
NEXT
DOWN DATA
DOWN
[next]

This will give you the first key for the first record by alternate key.

BTW... this is a prologue 1 file?! You may want to address that.

>> If I re-create the file by doing a CREATE/FDL and read all original records in sequence and write into the newly created file the two indices are OK again.

That is NOT a recreate. The program might be munging the data, in fact it appears it does.
A simple re-create, which is garantueed to clear corruption should there be any, is to use $CONVERT/FDL=xxx old new/STAT.

That will give a 100% solid proof whether an index is corrupt. Just convert and re-try. If the 'problem' stays after convert, but goes away after re-create by program then the program 'adds value'. And apparently the read-rewrite process substracts that value.

Speaking of convert. instead of the DCL loop script you might want to use covnert stats to see the counts:
$CONVERT/STAT/FDL=nl:/KEY=1 old NL:

Finally, while a dcl convert will garantueed create a correct file, it can become different from the original as dusplicate alternate keys for the original file are presented in order of arrival whilest they will be in order of primary key after the convert.

If this still does not resolve the confusion, then you may want to contact me by Email and we can discuss spending serious time on this. If performance for this file is at all important, then there are several suggestions to be made.

Cheers,
Hein.

[John Gillings]

Chris,

A very simple method to check if all records are accessible on all key chains, use CONVERT to extract the records by key:

$ CONVERT yourfile KEY0.REC/KEY=0/FDL="FILE; ORGANIZATION SEQUENTIAL"
$ SORT KEY0.REC KEY0.REC

$ CONVERT yourfile KEY1.REC/KEY=1/FDL="FILE; ORGANIZATION SEQUENTIAL"
$ SORT KEY1.REC KEY1.REC

$ CONVERT yourfile KEY2.REC/KEY=2/FDL="FILE; ORGANIZATION SEQUENTIAL"
$ SORT KEY2.REC KEY2.REC

etc... for each key

Now compare:

$ DIFF/MODE=HEX KEY0.REC KEY1.REC
$ DIFF/MODE=HEX KEY0.REC KEY2.REC

etc...

NOTE - a difference is not necessarily an error, as some of the keys may have null values.

(depending on the definitions, you may be able to just CONVERT/KEY and compare the output files)

[Christopher Blackburn]

Doug,
Thanks for input. I log any failure during the create of records in the index and nothing has been logged. There is plenty of free disk space etc.

Hein,
Thanks for input, I have tried the CONVERT using KEY=0 and is says 6 records were processed. When I use KEY=1, 0 records were processed. When I convert the complete file it does not identify any errors.
I still believe the file has become corrupted in some way (or at least the secondary index has become corrupted).
The fact that no records were returned when I looked for values GE and LE to a space would indicate to me that there are no records in the file using the secondary index.
I've also used the ANALYZE/RMS/INTER option to view data in the file. There are a large number of deleted keys in the file and this makes the output from ANALYZE difficult to make sense of. I have attached the output from analyse.
I will look at updating the prolog of the file to 2 (rather than 1) and see if there is another way of organising the key to be numeric rather than using an ASCII key to order binary data.

John,
Thanks for your suggestion. It confirms the secondary key has become corrupt and does not contain the correct number of entries.

Many thanks again for all your suggestions.

[Hein van den Heuvel · Accepted solution]

>> I have tried the CONVERT using KEY=0 and is says 6 records were processed. When I use KEY=1, 0 records were processed

The originally posted FDL showed many more records, so that was confusing.

Now I am starting to think corruption.
Since this file has fixed length records, and no null key every record should have a valid alternate index.

When I indicated no corruption seen, that was pretty much for prologue-3 files.
Prologue-1 files are rarely used and often come with a file which badly needs to be tuned anyway. (such as is the case for your file... it stinks).

RMS Engineering should be somewhat interested in corrupted prologue-1 files, specifically if it comes with a reproducing program + dataset + show RMS (to verify how many buffers are used by default).
However, it would be tempting to just say ' convert to prologue 3 and see if the problem goes away'.


>> Hein,
Thanks for input, I have tried the CONVERT using KEY=0 and is says 6 records were processed.

>> When I convert the complete file it does not identify any errors.

And does the count become 6/6 ?

>> There are a large number of deleted keys in the file and this makes the output from ANALYZE difficult to make sense of.

And would have caused a lot of work for RMS.
Convert to prologue 3 to improve matters!
Use a larger bucket size!

>> I will look at updating the prolog of the file to 2 (rather than 1)

Go prologue 3

>> see if there is another way of organising the key to be numeric rather than using an ASCII key to order binary data.

You'll have to squeeze a byte away out of the data as RMS supports only 8 byte binary/integer. You could possibly go packed-decimal, but there is very little experience with that. You could redefine the key, without changing the record, to become a segmented key defined LSB --> MSB to get the binary order correct. There again the limit is 8 segments, so you'd have to find a MBZ (must be zero) byte or an other 2 byte portion of the data for which the sort order is not relevant.


Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Christopher Blackburn]

Many thanks for all the input.

I will look at updating the index file to prolog 3 and increasing the bucket size to improve performance.

While investigating the problem I have found an obscure way that records with NULLs in their secondary key can be written to the file. It may well be these null value keys that are causing the corruption.