Interesting stuff. It does sound like a bug.
Willem, what exact VMS version/patch level? And how about an ANA/RMS/STATS output to make the picture complete?
Faris, can you reply with more details on your reproducer? Source code + fdl in attachment? How reliable dies this reproduce? Email me?
Willem>> This behaviour is gone after CONVERT, but all of a sudden, it does reappear,
Convert only uses record data, no old key structures, making completely new key/index structures. So it does sound like an alternate key corruption which is coming back (bug).
Bojan>> "For key 1: CHANGES yes and DUPLICATES yes which is not possible."
Correct, duplicates are not possible in this case. But please note that 'duplicates yes' is only a request to RMS to make sure no record with duplicate key value is being inserted. The file structure is exactly the same whether duplicates are allowed or not. So dups=no requests RMS to do a tiny bit more work. And since you know there can not be duplicates in this case it would seem a little cheaper not to ask to check. Whether it actually is cheaper or not does not matter, just showing why it may make sense to have this seemingly contradictory definition.
HOWEVER... whether duplicates are allowed or not DOES change the internal to RMS update behaviour. And RU-journalling adds yet an other twist.
So, Faris/Willem I'd like you to recovert with dups=no, to see whether this may deliver a workaround.
Jan>> kind of strange way to define the alternate key as a primary with one variable character IN FRONT
Actually, that would make sense for at least two reasons.
1) to de-duplicate the alternate record key. Otherwise you may have too many 'V' records and loose insert performance. Admittedly this is only significant with tens of thousands of dups and painfull millions of dups, but it is nto a bad idea.
2) to impose an order 'get next sequentially' with the key or refence set to an alternate key with dups will essentially return them in 'order of arrival' (which convert can destroy). That may or may not be desirable. Adding a primary key, or date field makes the order predictable.
Bojan>> when you have 2 states and need to read fast only records with one state) you can use null keys.
YES. but not the problem here.
Willem>> Thanks to Hein's marvelous program some light at least - see first part of attachement. All the records that have > 1 in the "dups" column (?) give problems
Thanks :-)! But the particular area you use in the program is little tests. It may well reported 'deleted' sidr array entries as real. RMS would/shoudl not return those. But since you have actuall problems on tose very records, I guee it is showing something.
Willem>>less than 48, a second 'valid' record will show op, and a third if 48 or higher.
Well, your FDL did NOT show the AREA info. If you did not do anyting special, then the bucket size might be just 2 (default) or even 1. Every sidr entry is 7 bytes. so with a bucket size of 2, you can store only (2*512 - 15 - keysize) / 7 =~ 150 dups / sidr. So with 48 or 60 sidr entries(for updated=deleted) you may well need to cross into a second bucket. This may well turn out to be the breaking point.
SO for an other potential workaround... reconvert with bucketsize = 24 or 63 or so (no science here, just a large number).
Faris>> The ones which have irc$v_deleted = 1
should not be returned by RMS.
Correct.
Hope this helps some,
Hein.
