Something strange happens when I am sequentially accessing records in indexed file.

[Sergejus Zabinskis]

Hi guys,
I have several processes (executing the same image) that are putting and updating records in indexed variable length record file (mrs=1024 bytes).
File has only one index that is made from the first 14 bytes or the record:
ID (__int64);
recordno (int)
fieldno (short).

Sometimes I need to copy some records from this file to another. I am sequentially accessing all records
in source file and put them to destination. Doing that I noticed some strange things with ordering in source file.
Here is record sequence that is good:
========================
ID recordno fieldno
========================
74138282771 0 0
74138282771 1 1
74138282771 1 2
74138282771 1 3
74138282771 1 4
74138282771 1 5
74138282771 1 6
74138282771 1 7
74138282771 1 8
74138282771 1 9
74138282771 2 1
74138282771 2 2
74138282771 2 3
74138282771 2 4
74138282771 2 5
74138282771 2 6
74138282771 2 7
74138282771 2 8
74138282771 2 9

and sometimes (for very long recordsets where recordno changes from 0 to 50000 or even more) I see something like that:

4815790862337 0 0
4815790862337 256 1
4815790862337 256 2
4815790862337 256 3
4815790862337 256 4
4815790862337 512 1
4815790862337 512 2
4815790862337 512 3
4815790862337 512 4
4815790862337 768 1
4815790862337 768 2
4815790862337 768 3
4815790862337 768 4
4815790862337 1024 1
4815790862337 1024 2
4815790862337 1024 3
4815790862337 1024 4
4815790862337 1280 1
4815790862337 1280 2
4815790862337 1280 3
4815790862337 1280 4
4815790862337 1536 1
4815790862337 1536 2
4815790862337 1536 3
4815790862337 1536 4
......
4815790862337 51200 1
4815790862337 51200 2
4815790862337 51200 3
4815790862337 51200 4
4815790862337 51968 1
4815790862337 51968 2
4815790862337 51968 3
4815790862337 51968 4
4815790862337 1 1
4815790862337 1 2
4815790862337 1 3
4815790862337 1 4
4815790862337 257 1
4815790862337 257 2
4815790862337 257 3
4815790862337 257 4
4815790862337 513 1
4815790862337 513 2
4815790862337 513 3
4815790862337 513 4
4815790862337 769 1
4815790862337 769 2
4815790862337 769 3
4815790862337 769 4
.....

To be clear, recordno is changing according this algorithm (Nmax - is max.number of records and
Nmax > 256).

recordno = 0;
fieldno = 0;

for(k = 0; k < 256; k++)
{
for(j=1; k + 256*j <= Nmax; j++)
{
recordno = k + 256*j;
}
}

I feel that if number of records for some ID is < 256 - recordset will be ordered correctly,
otherwise according algorithm given above.
Mayne somebody can explain this situation ....

Sorry for the very long story.
Kind regards
Sergejus

Here is some info on source file:

OMNITE $ dir DML_SERVER.TBL_TASK/full

Directory $1$DGA4342:[DMLTEST]

DML_SERVER.TBL_TASK;1 File ID: (6317,66,0)
Size: 434340/434340 Owner: [DMLTEST]
Created: 27-MAY-2005 16:00:42.63
Revised: 6-SEP-2005 14:46:58.13 (273)
Expires:
Backup:
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Indexed, Prolog: 3, Using 1 key
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 434340, Extend: 0, Maximum bucket size: 3, Global buffer count: 40, No version limit
Record format: Variable length, maximum 1024 bytes, longest 0 bytes
Record attributes: None
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWE, Group:, World:
Access Cntrl List: (IDENTIFIER=JUP_TEST1,ACCESS=READ+WRITE+DELETE+CONTROL)
(IDENTIFIER=JUP_TRAINING,ACCESS=READ+WRITE+DELETE+CONTROL)
Client attributes: None

Total of 1 file, 434340/434340 blocks.

[Bojan Nemec · Accepted solution]

Sergejus,

The key is probably of string type. Bytes are sorted as signed bytes. In the sequence (I ommit the ID part):

xxxx 0 0
xxxx 256 1
xxxx 512 1
xxxx 1 1

you have realy this sequence (because of little endians):

ID recordno fieldno
xxxx 0 0 0 0 0 0
xxxx 0 1 0 0 1 0
xxxx 0 2 0 0 1 0
xxxx 1 0 0 0 1 0

and if you look so the sequence is in right order. (The same thing is with the id).

Bojan

[Bojan Nemec]

Sorry,

A typo in previous post:

Bytes are sorted as unsigned bytes.

Bojan

[Hein van den Heuvel]

Sergejus,

Carefully read Bojan's reply.
I'm pretty sure he nailed the problem.
To verify this you need to look at the key definition. (and perhaps shared the with us)
You do this using ANAL/RMS/FDL
Bojan (and me) suspect to see a single unsegmented key something like:
SEG0_LENGTH 14
SEG0_POSITION 0
That will NOT do what you want.

You can not get what you want from RMS, but you can get close, by defining the key as SEGMENTED. Unfortunately RMS only allows TEXT attributes for segmented keys, So you are stuck with each segment being a single byte drilling down from most significant down to least. You'll need to drop some bytes from the sub-fields to get there. (if you can get there).

The easier/better solution is to change the application to use a real text key, a hex key, or a BCD key to get proper sorting in a single segment

Good luck,
Hein.

[Hein van den Heuvel]

Oh... I forgot to throw in the 'free advice'.
This file probably needs tuning.
That bucket size of 3 is the minimum to hold the maximum record size: 1024
It is unlikely to be optimal
(check out: ANAL/RMS/FDL ... EDIT/FDL/NOINT... CONV/FDL/FAST/NOSORT/STAT )

That 1024 MRS. That is an 'odd' number. Sounds like an application which tries to 'help' by adapting to what it thinks is a nice value to the system. Well, 1024 is not nice, or rather irrelevant for indexed files with variable length records (and compression).
(the 'not nice' part is that rms indexed file records are stored in buckets, where each bucket has 15 overhead bytes and each record 11 or 13 bytes overhead. So a two block 1024 byte bucket can hold at the most a 998 bytes... not that you would want just one record and no space for RRVs)

Hein.


Cheers,

[Sergejus Zabinskis]

Thanks Bojan and Hein.
Of course you are right.

Kind regards
Sergejus

[Sergejus Zabinskis]

I received right solution of my situation.