topic Re: Indexed file as a FIFO buffer in Operating System - OpenVMS

Indexed file as a FIFO buffer

Michael Moroney — Thu, 03 Jan 2008 17:10:02 GMT

I am dealing with some VMS software that generates messages to be sent to another system. In order not to lose the messages if the other system is down, it doesn't send the messages directly. It writes the messages to a temporary indexed file, which is just used as a FIFO buffer. Another process loops reading records from the file, sends them to the remote system, and if successful, deletes the record. If there is no record, it waits and tries again.

The indexed file has one key, the date/time. It's just there to keep the records in order.

The problem is, the file grows and grows, and access gets slower and slower, despite the fact it's usually empty and rarely has more than a couple of records in it. They have a hack that does the equivalent of a $CONVERT/RECLAIM every so many records, but there must be a better way. Is there some FDL magic to tune a file where it is expected that every record is deleted? Or is there a better way to implement a file-based FIFO on VMS? I say file-based since any data (or the fact there is no data) must survive reboots.

Re: Indexed file as a FIFO buffer

Hein van den Heuvel — Thu, 03 Jan 2008 17:36:49 GMT

There are better ways.

Specifically they should probably learn to keep a last-time-looked key value, possibly in a lock value block and use that for a KGE lookup 99 out of 100 times.
Then once every 100 elements, or 1000 seconds (whichever comes first) scan the file from the beginning to catch out of order arrivals.

Also, review the bucket size, and duplicate key requirements.
Switch dups off, and selecting a large bucket size may keep this file under control.

I suspect that this will already help enough. If not, a day of consulting should do it. Several folks in this forum would be eager to help with that (for fee). Or maybe acquire a special program I happen to have for these cases. Send me Email!

This topic has similarities to an other question today: 1189881. Check that out. Post an ANALYZE/RMS/FDL/OUT example ?!

Hope this help some,

Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting

Re: Indexed file as a FIFO buffer

Hoff — Thu, 03 Jan 2008 18:03:45 GMT

So you're writing some RTR-like middleware. OK. (Writing custom local middleware wouldn't be my first choice when I could purchase same and let somebody else deal with support and upkeep, but there can be cases when rolling your own can be necessary.)

RMS indexed is not the best RMS structure choice for this case, since you care less about the sorted access provided by indexed files than about a sequence of messages. Your file (as you've discovered) grows.

First pass design: Two queues, with pending and free lists. Probably using a relative or a block or fixed-length file for the static storage, or using memory queues and a file-backed section and periodic flushes -- and the flushes could be coded to increase in frequency in response to the remote end dropping off-line. This is standard AST-driven server stuff.

End-points would want to (or have to) maintain sliding windows for processing potentially duplicate data. These sliding windows can be based on what your indexed files are presently using for keys. (Indexed files don't deal with sliding windows all that well.)

Biggest wrinkle would be what to do when the far end receiver was slower than your storage was big. You can only enbiggen for so long before you have to decide to stall (eg: back-pressure) or to drop.

Also cromulent would be the use of 2PC and transactions, whether XA or otherwise with DECdtm.

Middleware such as RTR would be the easiest. This stuff is already tied in with DECdtm.

A different design might parallel how Erlang and its Mnesia avoids writing to disk entirely by always keeping and always replicating the information in memory across multiple hosts, and assuming that some subset of the hosts would survive. Microsoft Research has something logically similar with its Niobe replication platform.

I have pointers to some of this stuff posted at the web site, if you can't find details elsewhere. (And assuming you're not going straight after the queues mentioned earlier, or after the approach that Hein suggests, or after RTR or other middleware.)

Stephen Hoffman
HoffmanLabs LLC

Re: Indexed file as a FIFO buffer

David Jones_21 — Thu, 03 Jan 2008 18:17:14 GMT

Doesn't RMS have a rule about not re-using RFAs (record file addresses) in a file so even if you delete a record it still consumes a little bit of space? You end with buckets full of deleted record markers. Does turning off 'allow duplicates' affect this.

I wonder if it would help if you used 2 keys, a primary key whose values are recycled (records re-written) and the timestamp in a secondary key. You never delete records, just overwrite them. Robustly tracking the free list of primary key values for the next write/rewrite would be the biggest issue.

Re: Indexed file as a FIFO buffer

Hein van den Heuvel — Thu, 03 Jan 2008 18:42:34 GMT

Hoff is absolutely right in suggesting the many alternatives. A 'circular' buffer in a sequential or relative file may well be the better design. Optionally add to that a (clusterwide) lock value block to hold the next free, last finished numbers to be updated to disk every so often.
And several products adrress this: RTR, MQseries, ...

My recommendation were geared to making the most of the current setup with minimal changes. All too often a requirement to get anything done at all!

David>> Doesn't RMS have a rule about not re-using RFAs (record file addresses) in a file so even if you delete a record it still consumes a little bit of space?

It does have that rule, but it implemented through a simple 'next free id' concept. This is a fixed word in the bucket header, always there. It does force a bucket split every 65K records targetted for the same data bucket!

Deleted records are nicely expunged EXCEPT:
a) The last (first !? :-) in a bucket
b) In the case duplicated are NOT allowed.
c) An odd key compression case where deleting the record would cause the NEXT record key to become too big to fit.
in all the above cases RMS leaves the record header plus (compressed) primary KEY behind

>> You end with buckets full of deleted record markers. Does turning off 'allow duplicates' affect this.

You can get those (single bytes!) for ALTERNATE KEY, SIDR entries, not for primary keys.

>> I wonder if it would help if you used 2 keys, a primary key whose values are recycled (records re-written) and the timestamp in a secondary key. You never delete records, just overwrite them.

Then why not create a circular buffer.
Those 'alternate keys' you suggest would become a header records. Record 0. That's all.

>> Robustly tracking the free list of primary key values for the next write/rewrite would be the biggest issue.

Ayup!

Hein.

Re: Indexed file as a FIFO buffer

Michael Moroney — Thu, 03 Jan 2008 18:46:03 GMT

I'm not writing anything new, just dealing with an existing application. We're not about to do a full redesign or rewrite now, just looking for tuning tricks or ideas for minor changes. Right now they have to close the file, call CONV$RECLAIM on it and reopen the file every 5000 writes.

Right now the records are read with a KGE lookup with a key of 0 to get the oldest record, if any. Would just saving this key and doing the next KGE read with it set to the last key rather than zero help?

The remote end is a PC with an Oracle database.

Attached is a FDL of an "empty" version of the file. It has grown to 3231 blocks here.

Re: Indexed file as a FIFO buffer

Hein van den Heuvel — Thu, 03 Jan 2008 20:11:08 GMT

>> I'm not writing anything new, just dealing with an existing application.

I expected as much.

>> Right now they have to close the file, call CONV$RECLAIM on it and reopen the file every 5000 writes.

That's just crazy!

>> Would just saving this key and doing the next KGE read with it set to the last key rather than zero help?

Oh yeah. You will not recognize it.
The size will not change, the need to convert will just about go away, or be reduced to daily or so.

>> Attached is a FDL of an "empty" version of the file.

Can you snarf a copy (back/ignore=interlock?) while there still are records?
Or take an empty file and plunk in 1 dummy record just to trigger ANAL/RMS to output more data (DCL write)?
Or use my tune_check!

Better still... Run the attached program against (a copy of) the empty file and share the results in an attached txt file, or in an Email to me.

Cheers,
Hein.

Re: Indexed file as a FIFO buffer

Robert Gezelter — Thu, 03 Jan 2008 21:04:10 GMT

Michael,

I can think of several ways of dealing with indexed records that would not give rise to the need to reorganize the file on an ongoing basis.

The core of the problem as you describe it is the use of the date/time as the key. There are several alternative ways of organizing this to prevent a constantly changing primary key, which is effectively the source of your problem.

Depending on how the sources are implemented, the correction could be straightforward, or it could be more involved. As my colleagues would certainly agree, speculating on the ease of modifying sources without a thorough review of those sources is not a sound way to proceed.

I would agree with Hein's original comment, re: retaining a suitably experienced consultant to review the sources and suggest changes [Disclosure: such services are within our scope of practice].

Some of the approaches that I can imagine would virtually eliminate the need to reorganize the file, period.

- Bob Gezelter, http://www.rlgsc.com

Re: Indexed file as a FIFO buffer

Jon Pinkley — Thu, 03 Jan 2008 21:18:10 GMT

If messages must be delivered FIFO, how do you guarantee FIFO order when time is changed in a backward direction in the fall (assuming you are under daylight savings/summer time rules.)

Jon

Re: Indexed file as a FIFO buffer

Michael Moroney — Thu, 03 Jan 2008 21:25:15 GMT

Hein,

I took a copy of the "empty" file and added a bunch of legitimate data and generated the FDL again.

ANALYSIS_OF_KEY 0
DATA_FILL 94
DATA_KEY_COMPRESSION 83
DATA_RECORD_COMPRESSION -4
DATA_RECORD_COUNT 849
DATA_SPACE_OCCUPIED 144
DEPTH 1
INDEX_COMPRESSION 54
INDEX_FILL 2
INDEX_SPACE_OCCUPIED 12
LEVEL1_RECORD_COUNT 12
MEAN_DATA_LENGTH 84
MEAN_INDEX_LENGTH 22
LONGEST_RECORD_LENGTH 132

Your little program produces:

$ xxx -v cadsegment.sfl2

* 3-JAN-2008 16:24:15.45 ALQ=6528 BKS=12 GBC=0 cadsegment.sfl2

Bucket VBN Count Key
------- ---------- ----- ----------------------------------
1 3 76 20080103161046130000
2 15 77 20080103161046150015
3 27 76 20080103161046170015
4 39 75 20080103161046190017
5 51 71 20080103161046210014
6 63 75 20080103161046230004
7 75 79 20080103161046240040
8 87 77 20080103161046260038
9 99 73 20080103161046280034
10 111 77 20080103161046300028
11 123 75 20080103161046320031
12 135 30 20080103161046340032

Re: Indexed file as a FIFO buffer

Hein van den Heuvel — Thu, 03 Jan 2008 21:56:28 GMT

Thanks Mike, but I must not have been clear enough. I needed the start of the experiments to be a live datafile, just before a convert/reclaim. I believe that means that just under a multiple of 5000 records lived in the file.
Next...

>> plunk in 1 dummy record

What part of "1" did you not understand :-) :-) :-).

The 1 would just trigger the right ANAL/RMS display.

What you showed looks to me as if you started on a completelly clean file and added some records, not a snapshot of how a file deteriorates in production.

So maybe you want to try that again on a live copy, not on a contrived load.
Or maybe I'm reading/interpreting this all wrong!

Anyways, I'm afraid I am out of processing quota on this topic. Back to real work!

Good luck,
Hein.

Re: Indexed file as a FIFO buffer

Michael Moroney — Thu, 03 Jan 2008 23:50:30 GMT

That is from a copy of the live file, which was empty when I grabbed it. I have to go through contortions to access it, it was copied through CONVERT/SHARE and then ZIP "-V" Definitely not a clean file!

Your program produced:

* 3-JAN-2008 18:25:15.98 ALQ=6528 BKS=12 GBC=0 cadsegment.sfl
- INIT! Primary index has not been initialized?

on the empty file so I don't know if internal RMS stuff survived both conversions so it may have looked clean.
this,

Re: Indexed file as a FIFO buffer

Hein van den Heuvel — Fri, 04 Jan 2008 00:40:15 GMT

Convert/share will create a perfectly clean file.
It does NOT copy blocks, it just creates a clean empty file, then grabs any and all records, and inserts those.

What part of "back/ignore=interlock" did you not understand ?

:-) :-) :-)

Cheers,
Hein.

Re: Indexed file as a FIFO buffer

Michael Moroney — Fri, 04 Jan 2008 01:23:12 GMT

Oh, this looks more interesting.

$ xxx -v test.sfl

* 3-JAN-2008 20:20:47.28 ALQ=3264 BKS=12 GBC=0 test.sfl

Bucket VBN Count Key
------- ---------- ----- ----------------------------------
1 375 0 20070710033049870000
2 303 0 20070710134632020000
3 447 0 20070714080525170000
4 363 0 20070715102357510000
5 315 0 20070718012305160000
6 483 0 20070718090455470000
7 351 0 20070721142250520000
8 423 0 20070722204555300000
9 195 0 20070723130144630000
10 531 0 20070723191920290000
11 555 0 20070723210454290000
12 3 0 20070726114637160000
13 339 0 20070727073923170000
14 207 0 20070727082101340000
15 279 0 20070727135630700000
16 519 0 20070727152412180000
17 75 0 20070727195434830000
18 51 0 20070728220627250000
19 27 0 20070729122805830000
20 255 0 20070729215745500000
21 39 0 20070730114345450000
22 87 0 20070801123750730000
23 111 0 20070802151803490000
24 1143 0 20070806170401670000
25 63 0 20070807150010810000
26 219 0 20070810090147360000
27 327 0 20070810091551390000
28 159 0 20070811102111500000
29 891 0 20070814004246690000
30 639 0 20070817113221980000
31 171 0 20070819081143060000
32 723 0 20070821063223960000
33 135 0 20070822220930300000
34 1887 0 20070823143547820000
35 1119 0 20070831191351690000
36 1203 0 20070831191748080000
37 1911 0 20070902223428530000
38 615 0 20070903051929580000
39 711 0 20070905161151090000
40 867 0 20070905200538460000
41 567 0 20070906184151420000
42 1815 0 20070908095952620000
43 507 0 20070908190802700000
44 267 0 20070909103551000000
45 1863 0 20070909150727260000
46 99 0 20070909160621870000
47 1131 0 20070909190244680000
48 1107 0 20070910011119960000
49 1095 0 20070910132824220000
50 15 0 20070910162315710000
51 1311 0 20070911011841630000
52 231 0 20070911133316790000
53 915 0 20070911230722950000
54 591 0 20070912003745390000
55 2067 0 20070912142258950000
56 735 0 20070912195521810000
57 243 0 20070912213229460000
58 603 0 20070913223018910000
59 1827 0 20070915035315350000
60 699 0 20070915185632850000
61 843 0 20070915225413250000
62 1635 0 20070917073026390000
63 2031 0 20070917172333430000
64 2019 0 20070917181651890000
65 2043 0 20070918064515540000
66 831 0 20070918173915300000
67 1239 0 20070918213901040000
68 939 0 20070920194216150000
69 1851 0 20070921184258380000
70 1647 0 20070921232413670000
71 2007 0 20070922052833090000
72 783 0 20070922164054130000
73 651 0 20070923004157750000
74 1971 0 20070923015927480000
75 1227 0 20070923025509980000
76 627 0 20070923173834950000
77 1071 0 20070924072745590000
78 1659 0 20070924194313980000
79 1839 0 20070925153238980000
80 1995 0 20070926141941980000
81 147 0 20070926163157980000
82 1983 0 20070926183352360000
83 1959 0 20070926213325850000
84 1251 0 20070927135621800000
85 1479 0 20070928093934190000
86 2055 0 20070928143346640000
87 1935 0 20070928180048260000
88 1083 0 20070929100533130000
89 1923 0 20070929194846670000
90 1287 0 20070930035228140000
91 1875 0 20070930102907580000
92 1299 0 20070930132322940000
93 579 0 20070930201620780000
94 1215 0 20071001105120020000
95 1191 0 20071002091946560000
96 927 0 20071003205900870000
97 759 0 20071004003748120000
98 1455 0 20071005103325710000
99 1671 0 20071005120903910000
100 1059 0 20071005151800750000
101 399 0 20071011021430270000
102 495 0 20071013001031500000
103 387 0 20071013133621940000
104 471 0 20071014213756800000
105 1947 0 20071015205815970000
106 543 0 20071018104952070000
107 951 0 20071018201516680000
108 1803 0 20071019144134520000
109 1023 0 20071023134600950000
110 855 0 20071023143856560000
111 1791 0 20071023215207790000
112 1263 0 20071024144826850000
113 183 0 20071025075008230000
114 291 0 20071026010238000000
115 1599 0 20071026161210330000
116 1167 0 20071026205738090000
117 1707 0 20071028122647370000
118 1035 0 20071029042119900000
119 1275 0 20071029043916410000
120 411 0 20071029195631180000
121 963 0 20071030111521240000
122 1503 0 20071030213452290000
123 435 0 20071031003616610000
124 1743 0 20071101051809080000
125 123 0 20071101103844460000
126 1539 0 20071101133740090000
127 1683 0 20071102144251740000
128 687 0 20071102173059360000
129 1755 0 20071102213307350000
130 2079 0 20071103111305220000
131 1719 0 20071103131507990000
132 1575 0 20071104183754670000
133 1767 0 20071105065849740000
134 459 0 20071105073658790000
135 1047 0 20071105111120010000
136 1527 0 20071105171210160000
137 795 0 20071106003626660000
138 1179 0 20071106221113370000
139 1899 0 20071108211724920000
140 1587 0 20071109112314050000
141 2512 0 20071122173225260000
142 2476 0 20071123171112090000
143 2115 0 20071126232319230000
144 2091 0 20071127130012220000
145 2632 0 20071128222711800000
146 663 0 20071128230753970000
147 999 0 20071129134747670000
148 2356 0 20071203163815650000
149 2103 0 20071204180349300000
150 2320 0 20071204192519040000
151 2332 0 20071204223834300000
152 2452 0 20071205042844240000
153 2368 0 20071205225108750000
154 2488 0 20071206123623880000
155 2164 0 20071206123844020000
156 1011 0 20071206124021830000
157 1491 0 20071206185443640000
158 675 0 20071207025151370000
159 2440 0 20071207163511680000
160 2644 0 20071209115720830000
161 2584 0 20071209180916340000
162 2296 0 20071209203023170000
163 2152 0 20071210051337330000
164 2344 0 20071212175141390000
165 2500 0 20071213174901080000
166 2572 0 20071213181025200000
167 2560 0 20071213204818540000
168 2536 0 20071214105835490000
169 2428 0 20071214204440550000
170 2416 0 20071214215101860000
171 2404 0 20071215053124990000
172 1383 0 20071217165612300000
173 1371 0 20071218083158410000
174 903 0 20071218094406550000
175 747 0 20071218100331680000
176 975 0 20071218140648150000
177 987 0 20071218213139840000
178 2596 0 20071219112629870000
179 1695 0 20071219160052270000
180 2392 0 20071219161612300000
181 2908 0 20071228090507090000
182 771 0 20080101155859950000
183 2932 38081 20080103195921850000
184 3160 42578 20080103195957580000
185 2884 57017 20080103200028240000
186 3148 49434 20080103200103090000
187 2836 29924 20080103200123510000
188 2752 38642 20080103200124170000
189 2764 18891 20080103200204290000

Re: Indexed file as a FIFO buffer

Hein van den Heuvel — Fri, 04 Jan 2008 03:35:03 GMT

Right. That's more like it.

I'm going to try a joke here, so bear with me:

>> >> Right now they have to close the file, call CONV$RECLAIM on it and reopen the file every 5000 writes.
> That's just crazy!

What part of "That's just crazy!" did you not understand.
That was your cue to the solution all along.

They did not really understand what they were dealing with and actually made matters worse, trying to hide the real problem, not thinking through the real issue.

They should have done a daily 'full' convert every day and you would never had gotten into this predicament.

Just start that now. Tonite still.

I'm quoting 'full' as there will be just a few records and the convert with take less than a second.

Those 182 buckets with a count=0 have exhausted their 65K unique record ID's and can no longer be reclaimed. The other 7 or 8 buckets can, and will be reclaimed a few more times, but eventually will become locked in time, untill a real convert is finally done!

Drop the crazy convert/reclaim code!
Replace it with a 'full' convert every 100,000 - 1,000,000 records. (10*65000)
You may or might not want to bump the bucket size a little and you'll see no more growth ever.

Oh, and just in case it is not blatently obvious, the slowness comes from reading (through) 189 empty buckets before finding a record to work on (or not). With that comes of course also 189 ENQs and 189 DEQs fro the bucket locks, which may or might not have to go to an other cluster member.
Furthermore, they probably failed to set global buffers on this file, so if it is truly actively write shared in a cluster, then the XFC can not cache the file, and most of those 180 read IOs would be real IOs.

ok?

Hein.

Re: Indexed file as a FIFO buffer

Robert Atkinson — Fri, 04 Jan 2008 09:17:48 GMT

Michael, my two-penny worth.

Although it's not a particularly elegant solution, using a queue to deliver the information would eliminate the problems you're encountering, as the Queue Manager effectively handles the key recycling for you.

If there is a vast amount of information to be sent, then write it to a plain text file, then submit that to a stopped queue.

If there's a small amount of info, then use /PARAM to store the info in the entry itself.

Your 'sender' routine can then use F$GETQUI to run through the queue entries and delete them as each piece of data is sent.

Rob.

Re: Indexed file as a FIFO buffer

Michael Moroney — Fri, 04 Jan 2008 16:39:44 GMT

>> >> Right now they have to close the file, call CONV$RECLAIM on it and reopen the file every 5000 writes.
>> That's just crazy!

>What part of "That's just crazy!" did you not understand.
>That was your cue to the solution all along.

This SW is so strange that I'm rather immune to people calling it crazy. Also I thought you were referring to the need of doing it so often, not doing it at all.

>They should have done a daily 'full' convert every day and you would never had gotten into this predicament.

>Just start that now. Tonite still.

I have a better idea. Now that I see what is going on, why not simply recreate an empty file once in a while, as long as I know the original is empty?

>Those 182 buckets with a count=0 have exhausted their 65K unique record ID's and can no longer be reclaimed. The other 7 or 8 buckets can, and will be reclaimed a few more times, but eventually will become locked in time, untill a real convert is finally done!

I'm kind of disappointed in hearing that RMS files can be clogged with unusable, unreclaimable buckets like that. After all, if you do lots of file creates/deletes on disk drives, the space occupied by the files doesn't become permanently allocated and unusuable after 65K creates/deletes. But what do I know? I don't understand the magic behind making indexed files work.

Since I recognize the time stamps it is clear why it's slow, if it searches through them all every time. The reads do a KGE access with a key of 0 every time.

As to using a stopped queue, very interesting idea. But will this clog the VMS queue database files with the same problem? (I assume they're RMS indexed files of some sort)

Re: Indexed file as a FIFO buffer

Robert Gezelter — Fri, 04 Jan 2008 17:10:49 GMT

Michael,

Commenting without having had an opportunity to see the sources is always a hazard, but the problem is generally not with RMS per se, but is more often a question of how RMS is used.

RMS Indexed files are useful tools, but by no means magic. Changes to the key fields (those that are indexed) will clutter up the index structures.

Reading your description of the programs and their processing does lead me to suspect that there are far better ways of managing the queue file, that would not impact performance, and would, in all likelihood, not require the file to ever be reorganized.

- Bob Gezelter, http://www.rlgsc.com

Re: Indexed file as a FIFO buffer

Robert Atkinson — Mon, 07 Jan 2008 08:59:53 GMT

> As to using a stopped queue, very
> interesting idea. But will this clog the
> VMS queue database files with the same
> problem? (I assume they're RMS indexed
> files of some sort)

NO, because the queue manager will reuse the entry numbers, which is like reusing the empty buckets in your case.

As you will only have a few entries at any point in time, i.e. not thousands, you won't see any performance impact.

Rob.

Re: Indexed file as a FIFO buffer

Willem Grooters — Mon, 07 Jan 2008 13:40:54 GMT

To my experience, the way that RMS indexed files are organized and how RMS handles deletions 9and updates) is hardly known with most programmers that were educated with either flat files and (relational) databases. Some system managers seem to forget that Indexed sequential files need maintanance, the more updates (including deletes) are applied, the more often: CONVERT, after re-calculation of bucket sizes and buffers...
Looking at the dump you gave on Hein's request, I have the impression the file is HIGHLY fragmented - internally. If you have to walk the index buckets, it would mean re-reading data since your index buckets seem to be scattered over the disk - smashing performance to bits. I dealt with such problems in a not too distant past.

Try this:

$ pipe dum/header/blocks=(count:0) (your file)|searc/match=and sys$pipe "Count:", "LBN:"

and count the number of lines. There should be as few as possible.

If there are many, see if theer is a pattern that matches butcket sizes in this file's FDL.