Operating System - OpenVMS
1748144 Members
3445 Online
108758 Solutions
New Discussion юеВ

Re: Indexed file as a FIFO buffer

 
SOLVED
Go to solution
Hein van den Heuvel
Honored Contributor

Re: Indexed file as a FIFO buffer

Thanks Mike, but I must not have been clear enough. I needed the start of the experiments to be a live datafile, just before a convert/reclaim. I believe that means that just under a multiple of 5000 records lived in the file.
Next...

>> plunk in 1 dummy record

What part of "1" did you not understand :-) :-) :-).

The 1 would just trigger the right ANAL/RMS display.

What you showed looks to me as if you started on a completelly clean file and added some records, not a snapshot of how a file deteriorates in production.

So maybe you want to try that again on a live copy, not on a contrived load.
Or maybe I'm reading/interpreting this all wrong!

Anyways, I'm afraid I am out of processing quota on this topic. Back to real work!

Good luck,
Hein.
Michael Moroney
Frequent Advisor

Re: Indexed file as a FIFO buffer

That is from a copy of the live file, which was empty when I grabbed it. I have to go through contortions to access it, it was copied through CONVERT/SHARE and then ZIP "-V" Definitely not a clean file!

Your program produced:

* 3-JAN-2008 18:25:15.98 ALQ=6528 BKS=12 GBC=0 cadsegment.sfl
- INIT! Primary index has not been initialized?

on the empty file so I don't know if internal RMS stuff survived both conversions so it may have looked clean.
this,
Hein van den Heuvel
Honored Contributor

Re: Indexed file as a FIFO buffer

Convert/share will create a perfectly clean file.
It does NOT copy blocks, it just creates a clean empty file, then grabs any and all records, and inserts those.

What part of "back/ignore=interlock" did you not understand ?

:-) :-) :-)


Cheers,
Hein.
Michael Moroney
Frequent Advisor

Re: Indexed file as a FIFO buffer

Oh, this looks more interesting.

$ xxx -v test.sfl

* 3-JAN-2008 20:20:47.28 ALQ=3264 BKS=12 GBC=0 test.sfl

Bucket VBN Count Key
------- ---------- ----- ----------------------------------
1 375 0 20070710033049870000
2 303 0 20070710134632020000
3 447 0 20070714080525170000
4 363 0 20070715102357510000
5 315 0 20070718012305160000
6 483 0 20070718090455470000
7 351 0 20070721142250520000
8 423 0 20070722204555300000
9 195 0 20070723130144630000
10 531 0 20070723191920290000
11 555 0 20070723210454290000
12 3 0 20070726114637160000
13 339 0 20070727073923170000
14 207 0 20070727082101340000
15 279 0 20070727135630700000
16 519 0 20070727152412180000
17 75 0 20070727195434830000
18 51 0 20070728220627250000
19 27 0 20070729122805830000
20 255 0 20070729215745500000
21 39 0 20070730114345450000
22 87 0 20070801123750730000
23 111 0 20070802151803490000
24 1143 0 20070806170401670000
25 63 0 20070807150010810000
26 219 0 20070810090147360000
27 327 0 20070810091551390000
28 159 0 20070811102111500000
29 891 0 20070814004246690000
30 639 0 20070817113221980000
31 171 0 20070819081143060000
32 723 0 20070821063223960000
33 135 0 20070822220930300000
34 1887 0 20070823143547820000
35 1119 0 20070831191351690000
36 1203 0 20070831191748080000
37 1911 0 20070902223428530000
38 615 0 20070903051929580000
39 711 0 20070905161151090000
40 867 0 20070905200538460000
41 567 0 20070906184151420000
42 1815 0 20070908095952620000
43 507 0 20070908190802700000
44 267 0 20070909103551000000
45 1863 0 20070909150727260000
46 99 0 20070909160621870000
47 1131 0 20070909190244680000
48 1107 0 20070910011119960000
49 1095 0 20070910132824220000
50 15 0 20070910162315710000
51 1311 0 20070911011841630000
52 231 0 20070911133316790000
53 915 0 20070911230722950000
54 591 0 20070912003745390000
55 2067 0 20070912142258950000
56 735 0 20070912195521810000
57 243 0 20070912213229460000
58 603 0 20070913223018910000
59 1827 0 20070915035315350000
60 699 0 20070915185632850000
61 843 0 20070915225413250000
62 1635 0 20070917073026390000
63 2031 0 20070917172333430000
64 2019 0 20070917181651890000
65 2043 0 20070918064515540000
66 831 0 20070918173915300000
67 1239 0 20070918213901040000
68 939 0 20070920194216150000
69 1851 0 20070921184258380000
70 1647 0 20070921232413670000
71 2007 0 20070922052833090000
72 783 0 20070922164054130000
73 651 0 20070923004157750000
74 1971 0 20070923015927480000
75 1227 0 20070923025509980000
76 627 0 20070923173834950000
77 1071 0 20070924072745590000
78 1659 0 20070924194313980000
79 1839 0 20070925153238980000
80 1995 0 20070926141941980000
81 147 0 20070926163157980000
82 1983 0 20070926183352360000
83 1959 0 20070926213325850000
84 1251 0 20070927135621800000
85 1479 0 20070928093934190000
86 2055 0 20070928143346640000
87 1935 0 20070928180048260000
88 1083 0 20070929100533130000
89 1923 0 20070929194846670000
90 1287 0 20070930035228140000
91 1875 0 20070930102907580000
92 1299 0 20070930132322940000
93 579 0 20070930201620780000
94 1215 0 20071001105120020000
95 1191 0 20071002091946560000
96 927 0 20071003205900870000
97 759 0 20071004003748120000
98 1455 0 20071005103325710000
99 1671 0 20071005120903910000
100 1059 0 20071005151800750000
101 399 0 20071011021430270000
102 495 0 20071013001031500000
103 387 0 20071013133621940000
104 471 0 20071014213756800000
105 1947 0 20071015205815970000
106 543 0 20071018104952070000
107 951 0 20071018201516680000
108 1803 0 20071019144134520000
109 1023 0 20071023134600950000
110 855 0 20071023143856560000
111 1791 0 20071023215207790000
112 1263 0 20071024144826850000
113 183 0 20071025075008230000
114 291 0 20071026010238000000
115 1599 0 20071026161210330000
116 1167 0 20071026205738090000
117 1707 0 20071028122647370000
118 1035 0 20071029042119900000
119 1275 0 20071029043916410000
120 411 0 20071029195631180000
121 963 0 20071030111521240000
122 1503 0 20071030213452290000
123 435 0 20071031003616610000
124 1743 0 20071101051809080000
125 123 0 20071101103844460000
126 1539 0 20071101133740090000
127 1683 0 20071102144251740000
128 687 0 20071102173059360000
129 1755 0 20071102213307350000
130 2079 0 20071103111305220000
131 1719 0 20071103131507990000
132 1575 0 20071104183754670000
133 1767 0 20071105065849740000
134 459 0 20071105073658790000
135 1047 0 20071105111120010000
136 1527 0 20071105171210160000
137 795 0 20071106003626660000
138 1179 0 20071106221113370000
139 1899 0 20071108211724920000
140 1587 0 20071109112314050000
141 2512 0 20071122173225260000
142 2476 0 20071123171112090000
143 2115 0 20071126232319230000
144 2091 0 20071127130012220000
145 2632 0 20071128222711800000
146 663 0 20071128230753970000
147 999 0 20071129134747670000
148 2356 0 20071203163815650000
149 2103 0 20071204180349300000
150 2320 0 20071204192519040000
151 2332 0 20071204223834300000
152 2452 0 20071205042844240000
153 2368 0 20071205225108750000
154 2488 0 20071206123623880000
155 2164 0 20071206123844020000
156 1011 0 20071206124021830000
157 1491 0 20071206185443640000
158 675 0 20071207025151370000
159 2440 0 20071207163511680000
160 2644 0 20071209115720830000
161 2584 0 20071209180916340000
162 2296 0 20071209203023170000
163 2152 0 20071210051337330000
164 2344 0 20071212175141390000
165 2500 0 20071213174901080000
166 2572 0 20071213181025200000
167 2560 0 20071213204818540000
168 2536 0 20071214105835490000
169 2428 0 20071214204440550000
170 2416 0 20071214215101860000
171 2404 0 20071215053124990000
172 1383 0 20071217165612300000
173 1371 0 20071218083158410000
174 903 0 20071218094406550000
175 747 0 20071218100331680000
176 975 0 20071218140648150000
177 987 0 20071218213139840000
178 2596 0 20071219112629870000
179 1695 0 20071219160052270000
180 2392 0 20071219161612300000
181 2908 0 20071228090507090000
182 771 0 20080101155859950000
183 2932 38081 20080103195921850000
184 3160 42578 20080103195957580000
185 2884 57017 20080103200028240000
186 3148 49434 20080103200103090000
187 2836 29924 20080103200123510000
188 2752 38642 20080103200124170000
189 2764 18891 20080103200204290000
Hein van den Heuvel
Honored Contributor
Solution

Re: Indexed file as a FIFO buffer

Right. That's more like it.

I'm going to try a joke here, so bear with me:

>> >> Right now they have to close the file, call CONV$RECLAIM on it and reopen the file every 5000 writes.
> That's just crazy!

What part of "That's just crazy!" did you not understand.
That was your cue to the solution all along.

They did not really understand what they were dealing with and actually made matters worse, trying to hide the real problem, not thinking through the real issue.

They should have done a daily 'full' convert every day and you would never had gotten into this predicament.

Just start that now. Tonite still.

I'm quoting 'full' as there will be just a few records and the convert with take less than a second.

Those 182 buckets with a count=0 have exhausted their 65K unique record ID's and can no longer be reclaimed. The other 7 or 8 buckets can, and will be reclaimed a few more times, but eventually will become locked in time, untill a real convert is finally done!

Drop the crazy convert/reclaim code!
Replace it with a 'full' convert every 100,000 - 1,000,000 records. (10*65000)
You may or might not want to bump the bucket size a little and you'll see no more growth ever.

Oh, and just in case it is not blatently obvious, the slowness comes from reading (through) 189 empty buckets before finding a record to work on (or not). With that comes of course also 189 ENQs and 189 DEQs fro the bucket locks, which may or might not have to go to an other cluster member.
Furthermore, they probably failed to set global buffers on this file, so if it is truly actively write shared in a cluster, then the XFC can not cache the file, and most of those 180 read IOs would be real IOs.

ok?

Hein.

Robert Atkinson
Respected Contributor

Re: Indexed file as a FIFO buffer

Michael, my two-penny worth.

Although it's not a particularly elegant solution, using a queue to deliver the information would eliminate the problems you're encountering, as the Queue Manager effectively handles the key recycling for you.

If there is a vast amount of information to be sent, then write it to a plain text file, then submit that to a stopped queue.

If there's a small amount of info, then use /PARAM to store the info in the entry itself.

Your 'sender' routine can then use F$GETQUI to run through the queue entries and delete them as each piece of data is sent.

Rob.
Michael Moroney
Frequent Advisor

Re: Indexed file as a FIFO buffer

>> >> Right now they have to close the file, call CONV$RECLAIM on it and reopen the file every 5000 writes.
>> That's just crazy!

>What part of "That's just crazy!" did you not understand.
>That was your cue to the solution all along.

This SW is so strange that I'm rather immune to people calling it crazy. Also I thought you were referring to the need of doing it so often, not doing it at all.

>They should have done a daily 'full' convert every day and you would never had gotten into this predicament.

>Just start that now. Tonite still.

I have a better idea. Now that I see what is going on, why not simply recreate an empty file once in a while, as long as I know the original is empty?

>Those 182 buckets with a count=0 have exhausted their 65K unique record ID's and can no longer be reclaimed. The other 7 or 8 buckets can, and will be reclaimed a few more times, but eventually will become locked in time, untill a real convert is finally done!

I'm kind of disappointed in hearing that RMS files can be clogged with unusable, unreclaimable buckets like that. After all, if you do lots of file creates/deletes on disk drives, the space occupied by the files doesn't become permanently allocated and unusuable after 65K creates/deletes. But what do I know? I don't understand the magic behind making indexed files work.

Since I recognize the time stamps it is clear why it's slow, if it searches through them all every time. The reads do a KGE access with a key of 0 every time.

As to using a stopped queue, very interesting idea. But will this clog the VMS queue database files with the same problem? (I assume they're RMS indexed files of some sort)
Robert Gezelter
Honored Contributor

Re: Indexed file as a FIFO buffer

Michael,

Commenting without having had an opportunity to see the sources is always a hazard, but the problem is generally not with RMS per se, but is more often a question of how RMS is used.

RMS Indexed files are useful tools, but by no means magic. Changes to the key fields (those that are indexed) will clutter up the index structures.

Reading your description of the programs and their processing does lead me to suspect that there are far better ways of managing the queue file, that would not impact performance, and would, in all likelihood, not require the file to ever be reorganized.

- Bob Gezelter, http://www.rlgsc.com
Robert Atkinson
Respected Contributor

Re: Indexed file as a FIFO buffer

> As to using a stopped queue, very
> interesting idea. But will this clog the
> VMS queue database files with the same
> problem? (I assume they're RMS indexed
> files of some sort)

NO, because the queue manager will reuse the entry numbers, which is like reusing the empty buckets in your case.

As you will only have a few entries at any point in time, i.e. not thousands, you won't see any performance impact.

Rob.
Willem Grooters
Honored Contributor

Re: Indexed file as a FIFO buffer

To my experience, the way that RMS indexed files are organized and how RMS handles deletions 9and updates) is hardly known with most programmers that were educated with either flat files and (relational) databases. Some system managers seem to forget that Indexed sequential files need maintanance, the more updates (including deletes) are applied, the more often: CONVERT, after re-calculation of bucket sizes and buffers...
Looking at the dump you gave on Hein's request, I have the impression the file is HIGHLY fragmented - internally. If you have to walk the index buckets, it would mean re-reading data since your index buckets seem to be scattered over the disk - smashing performance to bits. I dealt with such problems in a not too distant past.

Try this:

$ pipe dum/header/blocks=(count:0) (your file)|searc/match=and sys$pipe "Count:", "LBN:"

and count the number of lines. There should be as few as possible.

If there are many, see if theer is a pattern that matches butcket sizes in this file's FDL.
Willem Grooters
OpenVMS Developer & System Manager