EDIT/FDL usage

[Art Wiens]

In a previous thread I asked for some help with some details regarding $STATUS to create command procedures to CONVERT our data files.

Now ... getting closer to the problem I'm trying to solve, the files in the application haven't been CONVERTed for several years now (the Windows/Unix solution that was supposed to replace it all failed ;-) hence the need for an updated procedure. There are ~200 files per division. I have produced .FDL's for them and wanted to take Hein's suggestion about optimizing them.

Finally my question: I want to run another command procedure to do the optimization but it doesn't want to do multiple EDIT/FDL's in the command procedure. It does the first one and exits. No error message or such.

The command I'm trying to use (multiple times) is similar to:

$ edit/fdl/analysis=[-.cirfdl]addrmas.fdl/nointeractive/output=[-.cirfdl]addrmas_opti.fdl [-.cirfdl]addrmas.fdl

I really don't fancy doing this by hand ~200 times for 10 divisions.

What DCL "trick" am I missing?

Thanks in advance,
Art

ps. any CHARON-VAX attendees next week in Nashua besides me? I hope you guys left the Sheraton standing! ;-)

Cheers.

[Eberhard Wacker]

Hi Art,
whatever the problem is, a $ SET NOON (disable all warning/error handling) at the beginning of the procedure should avoid a dropout of it.
Cheers,
EW

[labadie_1]

I have a procedure from Decus US, about 1990, that works fine (and has worked fine for years), doing this.

So it is doable :-)

I will try to find it and post it, if you are interested.

[Hein van den Heuvel · Accepted solution]

>> it doesn't want to do multiple EDIT/FDL's in the command procedure. It does the first one and exits. No error message or such.

That's not normal. Something must be happenign, something must be signalled. It is as if the /nointer is not functioning in which case edit/fdl will fail when not run from a terminal.
Tried the usual? $set veri, show symb $status and such?

$ edit/fdl/analysis

Good to see you use the /ANAL secondary input. I like that to seperate the basic layout which will not changes from the stats that will change (a little over time).

Now the actual lines used in the /anal file are but a few. You may want to reduce those FDLs to just that, and manual edit them for new anticipated record counts ( or target disk cluster sizes) based on your application understanding.

Below you'll find a script that you might want to use as a stating point to get a series of fdl and ana files going.

Once you've done this basic tune up, and the converts, then you may want to try my 'rms_tune_check' on the file.
Btw... I updated that rms_tune_check program last week, and will try to get the freeware updated real-soon-now.

>> ps. any CHARON-VAX attendees next week in Nashua besides me?

I'll be in Nashua, and maybe in the Tara :-)

Hein.

$! ---- optimize-fdl.com --- Hein van den Heuvel --- June 2005
$if p1.eqs."" then $exit
$last = ""
$loop:
$file = f$search(p1)
$if file .eqs. last then exit
$if file .eqs. "" then exit
$if f$file(file,"ORG").nes."IDX" then goto loop
$write sys$output "processing ''file'"
$last = file
$name = f$parse(file,,,"NAME")
$anal/rms/fdl/out='name'.fdl_orig 'file
$open/erro=oops fdl 'name'.fdl_orig
$crea 'name'.ana
$open/erro=oops/append ana 'name'.ana
$read/end=oops fdl record
$writ ana f$extr(0,28,record) + -
"minimal EDIT/FDL/NOINTER analyze file"""
$writ ana "FILE; CLUSTER_SIZE 12"
$writ ana " NAME """ + file + """"
$writ ana "ANALYSIS_OF_KEY 0"
$read_loop:
$read/end=read_done fdl record
$if f$loc("DATA_RECORD_COUNT",record).ne.f$len(record) then -
write ana record
$if f$loc("MEAN_DATA_LENGTH",record).eq.f$len(record) then goto read_loop
$write ana record
$close ana
$close fdl
$edit/fdl/nointer/output='name'.fdl/anal='name'.ana 'name'.fdl_orig
$goto loop
$oops:
$stat = $status
$close/nolog ana
$close/nolog fdl
$write sys$output "Oops!"
$exit 'stat

[Hein van den Heuvel]

I had an excuse to add a little more functionality to the script.
It now detects protential duplicate key problems.

Sample usage:

$ @OPTIMIZE_FDL.COM order_file.dat
processing U$1:[HEIN.TMP]ORDER_FILE.DAT;1
key=0 area=0 bks=12 space=1332 l1=111 buckets=111 missing=0
key=1 area=0 bks=12 space=120 l1=4 buckets=10 missing=6
** DUPLICATES on key 1 ? 6 buckets out of 120 have no index !!
key=2 area=0 bks=12 space=96 l1=8 buckets=8 missing=0
$ conve/stat/fast order_file.dat/fdl=order_file order_file.new
Total Records Processed: 5001
:
$ @OPTIMIZE_FDL.COM order_file.new
processing U$1:[HEIN.TMP]ORDER_FILE.NEW;1
key=0 area=0 bks=12 space=1224 l1=102 buckets=102 missing=0
key=1 area=2 bks=12 space=84 l1=3 buckets=7 missing=4
** DUPLICATES on key 1 ? 4 buckets out of 84 have no index !!
key=2 area=2 bks=12 space=108 l1=9 buckets=9 missing=0
$

New script is attached.

Hein.

[Art Wiens]

Somehow I missed the fact that some of FDL's I was trying to optimize weren't there. I just tried it again and it clearly gives me a :

%FDL-F-OPENFDL, error opening xxxxx;
-RMS-E-FNF, file not found

I know I tried this several times, not sure why I couldn't see the error message...the problem was most likely between the chair and the keyboard!

On to the next topic to do with the actual "optimization".

If the original file says this:

AREA 0
ALLOCATION 304006
BEST_TRY_CONTIGUOUS yes
BUCKET_SIZE 18
EXTENSION 14461

AREA 1
ALLOCATION 2120
BEST_TRY_CONTIGUOUS yes
BUCKET_SIZE 18
EXTENSION 216

AREA 2
ALLOCATION 248256
BEST_TRY_CONTIGUOUS yes
BUCKET_SIZE 18
EXTENSION 62334

And after the EDIT/FDL it says this:

AREA 0
ALLOCATION 163769
BEST_TRY_CONTIGUOUS yes
BUCKET_SIZE 12
EXTENSION 40920

AREA 1
ALLOCATION 4092
BEST_TRY_CONTIGUOUS yes
BUCKET_SIZE 12
EXTENSION 4092

AREA 2
ALLOCATION 316789
BEST_TRY_CONTIGUOUS yes
BUCKET_SIZE 6
EXTENSION 65535

Is this better?

RMS novice,
Art

[Hein van den Heuvel]

>> Somehow I missed the fact that some of FDL's I was trying to optimize weren't there --> %FDL-F-OPENFDL, error opening xxxxx;

The easiest explanation for that is the file being accessed when ANAL/RMS was attempted. It needs exclusive access.

AREA 0
ALLOCATION 304006
BUCKET_SIZE 18
EXTENSION 14461
:
And after the EDIT/FDL it says this:

AREA 0
ALLOCATION 163769
BUCKET_SIZE 12
EXTENSION 40920

>>> Is this better?

'it depends'.
It is certain to be reasonable for the information provided.
That is:
- the root level will not be too high
- the initial data will fit in the allocated space.
EDIT/FDL uses a 1/4 allocation for the extent, limited by 65K (system max).
It seems that the original file was larger then strictly needed to hold the data.
This may be due to an overdue convert, or it could be symptomatic for the data access patterns. In the latter case the new file is bound to quickly grow to approx the old size. THe generous extent will make sure the growth will not be too painfull if and when it is needed.

Cheers!
Hein.