Skipping Comments in DCL Scripts

[Jack Trachtman]

I've noticed that some people whose DCL scripts have comments at the top, will have as a first line a GOTO to a label immediately past the end of the comments.

Since the DCL interpreter has to look at each line anyway in search of the label, does this technique provide any real efficiency?

[labadie_1]

I have heard (but never met) about problems with "big" dcl code, which could not "see" a label too far, so this could explain some goto, at the beginning.

For me, a dcl code should not be bigger than 20 or 30 lines, or you should switch to perl, awk, python, a program in C / Pascal/ Fortran/ Dibol/ pick your favorite langage

:-)

[Antoniov.]

Hi Jack,
you can write a little DCL script to remove all remarks.
$ OPEN/READ SRC $ OPEN/WRITE TGT $MLOOP:
$ READ SRC MYLINE/END=XLOOP
$ MYLINE=F$EDIT(MYLINE,"UNCOMMENT,TRIM")
$ IF MYLINE.NES."$!"
$ THEN
$ WRITE TGT MYLINE
$ ENDIF
$ GOTO MLOOP
$XLOOP:
$ CLOSE TGT
$ CLOSE SRC
$ EXIT

@Antoniov

[Craig A Berry]

I suspect that the GOTO skips full parsing and interpretation of the intervening lines. Whether that helps with comments or not, I don't know for sure, but I have also seen comments placed at the end of a command file. Search google for Charlie Hammond's DCL_DIET, which strips the comments out.

[Ian Miller.]

for DCL_DIET see
http://dcl.openvms.org/stories.php?story=03/06/05/8816530

and also
http://dcl.openvms.org/stories.php?story=03/06/05/6066644

I suspect DCL labels are found in a optimized fashion which is quicker than the usual comment skipping (which has to look for things like $!'f$verify(0)

[Uwe Zessin]

Jack,
I haven't seen a discussion about this for quite some time, but as far as I can recall you are right: it will not speed up the script itself because the intervening lines, as Ian already wrote, need to be processed anyway.

However, in a previous life we, too, had comments at the top of the DCL procedures. We uses GOTOs to prevent these comment blocks from being displayed when the procedure was running with SET VERIFY.

I know about:
$ V = F$VERIFY(0)
$! useful comment
$ V = F$VERIFY(V)

but the 'internal standard' was to use GOTOs, so what.

[Jan van den Ende]

Well,

firstly, I am one of those that DID meet Gerard's limit.
Enlarging SYSGEN PIOPAGES helps, but the max is not THAT high. (Come to think about it,
ENGENEERING, memory nowadays ain't so scarce/expensive anymore, pump up that limit, so those who want to spend the mems that way can do it!!)

We have an internal standard that prescribes EVERY modification to be accounted for in the header (if possible a 1-liner, but at least indicate THAT someone did something, WHO did it, WHEN, preferably WHY, and a pointer to the change).
If that shows up in EVERY batch logfile, it quickly becomes annoying.
Some of us started using a $ GOTO PAST_COMMENTS, others a $ SAVE_VER=F$VERIFY(0) and after the comments $ dummy=F$VERIFY('SAVE_VER'), and both equally well prevent the annoyance, and hence, increase 'human' performance. As for machine performance, I don't think the differences are even measurable (except again the log-file sizes)

Me thinks, mostly a matter of style.

fwiw, ymmv

Jan

[Jack Trachtman]

Looks like everyone is in agreement that the GOTO technique doesn't save much of anything (I like the (obvious now) idea of using this approach to skip displaying comments in batch jobs).

But...

I had just remembered something (that probably appeared in comp.os.vms)

Create the following script:

$! 'F$VERIFY(0)'
$ SHOW TIME

Then type SET VERIFY and run the script.
Now try the same test but remove the apostrophes in the comment line.

This seems to indicate that DCL is doing more than is implied in the manual.

[Ian Miller.]

I think
$!'F$VERIFY(0)
is handled as a special case than the normal ignoring what is in a comment. I think its a historical feature.

[Hein van den Heuvel]

IMHO you are all thinking way to too deep.

The goto will simply skip long comments when run with SET VERIFY.

It is only of academic interest whether it has an impact on speed as 'it all depends'.
If you really care about speed, use a compiled language or perl to do you dcl stuff.


fwiw,
Hein.

[Martin P.J. Zinser]

Hello,

as to the long-comments to provide change
histroy, we do have this requirement too for our production software. Another easy way to avoid these comments cluttering your logfiles is simply to put them at the end of the DCL script (after $ exit) ;-)

Greetings, Martin

[Robert Atkinson]

Talking about DCL optimisation, is there any quick win way to speed DCL up?

I have a set of scripts that are written to dynamically handle database files, and they work very well except for they are slooooowwww.....something like a hundred records a minute as opposed to 5-10k.

This is because of all of the DCL going on. I can optimise the code (remove comments, unrequired characters, etc) but to make them usable a severe hike in performance is required.

This is one of the routines that needs speeding up :-

ALPHA_ROB$ typ UTL_DCF_READFILE.COM
$ ! "UTL_DCF_READFILE.COM"
$ VERIFY = 'F$VERIFY(0)'
$!
$!------------------------------------------------------------------------------
$! AUTHOR: Robert Atkinson - September 2001
$!------------------------------------------------------------------------------
$! DESCRIPTION: DCF routine for reading records from a file.
$! Records are read and build into fields called DCF$areaname_fieldname
$!
$! NOTE: This routine can currently only handle sequential files
$!
$!------------------------------------------------------------------------------
$! RUN FREQUENCY : As required
$! PRERUN REQUISITES : File has been opened using FILE%OPEN
$! RERUN PROCEDURE : None
$! SPECIAL NOTE :
$! USAGE : FILE%READ ERRORLOG
$! PARAMS : P1 = FILENAME (e.g. ERRORLOG)
$! P2 = OPTION (""/TRIM)
$!------------------------------------------------------------------------------
$! AMENDMENT LOG:
$! Date Oper Reason
$! 21-01-2002 RAA Code could not cope with embedded quotes in data.
$! 22-01-2002 RAA Code for building field using F$EXT had a bug.
$! 30-10-2002 RAA Add option to allow fields to be trimmed
$!------------------------------------------------------------------------------
$!
$ SET ON
$ ON ERROR THEN GOTO ERROR
$ !
$ FILENAME = "''P1'"
$ OPTION = "''P2'"
$ !
$ DCF$ACTION == "FALSE"
$ !
$START:
$ IF F$TRNLNM("DCL$DEBUG") .OR. F$TRNLNM("DCF$DEBUG")
$ THEN
$ DEBUG = "SH SYMBOL"
$ ELSE
$ DEBUG = "! "
$ ENDIF
$ !
$ IF F$TRNLNM("DCL$VERIFY") .OR. F$TRNLNM("DCF$VERIFY") THEN SET VERI
$ !
$READ_FILE:
$ READ /END=END_OF_FILE DCF$$'FILENAME' INDATA
$ !
$ AREANAME = DCF$'FILENAME'$AREANAME
$ !
$ DEBUG AREANAME
$ !
$BUILD_FIELDS:
$ COUNT = 0
$ !
$BUILD_FIELDS_LOOP:
$ IF F$TYPE(DCF$'AREANAME'$FIELD'COUNT') .EQS. "" THEN GOTO END_BUILD_FIELDS_LOOP
$ !
$ FIELDNAME = DCF$'AREANAME'$FIELD'COUNT'
$ FIELDSTART = F$ELEM(0,",",DCF$'AREANAME'_'FIELDNAME'_POS)
$ FIELDLEN = F$ELEM(1,",",DCF$'AREANAME'_'FIELDNAME'_POS)
$ FIELD = F$EXT(FIELDSTART,FIELDLEN,INDATA)
$ !
$ DCF_AREANAME_TMP = "DCF$''AREANAME'_''FIELDNAME'_POS"
$ DCF_AREASIZE_TMP = F$ELEM(1,",",'DCF_AREANAME_TMP')
$ DEBUG DCF_*_TMP
$ DCF$'AREANAME'_'FIELDNAME' == F$EXT(0,'DCF_AREASIZE_TMP',FIELD)
$ DEBUG DCF$'AREANAME'_'FIELDNAME'
$ !
$ IF OPTION .EQS. "TRIM"
$ THEN
$ DCF$'AREANAME'_'FIELDNAME' == F$EDIT(DCF$'AREANAME'_'FIELDNAME',"TRIM")
$ DEBUG DCF$'AREANAME'_'FIELDNAME'
$ ENDIF
$ !
$ COUNT = COUNT + 1
$ !
$ GOTO BUILD_FIELDS_LOOP
$ !
$END_BUILD_FIELDS_LOOP:
$ !
$ GOTO END
$ !


$END_OF_FILE:
$ DCF$ACTION == "TRUE"
$ !
$ GOTO END
$ !



$END:
$ DEBUG DCF$ACTION
$ !
$ VERIFY = 'F$VERIFY(VERIFY)'
$ !
$ EXIT %X00001
$ !
$ERROR:
$ DCF$ACTION == "TRUE"
$ !
$ VERIFY = 'F$VERIFY(VERIFY)'
$ !
$ EXIT %X00004
$ !
$ !
$ !

Robert.

[Ian Miller.]

It may be worth using
$ SET PREFIX (!12%T)
then running with verify and output to a log file to see which statement takes most elapsed time.

However it may be that you are best either running it less often.

I suspect other people here will point out alternative languages but the practicality of re-implementation in another language depends on your environment (what alternative language is available) and people(is there somebody to re-write it and support the result).

[Antoniov.]

Robert,
I can't see any optimitation over your code. I think you will gain a very little bit removing comments: no good idea.
Your problem, I think, DCL procedure require a lot memory to manage DCL variable. Analyze process statistics after process ended: you could find some clue.

@Antoniov

[Jan van den Ende · Accepted solution]

Robert,

let me first of all state what I -- THINK -- your procedure is doing, and if, -ONLY IF- that is correct, I may have some suggestions.

It looks like you are processing an inputfile, this routine is called for each record in an input file. After processing a record, you exit success.
This routine is activated 5K - 10K times.
File-opening is done outside this loop, and so is file-closing.

As I see it WITHIN your procedure, EACH TIME
you activate

$ FIELDNAME = DCF$'AREANAME'$FIELD'COUNT'
$ FIELDSTART = F$ELEM(0,",",DCF$'AREANAME'_'FIELDNAME'_POS)
$ FIELDLEN = F$ELEM(1,",",DCF$'AREANAME'_'FIELDNAME'_POS)
$ FIELD = F$EXT(FIELDSTART,FIELDLEN,INDATA)
$ !
$ DCF_AREANAME_TMP = "DCF$''AREANAME'_''FIELDNAME'_POS"
$ DCF_AREASIZE_TMP = F$ELEM(1,",",'DCF_AREANAME_TMP')
$ DEBUG DCF_*_TMP
$ DCF$'AREANAME'_'FIELDNAME' == F$EXT(0,'DCF_AREASIZE_TMP',FIELD)

This means, OUTSIDE the procedure you have defined for each AREANAME-FIELDNAME combination, a record DCF$'areaname'_'fieldname'_POS with comma-separated POS & LENGTH values.
And then, for EACH pass, you perform a number of operations thereon.

Suppose, instead, OUTSIDE the procedure you
define the _POS symbols to be JUST the position, and define corresponding _LEN symbols with the lengths.

Then this whole block gets replaced by a 1-liner, with a lot less functioncalls:

$ DFC$'areaname'_'fieldname' == f$extract(dfc$'areaname'_'fieldname'_POS,dfc$'areaname'_fieldname'_LEN,INDATA)

( I guess this will wrap, correct it back!)

This means you replace (if I counted correct) 5 lexical invocations by 1, and 7 symbol assignments by 1.

If you are gonna try it, please let us know IF, and if, then HOW MUCH, difference it made?


hth,


Jan

[Robert Atkinson]

Jan - thanks for your response. That was certainly something I hadn't considered.

I trimmed the code down to this :-

$BUILD_FIELDS_LOOP:
$ IF F$TYPE(DCF$'AREANAME'$FIELD'COUNT') .EQS. "" THEN GOTO END_BUILD_FIELDS_LOOP
$ FIELDNAME = DCF$'AREANAME'$FIELD'COUNT'
$ FIELD = F$EXT(DCF$'AREANAME'_'FIELDNAME'_POS,DCF$'AREANAME'_'FIELDNAME'_LEN,INDATA)
$ DCF$'AREANAME'_'FIELDNAME' == F$EXT(0,DCF$'AREANAME'_'FIELDNAME'_LEN,FIELD)
$ IF OPTION .EQS. "TRIM" THEN DCF$'AREANAME'_'FIELDNAME' == F$EDIT(DCF$'AREANAME'_'FIELDNAME',"TRIM")
$ COUNT = COUNT + 1
$ GOTO BUILD_FIELDS_LOOP
$ !
$END_BUILD_FIELDS_LOOP:

Unfortunately, the test time went from around 5.5 seconds to 4 seconds. If I ran this just using DCL commands it's less then half a second, so I still need to get rid of a lot of excess.

Rob.

[Antoniov.]

Robert,
I think Jna hint you store value start and len into variable outside the main loop such as:
$ FIELDSTART_01 = F$ELEM(0,",", DCF$'AREANAME'_'FIELDNAME'_POS)
$ FIELDLEN_01 = F$ELEM(1,",", DCF$'AREANAME'_'FIELDNAME'_POS)
Then inside loop
$ FIELD = F$EXT(FIELDSTART_01,FIELDLEN_01, INDATA)
This to avoid interpretation time to parsing FIELDSTART and FIELDLEN values.
But unfortunatly, if I understand, Robert doesn't know how many field are in record, so code seems difficult to change.

The Jan'idea may be good: you can reduce parsing line inside main loop to run quicky but you code become difficult to read.
Try also using hyphen at end of line to concatenate line; You'll got a long line but parsing can go quicly.
Also remove $! inside main loop.

@Antoniov

[Hein van den Heuvel]

While I am all for long and descriptive field names in general, in a tight loop they may not be called for:

$ FIELDNAME = DCF$'AREANAME'$FIELD'COUNT'
$ FIELD = F$EXT(DCF$'AREANAME'_'FIELDNAME'_POS,DCF$'AREANAME'_'FIELDNAME'_LEN,INDATA)
$ DCF$'AREANAME'_'FIELDNAME' == F$EXT(0,DCF$'AREANAME'_'FIELDNAME'_LEN,FIELD)


$ A = AREANAME
:
$ N = DCF$'A'$FIELD'COUNT'
$ F = F$EXT(DCF$'A'_'N'_POS, DCF$'A'_'N'_LEN, INDATA)
$ DCF$'A'_'N' == F$EXT (0, DCF$'A'_'F'_LEN, F)

And yes, like Jan wrote.. anything fixed that you can do outside a tight loop should be done outside.

Also, you cut out the real work in you example no? (If it was still there but I did not see it, then your comments/spacing/naming failed.)
I assume you compared performance of just a mock loop reading only with a real work loop?

How much (MB!) real data to process? DCL uses SMALL RMS buffers and no read-ahead.
How long (elapsed and IO count) does reading the input data take in DCL versus say CONV/STAT data NL:/FDL=NL:

fwiw,
Hein.

[Jan van den Ende]

Robert:

Look carefully:

$ FIELD = F$EXT(DCF$'AREANAME'_'FIELDNAME'_POS,DCF$'AREANAME'_'FIELDNAME'_LEN,INDATA)
$ DCF$'AREANAME'_'FIELDNAME' == F$EXT(0,DCF$'AREANAME'_'FIELDNAME'_LEN,FIELD)

The first line results in FIELD having length 'fieldname'_LEN.
Then in the second line you do a complete string extract function, specifying to start at the beginning ( POS 0 ), re-evaluate the symbol for the length, resulting exactly in the total length of "FIELD".
Meaning:
DCF$'AREANAME'_'FIELDNAME' will become equal to FIELD, and for the next field you loose the value of FIELD anyway. No functional difference if you skip the entire use of the intermediate symbol FIELD.

Another (I guess minor) application of the same idea would be to evaluate the symbol DEBUG only once, outside this routine.

And back to the original stream question:
In a routine that is executed SO often, Martin's idea of putting the comments at the end certainly won't hurt, but I don'nt know if it will generate very much gain.

btw, I assume that the calling routine itself does not do a lot of processing as well? Maybe it also deserves closer scrutiny?

And - at the bottom line -
How often do you use this? If rarely, then does it merit the effort? if often, and the running time really is an issue, then maybe the idea of a compiled program deserves another tought.

Guess I am out of further ideas for now.

Success!

Jan

[Jan van den Ende]

Well,
Hein's answer and mine crossed one another,
but why not take his idea one further:
FILENAME = P1 and OPTION =p2.
Thereafter both FILENAME and OPTION are used only once.
Why not just skip those assignments, and use P1 & P2 directly?
Remember, it will be done OFTEN!

Jan

[Robert Atkinson]

While I left you guys to digest this, I went back and trimmed the module down to it's bare bones, removing all comments, blank lines, etc.

The best I could bring the speed down to was 3 seconds, as opposed to the half a second for pure DCL.

Your point about writing in a compiled language is reasonable, but it would be difficult to mix DCL and EXE's, plus I'd have issues with finding a language, support, and a host of other things.

It's a shame really, because this set of 'DCF (DCL Call File)' utilities is really handy for dynamically handling database files, rather than having to do lots of F$EXT and field[pos,len] commands.

I've zipped the whole lot if anyone is intertested.

Rob.

[Wim Van den Wyngaert]

A DCL compiler would be a nice product ...

[Willem Grooters]

Compiled or not:

FIRST OF ALL: you should have at least SOME clue on what the program would have to do and how to achive that efficiently. Know your language well!

This is, of course, quite obvious but given the nature of DCL, it is VERY tempting to code-during-test. (Ok, I'm not that innocent, doing this myself sometimes ;-))
This _can_ lead to monstruous constructions that are hard to read, let alone maintain, and can lead to unsdesired inefficiency.

As for 'compiled DCL': I think that will never happen. IMHO, you'll lose a lot of flexibility when compiling DCL...

[Antoniov.]

For "Compiled DCL" read here http://www.lns.cornell.edu/~pvhp/perl/vms/web/OSU-cgi-other-langs.txt

@Antoniov

[Jan van den Ende]

I agree with Willem, but would take it even one step back:

"know your PROBLEM well"
And that is, IMHO, where most "point & drag" script kiddies are absolutely blindfolded.

Jan