topic Re: How to get the NUmber of lines in a file into a variable in Operating System - OpenVMS

How to get the NUmber of lines in a file into a variable

Prem Mohan — Tue, 04 Jan 2005 09:25:56 GMT

I want to get the number of lines in a file into a variable. I do get the number of lines in a files using this command
sear "" /log /nooutput.
But I have trouble assigning this output to a variable. In Unix I do wc -l . Any help is greatly appreciated.
Prem.

Re: How to get the NUmber of lines in a file into a variable

Kris Clippeleyr — Tue, 04 Jan 2005 09:53:05 GMT

Hi,

You might try the lexical function F$FILE_ATTRIBUTES, with FILE_LENGTH_HINT as item-code. Something like:
$ flh = f$file_attributes("TEST.TXT","FILE_LENGTH_HINT")
$ show symbol flh
See also HELP LEXICALS F$FILE_ATTRIBUTES

Greetz,

Kris (aka Qkcl)

Re: How to get the NUmber of lines in a file into a variable

Volker Halle — Tue, 04 Jan 2005 10:08:47 GMT

Prem,

you can also achiev this via PIPE (see examples in OpenVMS FAQ):

$ pipe search login.com ""/log/noout| ( read sys$input line ; wc = F$ELEMENT(3," ",line) ; define/job wc &wc)
$ x=F$TRNLNM("wc")
$ sho sym x
X = "79"
$ DEASS/JOB wc

FILE_LENGTH_HINT only works on ODS-5 volumes !

$ write sys$output f$file_attributes("login.com","file_length_hint")
(-1,-1)

Volker.

Re: How to get the NUmber of lines in a file into a variable

Ken Robinson — Tue, 04 Jan 2005 10:10:19 GMT

The new way is to use the PIPE command:

$ pipe sear/noout/stat "" | sear sys$pipe "Records sear" | ( read sys$pipe line ; num = f$ele(2," ",f$edi(line,"compress")) ; def/job numl &num )
$ numl = f$trn("numl")
$ sho sym numl

Ken

Re: How to get the NUmber of lines in a file into a variable

Ian Miller. — Tue, 04 Jan 2005 10:11:36 GMT

note that the file length hint may be invalid. The command ANAL/RMS/UPDATE attempts to update the file length hint. The help text for ANAL/RMS/UPDATE explains the file and record formats to which the file lenght hint applies.
You also need a recent version of VMS (its present in VMS V7.3-1 but no in V7.1).

Parhaps something ingenious could be done with the SEARCH command in a PIPE.

Re: How to get the NUmber of lines in a file into a variable

Craig A Berry — Tue, 04 Jan 2005 12:01:57 GMT

Prem,

If you have GNV installed, just do it the same way you do it on Unix:

$ bash -c wc -l

If not, a few lines of DCL should do the trick:

$ open/read a 'p1
$ count = 0
$ loop:
$ read/end=all_done a line
$ count = count + 1
$ goto loop
$ all_done:
$ close a
$ write sys$output "There are ",count," records in ", f$parse("''p1'")
$ exit

Re: How to get the NUmber of lines in a file into a variable

Jan van den Ende — Tue, 04 Jan 2005 13:44:39 GMT

Craig,

I am with you on the GNV solution.

However, for the DCL script, if the input file has some size, it tends to be VERY MUCH slower than the PIPE | search solution by Ken.

However, the DCL works on any version as far as I remember, the PIPE only in 7.2 & up.

(and am __I__ writing this? The DCL aficionado?? Then again, PIPE should now also be considered native DCL, so... )

Proost.

Have one on me.

Jan

Re: How to get the NUmber of lines in a file into a variable

Craig A Berry — Tue, 04 Jan 2005 15:13:56 GMT

Jan,

No doubt you're right about the speed of DCL reads, but it's something that has always worked and I think it always will work. Parsing the output of SEARCH will probably continue to work, but it's less certain.

Just for something completely different, here's how to get the number of lines in a file into a DCL symbol using a one-line Perl command:

$ perl -"MVMS::DCLsym" -ne "$i++; VMS::DCLsym->setsym('rcnt',$i,'GLOBAL');" test.txt
$ sh sym rcnt
RCNT == "8"

Re: How to get the NUmber of lines in a file into a variable

John Gillings — Tue, 04 Jan 2005 18:13:34 GMT

SORT is very fast. I've attached a small MACRO32 program that does nothing more than count the records. I'd thought that bypassing the string matching might speed things up a bit, but my program isn't significantly faster than SORT, even for large files.

However, it doesn't need PIPE to catch the record count, so you save yourself some process creations. File is LC.TXT. Rename to .MAR, then MACRO and LINK. Should work on all versions of OpenVMS, VAX, Alpha and IA64.

The interface is foreign command, so define a symbol, use MCR or define DCL$PATH to include the directory containing the image.

Return value is in R0, so it's in $STATUS. To prevent a spurious error, it's left shifted and the low bit is set (so it looks like a success status). To get the count, just divide $STATUS by 2. For example:

$ LC MYFILE.TXT
$ stat=$STATUS
$ IF stat
$ THEN
$ count=stat/2
$ ELSE
$ ! error
$ ENDIF

Re: How to get the NUmber of lines in a file into a variable

Hein van den Heuvel — Tue, 04 Jan 2005 18:19:39 GMT

An other standard VMS program that is willing to spit out the number of records in a sequential file is copy:

$ copy /log tmp.com nl:
%COPY-S-COPIED, xxxx copied to NL: (5 records)

This eliminates the need for a search or repeated read in the pipe line:

$ pipe copy login.com nl:/log | (read sys$pipe line ; x= f$elem(1,"(",line) - " records)" ; defi/job lines &x )

An other way for to solve this in perl with a logical name is:

$ perl -e "{} while <>; $ENV{'lines'}=$." < tmp.idx
$ show log lines
"LINES" = "2000" (LNM$PROCESS_TABLE)

Pretty clean and flexible. Cheaper than piping.

Or with perl and the return status:

$ perl -e "{} while <>; exit $.*10 + 1" < tmp.idx
$ show symb $status
$STATUS == "%X00004E21"
$ lines =$status/10
$ show symb lines
LINES = 2000 Hex = 000007D0 Octal = 00000003720

Enjoy,
Hein.

Re: How to get the NUmber of lines in a file into a variable

Aleksander Sinigoj — Wed, 05 Jan 2005 05:38:33 GMT

If you have large files to check, and you consider to use John's solution (macro program) you can speed up it a litle. Replace the following line:
$GET RAB=InRAB
by
$FIND RAB=InRAB.

Re: How to get the NUmber of lines in a file into a variable

Prem Mohan — Wed, 05 Jan 2005 05:41:57 GMT

Hi All,

First I am overwhelmed with the fast response I got from all of you. Thanks all of you I appreciate it. I would like to test which one of the procedures suggested is not resource intensive and use that

Hi Greetz,

I tried file_length_attributes before I put this question into the forum and it did not work properly.

Hi Volker,

It is working, But I am using the option given by Ken.

Hi Craig/Jan,

GNV is not installed, But I do not want to open the file and do the DCL commands the reason being the files are 200/300 MB and it will take long time to execute.

Hi Hein,

I would not use copy for the simple reason of the file size and I do not have perl installed in the system.

Re: How to get the NUmber of lines in a file into a variable

Ian Miller. — Wed, 05 Jan 2005 05:44:54 GMT

read a head with multiple multiblock buffers may also speed up the program.

Re: How to get the NUmber of lines in a file into a variable

Hein van den Heuvel — Wed, 05 Jan 2005 07:14:32 GMT

>> I would not use copy for the simple reason of the file size

Well, my suggestion does not acutally copy the file. The output goes to the null device (NL:). Try it, Time it.

>> 200+ MB, at least we'll know that any pipe overhead will be neglectable.

>> and I do not have perl installed in the system.

Well, you should. QED!
What problem are you _really_ trying to solve?
A number of lines is often interesting, but not the ultimate goal.
You may find that the subsequent processing, is more readily done in perl than anything else.

And if you were to special case a program for this, then indeed the RMS call $FIND will be faster than $GET, and use MBF=120, MBC=4, ROP=RAH, SHR=UPI. It return the byte count.

Be sure to try native C - IO or even COBOL or SORT to read it, because they will NOT use RMS for simple, unshared sequential files. The CPU overhead for going into RMS (Executive inner mode) and back is simply too high when all you do is count records and bytes.

Cheers,
Hein.

Re: How to get the NUmber of lines in a file into a variable

Ian Miller. — Wed, 05 Jan 2005 09:50:57 GMT

Hien, are you saying some C IO routines will not use RMS sometimes (do they use RMS to open then ACP QIO READ virtual block)?
I thought C RTL always used RMS record I/O but parhaps things have changed.

Re: How to get the NUmber of lines in a file into a variable

Hein van den Heuvel — Wed, 05 Jan 2005 10:40:57 GMT

Ian,

Yes indeed. I thought it was an opportune moment to point out that it has been recognized that RMS Record IO is some times TOO CPU expensive, due to the CMEXEC, argument probing, and so on. This is all there to support safe file sharing, and multiple file organization. If all you are doing is a simple sequential file read (variable, stream, fixed), then rolling your own using SYS$READ or QIO is more expedient.

The C RTL has recognized this first, next COBRTL (same folks more or less), and of course SORT.

If one really want to do this well, you would want to build in double (or more) buffering for max IO speed, and you even might consider IO_PERFORM for repeated reads fo very large simple files. And ideally you might consider sys$getjpi / getsyi to read default rms buffer settings, in case they are larger then the program defaults (C RTL does this, I happen to have coded the dcl support for this best I remember.).

Hein.

Re: How to get the NUmber of lines in a file into a variable

Ian Miller. — Wed, 05 Jan 2005 12:16:54 GMT

Hein, which version was this implemented for C and does it apply to read and fgets ?

Re: How to get the NUmber of lines in a file into a variable

Hein van den Heuvel — Wed, 05 Jan 2005 14:16:45 GMT

Oh, best I know this has been the case 'forever' in DECC and since 1990 or so for VAXC.

Hein.

Re: How to get the NUmber of lines in a file into a variable

Prem Mohan — Wed, 12 Jan 2005 08:37:13 GMT

Hi,

$ pipe sear/noout/stat "" | sear sys$pipe "Records sear" | -
( read sys$pipe line ; num = f$ele(2," ", f$edi(line,"compress")) ; def /job numl &num )
$ numl = f$trn("numl")
$ sh sym numl

works fine from the command line, But when I use the same into a com file to be executed the output shows 0.

Please tell me what am I doing wrong.
The com file is enclosed for your review.

Best Regards
Prem.

Re: How to get the NUmber of lines in a file into a variable

Jan van den Ende — Wed, 12 Jan 2005 09:14:00 GMT

Prem,

could it be that the default directory when you run the comfile is NOT the directory where your .PP file is located?
All through the script you speel out the locations of various files (which I HIGHLY endorse! It prevents so many glitches!), EXCEPT for the line:

pipe sear/noout/stat 0103_WAT002616_00.PP "" | sear sys$pipe "Records sear" | -

Check this first, please.

Proost.

Have one on me.

Jan