dectpu truncating lines longer than 79 characters

[Markus Waldorf_1]

Hello,

I would like to process a text file in DCL that contains text lines that are longer than 255 characters. Otherwise, using the text for sequential file operations returns:

%DCL-W-TKNOVF, command element is too long - shorten

And I also need to write output that does not ecxeed the 80 character limits. I've played around with dectpu, creating a procedure, but it's not that easy, and I haven't had too much fun so far. EVE (TPU) offers the Fill command, but it removes blank lines, etc. Unfortunatelly the source text does not have any paragraph markers. Basically all I need to do with the source text is to wrap lines that are longer than 79 characters - and only those lines - and then end each line with a LF and CR. Removing any text ident is another thing I need to apply.

Any ideas?

Best regads,
Markus

[John Abbott_2]

Hi Markus

Do you have access to a VMS V7.3-2 system ? as you can read beyond the previous limitation of 255 bytes, you can read up to 8192. What's your longest record length ?

J.

[Jim_McKinney]

Though DCL is limiting you to 255 characters on a single command line, the internal DCL command buffer is larger. Try breaking your longer command line into shorter segments and use the hyphen continuation character to string them across several command lines as seen in the following trivial example.

$ show -
system -
/noprocess

[Hein van den Heuvel]

Here is a basic 'wrap.com' DCL script that probably does what you want.
You may need to tweak the conditions to see whether 80 columns is acceptable and whether the space to wrap on needs to be retained or not.

Works for me. Used the following perl to verify:

$ @wrap a.tmp b.tmp
$ perl -pe "last if ($i++>5); $l=length; $h=substr($_,0,20); $_= ""$i $l $h\n""", b.tmp
1 78 And I also need to w
2 77 I've played around w
3 78 and I haven't had to
4 29 it removes blank lin
5 78 Unfortunatelly the s
6 55 all I need to do wit

$ perl -pe "last if ($i++>5); $l=length; $h=substr($_,0,20); $_= ""$i $l $h\n""", a.tmp
1 262 And I also need to w
2 133 Unfortunatelly the s
3 234 that are longer than
4 133 but it's not that ea
5 132 nfortunatelly the so
6 102 that are longer than

---- wrap.com -------

$open/read input 'p1
$if p2.eqs."" then p2="sys$output"
$open/write output 'p2
$record_loop:
$read/end=done input record
$word_loop:
$if f$len(record).lt.80
$then
$ write output record
$ goto record_loop
$else
$i=80
$byte_loop:
$i = i - 1
$if i.eq.0
$then
$write sys$output "OOOPS!"
$exit 0
$endif
$
$if f$extr(i,1,record).nes." " then goto byte_loop
$write output f$extr(0,i,record)
$record = f$extr(i+1,999,record)
$goto word_loop
$endif
$
$done:
$close input
$close output
$exit




hth,
Hein.

[Markus Waldorf_1]

Thanks a lot for the responses!

I was also looking at some perl scripts, which could do the job to reformat the text source, but I will have to install perl first.

What would be the latest perl supported under OpenVMS 7.1-2. It's an old AS2100 server. The last version of perl 5.8.6 seems to require 7.3-2. Since this is a production system, I'd like to make sure there won't be a problem. There is a link at the HP openvms perl support page for the previous version 5.6.1, but it does not outline the requirements.

The text file I'm trying to convert uses end of line as paragraph, probably CRLF.
As such it is not possible for the DECTPU "fill" command to determine the difference between a short and long line, and will wrap everything together. It also eliminates blank lines, which screws it up. Some of these lines are certainly longer than 500 characters. The file is actually an extract from vmsmail that I processed by "munpack" mime utility. The original email (quoted-printable) comes from a News agency.

I need to do some sequential file operations with DCL "read" to split the contents of the source file based on certain paragraph headers, save them into individual files and mail them again. But the "read" command will fail if a line of text is more than the max. record length, probably 255 characters. I'm afraid the solutions suggested so far won't work.

Is there really no way using DECTPU to reformat the text, or some other utility, beside Perl?

e.g.
He stressed that the entry denial is a "sovereign right of Belarus to contr
ol flows of foreigners entering the country." "Our invitation to an internationa
l institution such as the OSCE does not annul the Vienna Convention," he said.
%DCL-W-TKNOVF, command element is too long - shorten
\ Georgian Ambassador Grigor Katamadze insisted that the Georgians had the

[Jim_McKinney]

Are you sure that it is the READ that is failing? Could it be a subsequent command where you're attempting to process a symbol returned from the read? If so, what is the command?

[Robert_Boyd]

DECTPU will do what you want -- you just need to find the correct commands to set the right margin column, select the buffer and tell it to fill/wrap.

$ create wrap_text.eve:
SET RIGHT MARGIN 79
SET WRAP
SELECT ALL
FILL
EXIT
$!
$ edit/tpu my_input_file.txt -
/output=my_output_file.txt -
/nodisplay/initialization=wrap_text.eve

[Hein van den Heuvel]

I don't know exactly what the problem is, but is sure sounds like perl will be the solution :-) :-).

As you are talking about very basic text processsing I would not worry in the least about the exact perl version being used.

But first and foremost you need to e more clear about the exact DCL operation which is failing for you, hopefull reduced to its basics: As simple as possible, but no simpler. Probably something along the lines of
:
$read file record
$var = f$extr(p,l,record)
:
Add to that details on the failing condition:
$SHOW SYMB record
$WRITE SYS$OUTPUT F$LEN(record)
DUMP/RECOR=(START=x,COUNT=1) raw.dat
... whatever seems needed to folks that do not see your environment.

DCL can and will read lines longer than 255 bytes as I showed above. Now let's focus on what it does not do for you, and under which conditions it fails.

hth,
Hein.

[Robert_Boyd]

My apologies Markus,

I missed the part where you say you don't want blank lines removed.

There is a way to do it in TPU ... but I'll have to think about this some more.

It's really fairly straightforward... the simplest is to write a TPU procedure that scans forward one paragraph using calls to EVE$NEXT_PARAGRAPH at a time then does a call to EVE_FILL_PARAGRAPH. The loop should terminate when reaching the end of the buffer.

Robert

[Robert_Boyd]

Also, have you tried using WRITE/SYMBOL to get around the TKNOVF error?

Even on older OpenVMS systems this should let you write records up to about 1000 characters long.

Robert

[Peter Weaver_1]

Have you looked at the FILL RANGE command in EVE? FILL will try to do a FILL PARAGRAPH unless you have selected an area then it will do a FILL RANGE. So, if I understand your problem correctly, you can do a simple learn sequence that moves to the start of the line, selects the entire line then do a FILL RANGE command, you may want to insert a new line here then move to the next line and repeat.

If you do insert a blank line after the new paragraph then you will end up with two blank lines if the original file had one blank line. If that is not what you want then the code for EVE_FILL_RANGE and EVE_FILL_PARAGRAPH are in EVE$FORMAT.TPU, so you can use that as a starting point if you have to write some code to skip working on lines that are 0 bytes long.

[Hein van den Heuvel]

Btw... a perl 'one liner' for this wrap is:

perl -pe "while (length > 80){ if (/^.{0,79}\s/) {print ""$&\n""; $_=$'} else {die ""oops""}}" a.tmp

-p = loop over input printing $_ each time
-e = Program follows in text
while (length > 80){ = into sub-block if too long otherwise go print $_

if (/^.{0,79}\s/) = regular expression which matches :
^ == start at begin of line
. == any character
{0,79} == at least none, at most 79, greedy
\s == whitespace character

$& = matching string

$_=$' = Make default variable $_ the post-matching string (remainder)

else {die "oops"} = better find some space


Cheers,
Hein,

[Markus Waldorf_1]

Hello and thanks again for all the responses!

I've tried the Fill and Fill range tpu functions, but they don't seem to do the trick.

e.g.

If this is a line that's longer than the specified right margin, the tpu fill command will wrap it nicely.

however, if this line ends here.
and another line right there there.
The fill command will wrap the two lines together, which is not good..

I tried the above selecting (highlighting) one line, but it affects the subsequent line too, as it's not a paragraph. This would screw up list items.

Regarding the statement that DCL can read lines that are longer than 255 characters, even on older systems. I would assume there is probably some RMS parameter that can be tweaked, but in my case it seems to bail out at 315. Maybe writing the line into a symbol would work, but I can't see how to get the text out of the text file in the first place.


%DCL-W-TKNOVF, command element is too long - shorten
\ Mr. Milinkevich's close aides Anatol Lyabedzka and Alyaksandr Dabravolski
were reportedly arrested by police when they left the square to bring food. Als
o arrested during the night were Alyaksey Yanukevich, deputy chairman of the Bel
arusian Popular
315

I have attached a test.txt file and test.com to test for yourself if you like. Dir/full on the file test.txt shows variable length record, longest 566 bytes.


So far it seems I will need Perl ;-(.

Best regards,
Markus

[Hein van den Heuvel]

>>> So far it seems I will need Perl ;-(.

Yes you will need perl.
Everyone needs perl.
But not for this problem.

For this problem you needed to READ the advice offered more carefully, and WRITE/SYMBOL

Jim>> Are you sure that it is the READ that is failing?

Hein>> But first and foremost you need to e more clear about the exact DCL operation which is failing for you

Robert>> Also, have you tried using WRITE/SYMBOL to get around the TKNOVF error?


Thanks for attaching the script.
Runnning it with SET VERI (and cutting down the sample data some :-), it is soon clear that a WRITE is failing.
This can readily be fixed by using WRITE/SYMBOL

[ but don't just redefine your 'prn' as write /sym can only be used for symbols like your 'line', but not "x" + "y". You'd have to move that in a temp symbol first. ]

Cheers!
Hein.

[Jim_McKinney]

The attached will cure your TKNOVF WRITE problem.

[Markus Waldorf_1]

Thanks,

I was not aware of DCL write/symbol, and missunderstood the previous post. I did a few tests meanwhile, simply changing the example in test.com from write to write/symbol and it worked. Even though write/symbol has a limit still, I probably won't reach it in practice, I hope.

I will still need to work out the 79 character per line limit, as I will have to re-mail the message for further processing by a program that simply truncates everything that has more than 132 characters per line.

At least I can start thinking of doing it in DCL now, rather than DECTPU. I'm not very experienced in writing DECTPU procedures.

Thanks,
Markus

[Markus Waldorf_1]

Hello again,

Acutally the solution I'm going to choose is not really any of the above, but it did a great deal to help me going the right direction. I would certainly not have considered it without all of your valuable input.

As it turns out DCL can do the thing. The trick is to extract a substring from the line record that would not exceed the max. length of 255 characters when writing - which I want to do anyway.

Below is my modifed test.com. It's not very beautiful yet, and cuts in the middle of words, but it's a good start.

$!TEST.COM
$ prn := write sys$output
$ maxlen := 64
$!
$!
$ PREPARE:
$ open/read/error=ERROR infile test.txt
$ open/write/error=ERROR outfile out.txt
$ PREPARE10:
$ read/end = PREPARE30 infile line
$ curlen = f$lenght(line)
$ cursor = 0
$ PREPARE11:
$ if curlen .lt. maxlen
$ then
$ write outfile line
$ goto PREPARE10
$ else
$ line_temp = f$extract(cursor,maxlen,line)
$ write outfile line_temp
$ cursor = cursor + maxlen
$ if cursor .ge. curlen
$ then
$ line_temp = f$extract(cursor,maxlen,line)
$ goto PREPARE10
$ else
$ goto PREPARE11
$ endif
$ endif
$ PREPARE30:
$ close infile
$ close outfile
$ goto END
$ END:
$ exit
$!END

output looks like:

* Society/events
Events coming on Tuesday

Minsk, 20 March. A protest is expected to continue in centr
al Minsk against the official results of Sunday's presidential e
lection challenged by two opposition candidates.
European Union foreign ministers meet in Brussels to discus
s the situation in Belarus.
Belarusian and Hungarian transport officials meet in Budape
st.
More than 300 exhibitors are to take part in an internation
al construction fair that opens at the exhibition center on Yank
a Kupala Street in Minsk.

[Hein van den Heuvel · Accepted solution]

Good to see you are on a good road to a solution.

It is really a text editting job, and as such TPU should be a good solution, but personally i never opted to learn that language, so I couldn't help you with that.

> Acutally the solution I'm going to choose > is not really any of the above
:
> Below is my modifed test.com. It's not very beautiful yet, and cuts in the middle of words, but it's a good start.

Hmmm, the dcl script I posted earlier does NOT cut in the middle of words and ha slightly more readable labels.

Sample output from that earlier post with line length adjusted to 64.

$wrap test.txt
:

Minsk, 20 March. A protest is expected to continue in
central Minsk against the official results of Sunday's
presidential election challenged by two opposition candidates.
European Union foreign ministers meet in Brussels to
discuss the situation in Belarus.
Belarusian and Hungarian transport officials meet in
Budapest.
More than 300 exhibitors are to take part in an
international construction fair that opens at the exhibition
center on Yanka Kupala Street in Minsk.

Cheers,
Hein.

[Markus Waldorf_1]

Hello Hein,

True.

I somehow overlooked your "wrap.com". I was too much focused on perl and tpu, sorry.

Best regards,
Markus

[Markus Waldorf_1]

Well, looks like a problem:

$ if f$lenght(line) .lt. line_warp
%DCL-W-BUFOVF, command buffer overflow - shorten expression or command line

The line in questions has 1520 8-bit (cyrillic) Characters.

Apparently f$length has a limit. So will it have to be Perl than?

Best regards,
Markus

[Hein van den Heuvel]

Yeah... 'fraid so.

Unless... you want to 'play rms', which I do not recommend.

The file in question is probably already stream-lf, or you could make it be that with a simple convert.
Next you are do a
$SET file/ATTR=(RFM=FIX,LRL=512,MRS=512)
! or 512, or anything else really less than dcl-max minus your-max-line-length.

Now chug through the file records looking for the first linefeed before byte 80, before looking for to the last whitespace before 80 chars.
The (minor) added complexity is to always read a new record and add it to the currnet chunk as soon as the chunk drops under 80 bytes.
And you'll have to deal with the last, short record.

SMOP!

grins,
Hein.

[Hein van den Heuvel]

Well, there is a minor complication with my 'roll your own' suggestion, in that the last record read is a whole record, not just up untill the "end-of-file-byte" (FFB)
This makes reading in 512 byte chunks and looking out for the EOF required.

Please find below a 'wrap.com' which operates on stream-lf file which are zapped to become fixed/512 files.

Works for me... (but I'd use the excuse to load up perl)

Hein.

------------- wrap.com -------------

$ type wrap.com
$lf[0,8]=10
$
$open/read input 'p1
$if p2.eqs."" then p2="sys$output"
$open/write output 'p2
$read_record = 0
$eof = f$file(p1,"EOF")
$ffb = f$file(p1,"FFB")
$record = ""
$
$record_loop:
$read/end=done input new_chunk
$read_record = read_record + 1
$if read_record .eq. eof then new_chunk = f$extr(0,ffb,new_chunk)
$record = record + new_chunk
$
$eol_loop:
$eol = f$loc(lf,record)
$len = f$len(record)
$if eol.lt.80
$then
$ if eol.eq.len then goto record_loop
$ write sys$output f$extr(0,eol,record)
$ record = f$extr(eol+1,999,record)
$ goto eol_loop
$endif
$
$word_loop:
$if f$len(record).lt.80
$then
$ write output record
$ goto eol_loop
$endif
$
$i=80
$byte_loop:
$i = i - 1
$if i.eq.0
$then
$write sys$output "OOOPS!"
$exit 0
$endif
$
$if f$extr(i,1,record).nes." " then goto byte_loop
$write output f$extr(0,i,record)
$record = f$extr(i+1,999,record)
$goto eol_loop
$
$done:
$write output record
$close input
$close output
$exit

[Markus Waldorf_1]

Hello,

I was also wondering about making the text file into smaller records, but did not know how to accomplish it. Actually I'm still not really sure how it works. I thought that the read command reads a record which is usually a line terminated by a LF or CR (?). Setting the text file to fixed records even works around the problem of a buffer overflow with the previous wrap.com, but the format of the output is messed up - don't know why yet. Anyway below are some results.


2006033016575382.OUT
Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Record attributes: Carriage return carriage control

$ copy 2006033016575382.OUT 2006033016575382.fix
$ set file/attr=(RFM=FIX,LRL=512,MRS=512) 2006033016575382.fix

2006033016575382.FIX
Record format: Fixed length 512 byte records
Record attributes: Carriage return carriage control

$ @wrap 2006033016575382.OUT 1.tmp
%DCL-W-BUFOVF, command buffer overflow - shorten expression or command line

$ @wrap 2006033016575382.fix 2.tmp
ok.

$ @wrap_new 2006033016575382.fix 3.tmp
ok.

$ @wrap_new 2006033016575382.out 4.tmp
OOOPS!
%NONAME-W-NOMSG, Message number 00000000


$ dir /size *.tmp
1.TMP;1 23
2.TMP;1 42
3.TMP;1 42
4.TMP;1 1

Results:

1.tmp - format ok, but incomplete.
2.tmp - format wrong, complete
3.tmp - fine.
4.tmp - format wrong, incomplete


P.S. I modifed the new wrap.com
$ write sys$output f$extr(0,eol,record) to
$ write output f$extr(0,eol,record).

I will play with RMS for now and see how it works.

Many thanks!

[Hein van den Heuvel]

>> I thought that the read command reads a record which is usually a line terminated by a LF or CR (?).

The read command, through RMS, listens to the file fiel attributes to know what makes up a record. For the .COM file itself, or any other 'normal' VMS text file this is done through word-counted, word-alligned strings. C programs often make 'stream-lf' files, where each record is delimited by a linefeed, and a linefeed alone. Raw PC files have records delimited by CR-LF pairs.

>> Setting the text file to fixed records even works around the problem of a buffer overflow with the previous wrap.com, but the format of the output is messed up - don't know why yet.

It is a 'white lie' to RMS. RMS now just blindy cuts the file into 512 bytes chunks, which willl never cause the TokenNotFound error.

the wrap_new, knows this and knows to look for linefeeds itself, re-constituting the records. It can only work with the 512-byte input files, not with the original LF-seperated records. Visa versa teh origianl wrap.com really expects an input file with the LF terminators 'active' and will misbehave on the fixed-length-pretend data as it finds surpise line endings in the middle of what it thinks are records.

>>> 3.tmp - fine.


So this is good!

Hein.

[Hein van den Heuvel]

Oh, btw.. if you are really interested how these attribute changes work, which I would encourage your to understand, then just DUMP a few blocks/records from the normal files and the 'FIXed' files.

DUMP/BLOC=(COUNT=3,STAR=1) will NOT show a difference.
DUMP/RECO=(COUNT=2,START=1) _will_ show serious difference.

Now DUMP/BLOC=COUNT=1 wrap.com and see if you can figure out the records.

Hein.