topic Re: dectpu truncating lines longer than 79 characters in Operating System - OpenVMS

dectpu truncating lines longer than 79 characters

Markus Waldorf_1 — Tue, 21 Mar 2006 09:25:27 GMT

Hello,

I would like to process a text file in DCL that contains text lines that are longer than 255 characters. Otherwise, using the text for sequential file operations returns:

%DCL-W-TKNOVF, command element is too long - shorten

And I also need to write output that does not ecxeed the 80 character limits. I've played around with dectpu, creating a procedure, but it's not that easy, and I haven't had too much fun so far. EVE (TPU) offers the Fill command, but it removes blank lines, etc. Unfortunatelly the source text does not have any paragraph markers. Basically all I need to do with the source text is to wrap lines that are longer than 79 characters - and only those lines - and then end each line with a LF and CR. Removing any text ident is another thing I need to apply.

Any ideas?

Best regads,
Markus

Re: dectpu truncating lines longer than 79 characters

John Abbott_2 — Tue, 21 Mar 2006 09:55:01 GMT

Hi Markus

Do you have access to a VMS V7.3-2 system ? as you can read beyond the previous limitation of 255 bytes, you can read up to 8192. What's your longest record length ?

J.

Re: dectpu truncating lines longer than 79 characters

Jim_McKinney — Tue, 21 Mar 2006 10:08:33 GMT

Though DCL is limiting you to 255 characters on a single command line, the internal DCL command buffer is larger. Try breaking your longer command line into shorter segments and use the hyphen continuation character to string them across several command lines as seen in the following trivial example.

$ show -
system -
/noprocess

Re: dectpu truncating lines longer than 79 characters

Hein van den Heuvel — Tue, 21 Mar 2006 10:21:00 GMT

Here is a basic 'wrap.com' DCL script that probably does what you want.
You may need to tweak the conditions to see whether 80 columns is acceptable and whether the space to wrap on needs to be retained or not.

Works for me. Used the following perl to verify:

$ @wrap a.tmp b.tmp
$ perl -pe "last if ($i++>5); $l=length; $h=substr($_,0,20); $_= ""$i $l $h\n""", b.tmp
1 78 And I also need to w
2 77 I've played around w
3 78 and I haven't had to
4 29 it removes blank lin
5 78 Unfortunatelly the s
6 55 all I need to do wit

$ perl -pe "last if ($i++>5); $l=length; $h=substr($_,0,20); $_= ""$i $l $h\n""", a.tmp
1 262 And I also need to w
2 133 Unfortunatelly the s
3 234 that are longer than
4 133 but it's not that ea
5 132 nfortunatelly the so
6 102 that are longer than

---- wrap.com -------

$open/read input 'p1
$if p2.eqs."" then p2="sys$output"
$open/write output 'p2
$record_loop:
$read/end=done input record
$word_loop:
$if f$len(record).lt.80
$then
$ write output record
$ goto record_loop
$else
$i=80
$byte_loop:
$i = i - 1
$if i.eq.0
$then
$write sys$output "OOOPS!"
$exit 0
$endif
$
$if f$extr(i,1,record).nes." " then goto byte_loop
$write output f$extr(0,i,record)
$record = f$extr(i+1,999,record)
$goto word_loop
$endif
$
$done:
$close input
$close output
$exit

hth,
Hein.

Re: dectpu truncating lines longer than 79 characters

Markus Waldorf_1 — Tue, 21 Mar 2006 11:43:26 GMT

Thanks a lot for the responses!

I was also looking at some perl scripts, which could do the job to reformat the text source, but I will have to install perl first.

What would be the latest perl supported under OpenVMS 7.1-2. It's an old AS2100 server. The last version of perl 5.8.6 seems to require 7.3-2. Since this is a production system, I'd like to make sure there won't be a problem. There is a link at the HP openvms perl support page for the previous version 5.6.1, but it does not outline the requirements.

The text file I'm trying to convert uses end of line as paragraph, probably CRLF.
As such it is not possible for the DECTPU "fill" command to determine the difference between a short and long line, and will wrap everything together. It also eliminates blank lines, which screws it up. Some of these lines are certainly longer than 500 characters. The file is actually an extract from vmsmail that I processed by "munpack" mime utility. The original email (quoted-printable) comes from a News agency.

I need to do some sequential file operations with DCL "read" to split the contents of the source file based on certain paragraph headers, save them into individual files and mail them again. But the "read" command will fail if a line of text is more than the max. record length, probably 255 characters. I'm afraid the solutions suggested so far won't work.

Is there really no way using DECTPU to reformat the text, or some other utility, beside Perl?

e.g.
He stressed that the entry denial is a "sovereign right of Belarus to contr
ol flows of foreigners entering the country." "Our invitation to an internationa
l institution such as the OSCE does not annul the Vienna Convention," he said.
%DCL-W-TKNOVF, command element is too long - shorten
\ Georgian Ambassador Grigor Katamadze insisted that the Georgians had the

Re: dectpu truncating lines longer than 79 characters

Jim_McKinney — Tue, 21 Mar 2006 12:17:47 GMT

Are you sure that it is the READ that is failing? Could it be a subsequent command where you're attempting to process a symbol returned from the read? If so, what is the command?

Re: dectpu truncating lines longer than 79 characters

Robert_Boyd — Tue, 21 Mar 2006 12:18:36 GMT

DECTPU will do what you want -- you just need to find the correct commands to set the right margin column, select the buffer and tell it to fill/wrap.

$ create wrap_text.eve:
SET RIGHT MARGIN 79
SET WRAP
SELECT ALL
FILL
EXIT
$!
$ edit/tpu my_input_file.txt -
/output=my_output_file.txt -
/nodisplay/initialization=wrap_text.eve

Re: dectpu truncating lines longer than 79 characters

Hein van den Heuvel — Tue, 21 Mar 2006 12:24:47 GMT

I don't know exactly what the problem is, but is sure sounds like perl will be the solution :-) :-).

As you are talking about very basic text processsing I would not worry in the least about the exact perl version being used.

But first and foremost you need to e more clear about the exact DCL operation which is failing for you, hopefull reduced to its basics: As simple as possible, but no simpler. Probably something along the lines of
:
$read file record
$var = f$extr(p,l,record)
:
Add to that details on the failing condition:
$SHOW SYMB record
$WRITE SYS$OUTPUT F$LEN(record)
DUMP/RECOR=(START=x,COUNT=1) raw.dat
... whatever seems needed to folks that do not see your environment.

DCL can and will read lines longer than 255 bytes as I showed above. Now let's focus on what it does not do for you, and under which conditions it fails.

hth,
Hein.

Re: dectpu truncating lines longer than 79 characters

Robert_Boyd — Tue, 21 Mar 2006 12:28:33 GMT

My apologies Markus,

I missed the part where you say you don't want blank lines removed.

There is a way to do it in TPU ... but I'll have to think about this some more.

It's really fairly straightforward... the simplest is to write a TPU procedure that scans forward one paragraph using calls to EVE$NEXT_PARAGRAPH at a time then does a call to EVE_FILL_PARAGRAPH. The loop should terminate when reaching the end of the buffer.

Robert

Re: dectpu truncating lines longer than 79 characters

Robert_Boyd — Tue, 21 Mar 2006 12:43:34 GMT

Also, have you tried using WRITE/SYMBOL to get around the TKNOVF error?

Even on older OpenVMS systems this should let you write records up to about 1000 characters long.

Robert

Re: dectpu truncating lines longer than 79 characters

Peter Weaver_1 — Tue, 21 Mar 2006 14:46:29 GMT

Have you looked at the FILL RANGE command in EVE? FILL will try to do a FILL PARAGRAPH unless you have selected an area then it will do a FILL RANGE. So, if I understand your problem correctly, you can do a simple learn sequence that moves to the start of the line, selects the entire line then do a FILL RANGE command, you may want to insert a new line here then move to the next line and repeat.

If you do insert a blank line after the new paragraph then you will end up with two blank lines if the original file had one blank line. If that is not what you want then the code for EVE_FILL_RANGE and EVE_FILL_PARAGRAPH are in EVE$FORMAT.TPU, so you can use that as a starting point if you have to write some code to skip working on lines that are 0 bytes long.

Re: dectpu truncating lines longer than 79 characters

Hein van den Heuvel — Tue, 21 Mar 2006 21:27:25 GMT

Btw... a perl 'one liner' for this wrap is:

perl -pe "while (length > 80){ if (/^.{0,79}\s/) {print ""$&\n""; $_=$'} else {die ""oops""}}" a.tmp

-p = loop over input printing $_ each time
-e = Program follows in text
while (length > 80){ = into sub-block if too long otherwise go print $_

if (/^.{0,79}\s/) = regular expression which matches :
^ == start at begin of line
. == any character
{0,79} == at least none, at most 79, greedy
\s == whitespace character

$& = matching string

$_=$' = Make default variable $_ the post-matching string (remainder)

else {die "oops"} = better find some space

Cheers,
Hein,

Re: dectpu truncating lines longer than 79 characters

Markus Waldorf_1 — Wed, 22 Mar 2006 04:36:57 GMT

Hello and thanks again for all the responses!

I've tried the Fill and Fill range tpu functions, but they don't seem to do the trick.

e.g.

If this is a line that's longer than the specified right margin, the tpu fill command will wrap it nicely.

however, if this line ends here.
and another line right there there.
The fill command will wrap the two lines together, which is not good..

I tried the above selecting (highlighting) one line, but it affects the subsequent line too, as it's not a paragraph. This would screw up list items.

Regarding the statement that DCL can read lines that are longer than 255 characters, even on older systems. I would assume there is probably some RMS parameter that can be tweaked, but in my case it seems to bail out at 315. Maybe writing the line into a symbol would work, but I can't see how to get the text out of the text file in the first place.

%DCL-W-TKNOVF, command element is too long - shorten
\ Mr. Milinkevich's close aides Anatol Lyabedzka and Alyaksandr Dabravolski
were reportedly arrested by police when they left the square to bring food. Als
o arrested during the night were Alyaksey Yanukevich, deputy chairman of the Bel
arusian Popular
315

I have attached a test.txt file and test.com to test for yourself if you like. Dir/full on the file test.txt shows variable length record, longest 566 bytes.

So far it seems I will need Perl ;-(.

Best regards,
Markus

Re: dectpu truncating lines longer than 79 characters

Hein van den Heuvel — Wed, 22 Mar 2006 07:42:34 GMT

>>> So far it seems I will need Perl ;-(.

Yes you will need perl.
Everyone needs perl.
But not for this problem.

For this problem you needed to READ the advice offered more carefully, and WRITE/SYMBOL

Jim>> Are you sure that it is the READ that is failing?

Hein>> But first and foremost you need to e more clear about the exact DCL operation which is failing for you

Robert>> Also, have you tried using WRITE/SYMBOL to get around the TKNOVF error?

Thanks for attaching the script.
Runnning it with SET VERI (and cutting down the sample data some :-), it is soon clear that a WRITE is failing.
This can readily be fixed by using WRITE/SYMBOL

[ but don't just redefine your 'prn' as write /sym can only be used for symbols like your 'line', but not "x" + "y". You'd have to move that in a temp symbol first. ]

Cheers!
Hein.

Re: dectpu truncating lines longer than 79 characters

Jim_McKinney — Wed, 22 Mar 2006 07:52:12 GMT

The attached will cure your TKNOVF WRITE problem.

Re: dectpu truncating lines longer than 79 characters

Markus Waldorf_1 — Wed, 22 Mar 2006 08:28:53 GMT

Thanks,

I was not aware of DCL write/symbol, and missunderstood the previous post. I did a few tests meanwhile, simply changing the example in test.com from write to write/symbol and it worked. Even though write/symbol has a limit still, I probably won't reach it in practice, I hope.

I will still need to work out the 79 character per line limit, as I will have to re-mail the message for further processing by a program that simply truncates everything that has more than 132 characters per line.

At least I can start thinking of doing it in DCL now, rather than DECTPU. I'm not very experienced in writing DECTPU procedures.

Thanks,
Markus

Re: dectpu truncating lines longer than 79 characters

Markus Waldorf_1 — Wed, 22 Mar 2006 11:47:34 GMT

Hello again,

Acutally the solution I'm going to choose is not really any of the above, but it did a great deal to help me going the right direction. I would certainly not have considered it without all of your valuable input.

As it turns out DCL can do the thing. The trick is to extract a substring from the line record that would not exceed the max. length of 255 characters when writing - which I want to do anyway.

Below is my modifed test.com. It's not very beautiful yet, and cuts in the middle of words, but it's a good start.

$!TEST.COM
$ prn := write sys$output
$ maxlen := 64
$!
$!
$ PREPARE:
$ open/read/error=ERROR infile test.txt
$ open/write/error=ERROR outfile out.txt
$ PREPARE10:
$ read/end = PREPARE30 infile line
$ curlen = f$lenght(line)
$ cursor = 0
$ PREPARE11:
$ if curlen .lt. maxlen
$ then
$ write outfile line
$ goto PREPARE10
$ else
$ line_temp = f$extract(cursor,maxlen,line)
$ write outfile line_temp
$ cursor = cursor + maxlen
$ if cursor .ge. curlen
$ then
$ line_temp = f$extract(cursor,maxlen,line)
$ goto PREPARE10
$ else
$ goto PREPARE11
$ endif
$ endif
$ PREPARE30:
$ close infile
$ close outfile
$ goto END
$ END:
$ exit
$!END

output looks like:

* Society/events
Events coming on Tuesday

Minsk, 20 March. A protest is expected to continue in centr
al Minsk against the official results of Sunday's presidential e
lection challenged by two opposition candidates.
European Union foreign ministers meet in Brussels to discus
s the situation in Belarus.
Belarusian and Hungarian transport officials meet in Budape
st.
More than 300 exhibitors are to take part in an internation
al construction fair that opens at the exhibition center on Yank
a Kupala Street in Minsk.

Re: dectpu truncating lines longer than 79 characters

Hein van den Heuvel — Wed, 22 Mar 2006 14:17:45 GMT

Good to see you are on a good road to a solution.

It is really a text editting job, and as such TPU should be a good solution, but personally i never opted to learn that language, so I couldn't help you with that.

> Acutally the solution I'm going to choose > is not really any of the above
:
> Below is my modifed test.com. It's not very beautiful yet, and cuts in the middle of words, but it's a good start.

Hmmm, the dcl script I posted earlier does NOT cut in the middle of words and ha slightly more readable labels.

Sample output from that earlier post with line length adjusted to 64.

$wrap test.txt
:

Minsk, 20 March. A protest is expected to continue in
central Minsk against the official results of Sunday's
presidential election challenged by two opposition candidates.
European Union foreign ministers meet in Brussels to
discuss the situation in Belarus.
Belarusian and Hungarian transport officials meet in
Budapest.
More than 300 exhibitors are to take part in an
international construction fair that opens at the exhibition
center on Yanka Kupala Street in Minsk.

Cheers,
Hein.

Re: dectpu truncating lines longer than 79 characters

Markus Waldorf_1 — Thu, 23 Mar 2006 04:18:21 GMT

Hello Hein,

True.

I somehow overlooked your "wrap.com". I was too much focused on perl and tpu, sorry.

Best regards,
Markus

Re: dectpu truncating lines longer than 79 characters

Markus Waldorf_1 — Thu, 30 Mar 2006 12:28:33 GMT

Well, looks like a problem:

$ if f$lenght(line) .lt. line_warp
%DCL-W-BUFOVF, command buffer overflow - shorten expression or command line

The line in questions has 1520 8-bit (cyrillic) Characters.

Apparently f$length has a limit. So will it have to be Perl than?

Best regards,
Markus