DCL command buffer overflow

[Wim Van den Wyngaert]

VMS 7.3

$ Read x x_rec
$ y=f$extr(0,10,x_rec)
DCL-W-BUFOV, command buffer overflow

x is a file with long records (1200 char).

How to get around this overflow ?

Wim

[Karl Rohwedder]

Seems to be on VAX :-)

You may try the EXTRACT (freeware CD) utility, gathering the output via the PIPE command:

$ pip extract sc:a.dat /col=(1:10) | (read sys$pipe a ; def/job test &A)
$ y = F$trnlnm("TEST")

or

write your own program.

regards Kalle

[Wim Van den Wyngaert]

No VAX but Alpha. No way getting around without installing stuff ?

BTW : I don't need the whole record, only the beginning and I prefer not to do a convert to truncate the file.

Wim

[Volker Halle]

Wim,

if you are ONLY interested in the first 10 bytes of each record, CONVERT/FDL=.../TRUNCATE to a temp file first, then process the temp file.

Or upgrade to V7.3-2 or higher to be able to use Extended DCL - but you know that ;-)

Volker.

[Wim Van den Wyngaert]

I tried this but no error and not working. Funny bug or feature.

>open/read x login.com
>read x xx
>sh symb xx
XX = "$ set nover"
>read x xx[0,10]
>sh symb xx
XX = "$ set nover"

(and there is only ser verify)

Wim

[Robert Gezelter]

Wim,

I do not believe that partial assignments to strings were supported in the DCL READ command (the HELP text is specific--"symbol-name").

The lack of a reported syntax error is certainly a bug (although I do not have the time to verify it, or its age).

- Bob Gezelter, http://www.rlgsc.com

[John Gillings]

Wim,

(sigh), Yes, DCL has limitations. That particular limitation was extended several years ago (>5) in EDCL, available in V7.3-2 and above (why are you still on V7.3? -2 is guaranteed to be binary compatible with all non-privileged code - it's a trivial upgrade, well tested and very well understood). Sorry, there are no valid excuses for running V7.3. I'll accept there are some arguments for V6.2 and V5.5-2, but NOT V7.3.

If you're not prepared to install a later version, then use a different programming language that doesn't have the same limitations. Even if you don't have any other compilers, for the specific issue here, (copy the first 10 characters of the first record in a file into a specific symbol), you could write a less than 20 line MACRO32 program which would work on any OpenVMS system, any version, any platform.

[Wim Van den Wyngaert]

John,

This is a bank and lots of money flows thru the VMS platform. No way to upgrade (already on lots of unsupported stuff of Sybase and Reuters, core people gone with the wind) unless you sign a document that you will pay for all the damages. But bring a large check in that case. And a very rich sponsor.

BTW : it's 7.3 not fully patched.

Wim

[Ian Miller.]

*insert usual stuff about risks of running unsupported versions versus risk of upgrading*

Follow the advice of JG and use another language. The version of DCL you are running has limits and you have hit one of them.

[Wim Van den Wyngaert]

I went for the convert of the file before using it.

[John Gillings]

Wim,

That's a load of *&^%#*($

One would have expected that "a bank and lots of money flows thru" would have a much better appreciation risk management than demonstrated by insistence on remaining on an old, unsupported, and unsupportable version of an operating system, with numerous KNOWN problems.

How will they ever upgrade? What will they do if they hit one of the many bugs which have long since been fixed by patches that are tried and tested across the planet? How will anyone even diagnose problems on a system with unique version and patch levels? Don't they have test systems? What if they have a need for a new or fixed feature? (for example, longer command line buffers).

It seems obvious to me that the risk they think they're avoiding is FAR FAR less than the risks they expose themselves to by NOT upgrading.

I could never understand why some users would refuse to upgrade a supported, QA tested minor version, but would happily install and run with a hacked up, untested, unique patch when they hit a problem.

They same bank no doubt blindly downloads and installs all the patches that another software company throws at them...

This is a huge issue in the OpenVMS world. Far too many versions, and far too many people unnecessarily afraid of the upgrade bogeyman. It's up to professionals like YOU to educate those that make the decisions.

OpenVMS should NOT be fragmented into a zillion incremental versions. It makes everyone's life harder and GENERATES risk, rather than mitigating it.

[Jon Pinkley]

John,

Although I agree with you about running old versions, if you were at that site, and given the same restrictions, specifically, "guarantee that this upgrade won't break anything, or pay for all the damages", your only choices are to leave it alone or find another client.

Yes, you can try to educate them, and point out that the risks associated with old versions may be higher that the risk of upgrade. However, if something breaks because of a deficiency of the old system, that can't be blamed on you. Any problems that occur after the upgrade will be blamed on you whether they are in fact related to the upgrade or not.

"core people gone with the wind", this is another problem. What if there is something that breaks as the result of a change? And there is no documentation? Hopefully the those problems would show up before going live, but you often can't do a 100% test on a test system, and with a banking system, I would expect a lot of interaction with outside systems that would be hard to faithfully emulate.

They should definitely have a test system, but doing a good regression test/parallel testing, etc. is not a trivial matter. Although I would be interested in ideas about how those can be done with good coverage and minimal human resources.

I remember when I was in high school (~1973) we went to a NORAD (Air Defense) monitoring site as a school field trip for our Physics class. They were still using vacuum tube computers at that time (possibly due to reduced susceptibility to EMP). So there are some industries/markets that are extremely lethargic in their upgrade schedules. And they may not upgrade until it is a absolute requirement because of non-support of the replacement platform.

Jon

[Wim Van den Wyngaert]

Jon(h),

Sorry but can't give more details but it's worse than you think. Officially we are in the last year of VMS ...

Wim

[Jan van den Ende]

Wim,

>>>
Officially we are in the last year of VMS ...
<<<

And THAT at a bank that does not even dare to upgrade???

Are you at liberty to tell us what will be the replacement platform those "... lots of money flows thru" ?
Try to (let them!) explain how that "minor upgrade" will be less risky than the "giant move" of VMS v7.3 to v7.3-2, or, even better, v8.3 !!
And in the course of that, let them explain how they intend to be more trustworthy on the new platform, because somehow I doubt that the replacement would be NSK...

Then again, probably all caused by the lack of (visual) support for VMS by the current owners... Beuh.

just rambling....

Proost.

Have one on me.

jpe

[Wim Van den Wyngaert]

Jan

Most of the stuff is going to Sun. It concerns big trades and currency swaps. They will just do them in other systems (package based, need a lot more Sun hardware than VMS hardware).

But the last 20% will be tricky ...


Wim

[John Gillings]

Wim,

> Officially we are in the last year of VMS ...

Don't worry, I know plenty of people who've been in "the last year of VMS..." perpetually since 1990! ;-)

[Aaron Sakovich]

SOAPBOX:

It never ceases to amaze me at the haughty, condescending attitudes exhibited by some when a person comes here for help, and receives derision for being on an old version of VMS.

Folks, if the user is running some obsolete version of the OS or an app, tell him why his problem exists, suggest he upgrade because that problem was fixed in a later rev, but don't flame him! Some people are overwhelmed as it is, don't need scorn poured on them, and we will never garner converts to VMS if we ridicule people who come here for help!

Geezzzz.... (off my soapbox now...)

[Jon Pinkley]

Wim,

No guarantees this will work, but it does on 7.3-2 with a 8172 length string.

$ read x x_rec
$ y=f$fao("!10AS",x_rec) ! same effect as f$extr(0,10,x_rec) but may work places f$extract fails.

Can you see if that works on your version?

Jon

P.S. here's what I did on 7.3-2 (patched)

$ abc=f$fao("!8173*a")
%DCL-W-BUFOVF, command buffer overflow - shorten expression or command line
$ abc = f$fao("!8172*a") ! 8172 "a" characters 8173 gets buffer overflow
$ write/symbol sys$output abc
%RMS-F-SYS, QIO system service request failed
-SYSTEM-F-EXQUOTA, process quota exceeded
$ new=""
$ new=f$fao("!10AS",abc)
$ sho sym new
NEW = "aaaaaaaaaa"
$
$ write/symbol sys$output abc

[Karl Rohwedder]

Joe,

nice idea with F$Fao, helps on VAX:

$ lang=F$Fao("!1000*A")
$ kurz=F$Extract(0,10,Lang)
%DCL-W-BUFOVF, command buffer overflow - shorten expression or command line
$ kurz=F$Fao("!10AS",lang)
$ sh sym kurz
KURZ = "AAAAAAAAAA"

regards Kalle

[Jess Goodman]

In VMS 7.3 the DOCUMENTED size limit for a DCL symbol was 1024 bytes. With 7.3-2 and Extended DCL this was increased to 8192 bytes.

However a problem wither DCL or EDCL is that when using symbols that are close to these documented limits with most DCL commands you will get the dreaded:
%DCL-W-BUFOVF, command buffer overflow

For instance under VMS 7.3-2:
$ x_rec=f$fao("!8192*a")
%DCL-W-BUFOVF, command buffer overflow
$ x=f$fa("!8175*a") !longest that works
$ write sys$output f$len(x)
%DCL-W-BUFOVF, command buffer overflow
$ say := write sys$output
$ say "''f$len(x)'"
8175

So minor changes in the DCL commands you use can prevent or cause BUFOVF when you are *CLOSE* to the limit. Jon's choice of F$FAO instead of F$EXTRACT is another example of this.

READ and WRITE/SYMBOL work up to and even over the documented limit, but there's no way under VMS 7.3 to do anything else with a symbol size of more than 1024 bytes.

Another limitation under V7.3-2 is LIB$SET_SYMBOL and LIB$GET_SYMBOL only work for symbols up to 4096 bytes.

[Jess Goodman]

Karl,

I think you will find that:

$ kurz=F$Ex(0,10,Lang)

will work also.

The full text of the error message is, after all, "command buffer overflow - shorten expression or command line", and using abreviations is one way to shorten the command line.

[Hein van den Heuvel]

You're joking right?!

Watch check... noop, not 1-apr yet.

Hein.

[Jess Goodman]

No, I wasn't joking. The reason that Jon's
F$FAO("!10AS",x_Rec)
worked, when
F$EXTRACT(0,10,x_Rec)
didn't is becayse it is one byte shorter.

Using F$EX instead of F$EXTRACT saves 5 bytes. Using shorter symbol names is another way of shortening the command line.

The longest symbol that can be defined directly with F$FAO under VMS 7.3-2 is, I think, 8181 bytes:
$ f = "!8181*a"
$ x=f$fa(f)
Add extra spaces around the = or a second letter to a symbol name and it fails.

Now, if you thought I meant that you should depend on one of these commands working at the very edge of the DCL command buffer limit in operational code, then no - that would be a joke.

[John Gillings]

re: Arron's SOAPBOX comment (obviously meant for me)

I don't have any problem with people running old versions of VMS. As I said, there are legitimate reasons for running V7.3-2, V6.2 or V5.5-2 or even in some cases V4.7. I've spent many years supporting customers on all versions.

However, there are some versions that are just plain silly to run. Probably the worst is V7.0 - when released customers were specifically told not to run it in production, but I've still come across cases of people running it years later "because their application is only supported on V7.0". Nonsense!

V7.2 isn't quite so absurd, but it's not far from it! V7.2 was released in 2001. It's now 7 years old. In that time there have been many improvements and corrections. Apart from the newer major versions of OpenVMS there are also the "dash" releases -1 and -2. OpenVMS went to great pains to make sure these releases were cross compatible with V7.3. The wording of the is unequivocal:

"Enhancement Releases for OpenVMS contain enhancements to existing features and maintenance updates.
The version number increases to show a revision by using a dash (e.g., 7.2-1 for VMS)
Enhancement releases are shipped to service customers who have a valid service update contract, however, License Subscription service is not required.

Application Impact: The release may contain new hardware support, software enhancements and maintenance, but the changes are isolated and have no impact on applications. We are confident that the new release has 100% binary compatibility with the previous release. If an application compatibility problem is discovered, we will assign the problem a high priority and commit to providing a fix.

There is no need for ISVs to test on the new release or produce a new application kit."

In other words, if software is qualified on V7.3 then it is automatically qualified on V7.3-1 and V7.3-2 without the need for further testing. That is the guarantee of OpenVMS engineering.

What never ceases to amaze me is people who complain about a limitation or bug in an old version that's long since been fixed, but who refuse to install a guaranteed, qualified and historically stable fix for the problem they're reporting. What's the point of complaining about it if you're not going to accept the fix?

The other issue here is the FUD factor - people assuming that the risk of upgrading is worse than the risk of not upgrading. In a very stable environment that may be the case, but if you're actually experiencing problems, it clearly is not. You also need to factor in the lack of support from OpenVMS engineering.

No doubt there are people who could chime in with "I upgraded from 7.3 to V7.3-2 and X or Y broke, so it must the fault of the upgrade". Having worked for many years in a support role and seen numerous instances of such reports, the vast majority were NOT due to the upgrade itself, but mostly just the reboot itself. OpenVMS has a double edged sword of reliability. In many cases it's so long since the last reboot that various things have been forgotten or changed without realisation. A good tip is to perform a reboot before upgrading just to shake out any lurking bugs, and distinguish them from real problems associated with the upgrade itself.

When done in a professional, researched and planned manner, the vast majority of V7.3 to V7.3-2 upgrades (or indeed ANY OpenVMS upgrades) are quick, smooth and trouble free.

Run whatever version you like, but if you're not running a supported version, and you come across a problem that's already been fixed, expect to be told it's already been fixed. Yes?

[Hoff]

Derision here would be an inference, and not an intended implication.

Running an old software version or old hardware is not without cost, and older versions and older hardware can and does fail, and latent bugs can and will crop up. Older versions can and do avoid the upgrade costs, and avoiding upgrades is not without costs.

I've seen production sites slammed hard -- hard down, corruption -- because they didn't keep current on their software version and/or their ECOs or their hardware. John has also seen these. The panic is palpable.

And I've seen folks spend days figuring out, hey, the bug in version Vx.y is known and fixed in version Vx.y+n, and, well, we're going to have to pay for a backport or we're going to have to quick-upgrade. The result of spending some number of days (and possibly an associated partial or full outage) on the investigation doesn't change the results here. And worse, the quick-upgrade can occur under a "get it going NOW" deadline, and can open up exposures that a more controlled upgrade might have detected.

Certainly maintain your environment to meet your needs, but recognize that a decision to not to spend money on upgrades or ECOs is inherently equivalent to a decision to spend some lesser or larger amount of money at some later and usually unspecified date.

TANSTAAFL, as Mr Heinlein once wrote. There ain't no such thing as a free lunch. Staying on old versions is cheap for a while, though can quickly become surprisingly expensive, and the repairs are increasingly large and increasingly difficult to predict.

--

Written text can be difficult to interpret as the author had intended. I know John, and I seriously doubt any derision was intended. You'll have to trust me on this, but John knows how to get the cudgel out. If he's truly unhappy with something, you'll know it. I seriously doubt he intended derision.