DCL verify default setting

[Claus Olesen]

Is there a way to change the default of DCL verify to noverify? for say all processes within the same group (same uic)?

The openvms faq has a discussion on how to turn off verify using logical sylogin_verify but it is not a logical that loginout.exe appears to be sensitive to.

Many of our DCL processes have nothing to say on sys$output aside from "$set noverify" and I'd like to not get all these log files from processes that have nothing else to say. (If f$verify() had had an option that allowed it to work similarly to "recall/erase" in terms of not leaving a footprint then I could have used that).

A non DCL process that has nothing to say does not leave a (empty) sys$output file behind i.e. only if it has something to say will the file be created.

[Hoff · Accepted solution]

Batch verification default is on. The DCL default is generally off. There are no knobs for this.

GNV/bash may well have something here, I haven't looked.

The OpenVMS FAQ doesn't touch on f$verify nor anything related to SET VERIFY. Not sure where you're looking here with what you are reporting, but it's apparently either a stale reference, or it's not the same FAQ I'm looking at. The most recent copy is over at www.hoffmanlabs.com/vmsfaq, or at /node/1 over at the new site.

I do see a reference to this logical name over at the now-ancient and long-closed OpenVMS Ask The Wizard area. Specifically here:

http://h71000.www7.hp.com/wizard/wiz_2328.html

If this reference is similar to what you are seeing, the sylogin_verify logical name shown is a mechanism explicitly coded into your code, and is completely unrelated to anything OpenVMS might or might not be doing here. It's an entirely user-level and user-written implementation. No magic.

Oh, there is one subtlety in this area that might be useful in your quest. The following substitution is processed by DCL fully inside the context of the comment, and is intentional behavior:

$! 'f$verify(0)'

Now as for the log files, if you'd like to not get the logs, either don't create the logs, create the logs in a scratch area (and set up a purge tool), aim the logs at NLA0:, or other such. You could always find and delete zero-length logs, too. That's pretty easy DCL, using a simple f$search loop and a f$file to get the size. And a DELETE.

I'd probably be looking to automatically scan the various logs for error signatures and/or DIFFERENCES (akin to what DTM does, as that sort of model can be very handy for finding weirdnesses) and to get rid of what log files I don't care about automatically. And I do generally have verify enabled here, as it makes figuring out what happened a whole lot easier when something does go off the rails.

Stephen Hoffman
HoffmanLabs LLC

[DECxchange]

Hello,
This is just an idea, but maybe it would be possible to add a symbol or logical to sylogin.com or sylogicals.com that equates the word verify to noverify. I have actually never tried this.

I think it's possible in the Authorize utility to change the LGICMD to point to a particular login command file, other than the standard ones. Maybe this is the way to test this rather than risk affecting something critical that you would not want to affect.

Are these batch or interactive DCL files?

Another thing you could try is write a command file that runs in batch that would automatically clean up (delete or purge[/keep] ...) these files periodically. That way, you would not have to make any modifications to your current application. You could create a separate batch queue to do this.

If you don't like queues, you could write a program in a language like C, Fortran, etc. (what ever language you're using) that spawns DCL commands that will automatically delete or purge these files.

Anyhow, there should be a number of ways to get around this, to prevent your hard disc from filling up and to automate this task.

[Jess Goodman]

Claus,

I'm guessing that when you say "DCL processes" what you mean is BATCH processes. Interactive processes also use DCL but of course they don't create log files and for them, unlike batch processes, verify is turned off at process startup.

To turn verify off for batch processes add these two lines to the top of the login command procedure(s) of the users the batch jobs run under.

$ VERIFY = 'F$VERIFY(0)'
$ IF (F$MODE().NES."BATCH") THEN VERIFY = F$VERIFY(VERIFY)

However if your objective is to have a completely empty batch .LOG file unless the job "has something to say" this won't do it.

Batch processes also write out 7 lines of accounting information to the log file when they exit. You can reduce this to a single "job terminated" line by replacing or redefining EXIT to be "LOGOUT/BRIEF !" in the batch .COM file. But the only way I know to get an empty log batch log file is to replace/redefine EXIT to be "STOP/ID=0 !" but I really would not recommend that; it is not a clean way of ending a process.

[Hoff]

Good suggestions.

[[[This is just an idea, but maybe it would be possible to add a symbol or logical to sylogin.com or sylogicals.com that equates the word verify to noverify. I have actually never tried this.]]]

The Ask The Wizard topic shows how to do this, using a logical name.

The wrinkle here looks to be that the SET NOVERIFY command or the f$verify shows up in the log file when VERIFY is already enabled, and I'm inferring that having the command show up in the logs is badness.

[[[I think it's possible in the Authorize utility to change the LGICMD to point to a particular login command file, other than the standard ones. Maybe this is the way to test this rather than risk affecting something critical that you would not want to affect.]]]

Correct; a test account is certainly advisable, or testing on a non-production server. Or both.

Other related approaches can include a test for a particular connection source or particular serial device or a particular username in the LOGIN or SYLOGIN procedure. It's common to locate this sort of processing in SYLOGIN, for tasks such as this or for detecting and punting upon multiple logins or other such site-specific restrictions not directly supported in AUTHORIZE.

[[[Another thing you could try is write a command file that runs in batch that would automatically clean up (delete or purge[/keep] ...) these files periodically. That way, you would not have to make any modifications to your current application. You could create a separate batch queue to do this.]]]

That processing is needed regardless, in my experience. Cruft builds up, and it's best to automate the scanning for errors -- either a SEARCH or a DIFFERENCES or other such -- and to flag interesting" events via email, web page or other such. To de-cruft without specific intervention.

DCL search can target "-W-", "-F-", etc., strings, and then DIFFERENCES the results to weed out the expected cases of these. If the code is less integrated with OpenVMS (and not using standard message structures), some local processing can be required.

[[[If you don't like queues, you could write a program in a language like C, Fortran, etc. (what ever language you're using) that spawns DCL commands that will automatically delete or purge these files.]]]

I've tended to avoid that approach. If you're writing a program, use the provided callable APIs, and not spawn. Spawn (and the C system call) can and do look quite inviting, but are really nasty when it comes to overall performance (frequent spawn is a performance hit on OpenVMS), stability (failures on the spawn and command failures are more effort to field) and debugging (more processes, more "fun", and you can either use mailboxes to pass in commands (more work) or stub files (more work, more files), or you're constructing filenames and commands).

There are APIs here, ranging from the simple and direct lib$delete_file call and such -- and integrated functions in C for this, such as the C remove or OpenVMS-ish delete calls -- to more involved calls. Down the stack are the RMS and XQP calls. These are way more work to set up (until you locate an existing example and build a wrapper around FAB$V_DLT on $close and such), but way more flexible. Most folks use lib$delete_file.

Regardless of the use of spawn or lib$delete_file, do avoid tasks involving the local and home-grown parsing filenames. That way leads only to pain.

Purging implies logs use the same filename is in use, which can get interesting when file processing collides with file locks. File versions are a two-edged sword, particularly when back versions stack up. For stuff I want to keep, I tend to use my own sequence number or a name-embedded date-time or such. Figuring out what happened and when within a giant stack of same-name back file versions can be a slog.

[Claus Olesen]

Thank you for your replies.

'f$verify(0)' is ever so close to what I wanted. With that the log file is completely empty but unfortunately there. But it is better and I'll use it. I see it used also in for example the sys$manager login template file. And that it must be specified in every file for which the effect is desired. if I should guess at how it works then DCL prints DCL statements BEFORE they are executed but not until fully evaluated. And the single quotes turns f$verify into a statement that is not fully evaluated.

The processes that I wanted this for are created using $creprc's equivalent to
$ run sys$system:loginout /input= /out= /proc= /detach
and of the forever type i.e. never ending.

"logout/brief" is a good one too. In my testing I learned that the accounting info is printed but only with /detach. I'll probably not use it because the acc info might be usefull now that the log file will be there anyway. And because to eliminate the acc info might require adding a DCL exit handler in all the DCL files to ensure that the process always and only exits via logout/brief.

But this entire issue is not an issue if the image is a user image as in
$ run myimage /out= /err= /proc= /detach

In that case, if the process has nothing to say then no log file is created.

Sorry about stating that it was the OpenVMS FAQ I found a discussion about syslogin_verify. The site I looked at is the one that Google returns for "Tricks with F$VERIFY()". Besides, Google found quite a lot references to that name. Which I took as it being legit. My mistake.

[Hoff]

Ah, ok. Here's what turns up for that search string, for future reference and should Google rejigger its search again and/or should somebody "borrow" Martin's text...

http://pdv.pdv-systeme.de/users/martinv/VMS_Programming_FAQ.html

That's the "OpenVMS Programming FAQ" from Martin Vorlander (a umlaut in the surname, but that's not encoded correctly in ITRC), and not the "OpenVMS FAQ". Quite useful, of course. (As with the rest of OpenVMS and its messages and such, the full and complete specification of the associated text is often key to a correct diagnosis. Context is everything.)

I don't think you're going to get the best water digging in this an-entirely-empty log file well, and would think you'll find spending time digging a well rather closer to the automated log file processing and automated log file maintenance mentioned earlier will have a far better payback. That's certainly where I'd dig. I'd not expect to be able to reduce all log files down to nothing, and it's difficult to debug a log file that have been successfully crafted to the plateau of nothingness should that job then go off the rails. Errors lacking diagnostics are tough to debug.

[Jon Pinkley]

Claus,

I have to agree with Hoff, if you want empty log files, what's the point of having them? You may as well specify /nolog when you submit the jobs. But then there will be no log file to look at if there are problems. This is also true for the case you mention:

-------
But this entire issue is not an issue if the image is a user image as in
$ run myimage /out= /err= /proc= /detach

In that case, if the process has nothing to say then no log file is created.
-------

However if the process has something to say, it has no way to do so, unless you use some other mechanism (like mail).


If you want the "uninteresting" log files to be automatically deleted, and you have logic in your command procedures to determine it there is something you want to be saved, then you could for instance set a dcl symbol with a value when you want to save the file, and otherwise at the end of the batch job, check the flag, and it it does not indicate the intent to save the file, run ppf to get the file specification of the log file, and then issue a delete of the log file. This will mark the file for delete, and when the process rundown closes the file, it will be deleted.

To get the ppf program, see the following thread.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1138609

See attachment to response dated Jun 22, 2007 21:00:12 GMT. It has the source code to a program ppf that can return the specific file (including version) for the log file of a batch job.

Example:

$ delete_log = 1 ! assume we will delete
...
$ if error_condition
$ then
$ delete_log = 0 ! don't delete log file, so we can debug
$ endif

...

$_common_exit:
$ if delete_log
$ then
$ ppf
$ delete 'fid_file_name' ! this only marks for deletion, since the file is still open
$ endif
$ exit

When process is deleted, the file will be deleted.

However, this "solution" requires that you modify the procedures.

Alternatively, write a command file or program to examine and delete log files that are not interesting, as suggested by others.

Good luck,

Jon

[Claus Olesen]

The basic issue for us and our customer is that it would be less intimidating to discover say 10 log files in the log file directory after a restart as opposed to 700. And then when most are empty or just contain fluff then that can make anyone tired right then and there and not want to look at the next log file. I'm exaggerating. We do have 700 processes on our next delivery but we try to avoid DCL with this being one of the reasons and as such we get only about 70 log files after a restart.

[Jess Goodman]

I understand you Claus. The command file which executes as a detached process runs one or more programs that normally have no output. So you would prefer that normally there would not be any log file, even an empty one.

If the application program(s) need to output something due to an unusal event or some error condition in the program itself, only then do you want the process's log file to be created.

Also you don't care about logging for the DCL code that runs the program(s), probably because you are confident that there will be no problems with it.

The only way I know you can do this is to use /OUTPUT=NL: on the RUN or the equivalent $CREPRC output argument, and then in the the input command file add a "DEFINE/USER SYS$OUTPUT logfile_spec" before each program activation.

The downsides are:
* You will have a separate log file created each time a program is activated.
* After a program writes to SYS$OUTPUT the log file is created but it will be locked until the program exits.
* If program rundown never occurs for some reason (STOP/ID=n or system crash) its output may never get flushed to the file.

But if all you are worried about is missing any errors that could be signalled by your program(s), then all you need do is add a a "DEFINE SYS$ERROR file_spec" to the input command procedure.

Note that you can not specify this filespec with RUN /ERROR= or the $CREPRC error argument since you are running LOGINOUT.EXE.

This error filespec will be created the first time a program signals an error. If that program exits and another program later signals an error it will append to the same file.

[Jess Goodman]

I should have said:
* You will have a separate log file created each time a program is activated *if* the program creates any output.

[John Gillings]

Claus,

My command procedure template starts with:

$ verf='F$VERIFY(F$TRNLNM(F$PARSE(F$ENVIRONMENT("PROCEDURE"),,,"NAME")+"_VERIFY"))

and ends with:

$ EXIT 'stat'+(F$VERIFY(verf).AND.0)

This gives a mechanism for easily turning verification on for debugging. By default verification is turned of, and restored when the procedure completes, with no residual output. Using a logical name to control the switch means you can turn it on even for detached processes by defining the logical name in a high enough table (LNM$GROUP or even LNM$SYSTEM).

You can take care of automatically cleaning up log files using version limits. Even the presence of an empty log file can sometimes be a valuable debugging aid.

[John Gillings]

Just a thought...

If you really want to eliminate the log file, one potential solution is to implement a "log server". Run a detached process which creates a permanent mailbox to receive log data. Specify that mailbox as /OUTPUT for your other detached processes.

Of course, if you use it for multiple processes you need to make sure anything you write is appropriately tagged. The log server can time stamp the messages and decide when (or if) to write them. Maybe also manage sending mail, SMS or other alarm channels, depending on the content or coded flags in the message.

[Claus Olesen]

Thank you Jess. You nailed both the requirements in your first 3 sentences and a solution in the 4th sentence (why didn't I think of that?). Also, getting a new log for every run is not a drawback - that's how we want it.

I think we can use that i.e. as in
$ run loginout /input=($def/user sys$output proc.out;$run proc) /out=nla0: ...

Sure, sys$output is then no longer shared and flushed. But that's no worse but the same as with non DCL procs. And I already worked around that somewhat by making library routines that re-open sys$output for shared access and printf wrappers that flush and sync on every call. As loginout does. We don't log that much so that's fine. loginout does one thing that I have not been able to reproduce. that is make tbk and printf share the same target file without crippling each others output. I'd like to know what that magic is. But if we establish a rule to never directly write to sys$error then tbk's on sys$error are fine. Still, it would have been nice to have tbk's on sys$output too for the context provided by other output. But that's an entirely different issue.

Thank you very much for all your input.

[Jess Goodman]

Claus,

I know exactly what you mean about the problem with integrating program output and VMS signal output when they are both going to the same user-mode output stream.

I have atttached a macro subroutine, PUT_OUTPUT_MSG.MAR that can be used as a work-around for this problem. You call it exactly like you would call LIB$PUT_OUTPUT.

However you must change *all* program output other than signals to use this subroutine, or you only make the problem worse. So you would have to change all printf()s to be sprintf()s and then pass the output to this routine by descriptor.

[Hoff]

It might be appropriate to manage your own output stream from within server processes; the server opens and manages its own logfile. And these days, probably also writing messages out to the syslogd server and mayhap an opcom/auditing or such; to a centralized logging daemon.

In these environments, the only time something is written out to sys$output or sys$error -- beyond something such as a name and version display upon startup -- might be when something very bad happens, and when the badness isn't handled. This includes dispatching crash reports and the stack trace.

As the environment is scaled upwards, the various processes themselves are usually managed in a network using a scheduling package, and distributed management interfaces and tools.

Yes, the debugger and log files are nice and comfortable, and quite useful when things get severely weird. Though within most any production environment, the more the tasks and logs and management can be automated, the better.

[Claus Olesen]

Thank you, Jess. The problem with that approach for us is the amount of changes involved. Allthough they'd be safe to make without retesting. Perhaps that is "the magic" in loginout.
We also reported this problem to our vendor who reported it to HP engineering who just today replied that they acknowledge the issue and that it exists also with say Fortran, that it has existed for a long time (your attached mar file is dated 94) and that our report is the first for them on this. The reply also said to keep an eye on future release notes. There's hope.