Directory Change Notification - ODS-2/ODS-5

[Uwe Zessin · Accepted solution]

Well, thanks for stepping forward and asking a question ;-)

In a forum there are typically many read-only users that will learn from the responses as well. And even an expert has a chance to learn something new or see a topic from a different angle.

[John Gillings]

Some further experimentation...

As mentioned earlier, sitting on the serialisation lock will notify you of all accesses to the directory. I haven't been able to find a way to distinguish between readers and writers. I suppose you could use $GETLKI to trace back and find out who is asking, then examine the process, perhaps to identify the image? But then the objective here was to save CPU time. If this is a busy directory, the waiting process will end up doing a lot of work.

I also tried reading ATR$C_DIRSEQ, but that also gets incremented on every access, so it doesn't help.

Since directory modifications will always occur at the START of the file transfer, when you really want to know when it's completed, I think the most elegant solution is to use the serialisation lock of a file within the directory, and have the writer process touch the file AFTER it's written to the directory. For example, assume we create an empty file in the target directory called:

DIRECTORY.MODIFIED

The reader process can then:

$ loop:
$ WAIT_FOR [MYDIR]DIRECTORY.MODIFIED
$ process directory
$ GOTO loop

the writer looks like:

$ put file(s) in directory
$ TYPE [MYDIR]DIRECTORY.MODIFIED

The code for WAIT_FOR is similar to "watchdir" referenced above, coded to read the parameter with LIB$GET_FOREIGN.

Note that this will work across a cluster, and even across a network, as

$ TYPE node::dev:[dir]DIRECTORY.MODIFIED

will trigger the lock BLAST. Use

$ COPY/FTP file NL:

in an IP only environment.

[Richard J Maher]

Hi,

FWIW attached is how I do it. (The code is slightly different to that attached to a previous post.) There is a good discussion(s) in comp.os.vms about this topic, with some valuable input from Keith Parris and Jim Brankin. (Search for "BLASTed directory locks".)

If you want to know when a file is completed rather than started then any of the suggestions already mentioned in this thread would work.

I believe Connect Direct does something really tricky like create a second *.Trigger file after it has FTPed the main file. (Sells reasonably well I believe.)

Also IMHO it would be ludicrous to suggest that this strategy would be costly in terms of performance. Polling is anathema to me, but if it floats your boat then great. But unless you set your polling interval in Days then please don't tell me it's more efficient than the attached code. (And it's certainly less functional.) Stll each to their own.

Cheers Richard

--------------------------------------------------------------------------------
T3$WAITFR_FILE

Monitor an on-disk directory until the user-specified file has
been created. This service completes asynchronously.

For additional information about system service completion,
refer to the Synchronize ($SYNCH) service

--------------------------------------------------------------------------------
FORMAT T3$WAITFR_FILE [efn] ,filespec ,resultant-filespec
[,iosb] [,astadr] [,astprm]

--------------------------------------------------------------------------------
ARGUMENTS efn
OpenVMS usage: ef_number
type: longword (unsigned)
access: read only
mechanism: by value

Event flag that T3$WAITFR_FILE is to set once this service
detects that the user-specified file has been created. The efn
argument is a longword value containing the number of the event
flag; however, T3$WAITFR_FILE uses only the low-order byte. If
you do not specify efn, event flag 0 is set.

When T3$WAITFR_FILE begins execution, it clears the specified
event flag or event flag 0 if efn was not specified.

filespec
VMS usage: char_string
type: character-coded text
access: read only
mechanism: by descriptor--fixed length string descriptor

File specification, which may not contain wildcards, that
T3$WAITFR_FILE uses to search for the desired file. The
filespec argument is the address of a descriptor pointing to
the file specification. The maximum length of the file
specification is 255 bytes.

Additionally, the file specification may not contain a search
list logical name, a node specification or a device name that
refers to a device other than a disk device.

T3$WAITFR_FILE uses RMS to parse the filespec argument. If RMS
is unable to parse the file specification then T3$WAITFR_FILE
will place the secondary RMS error status (STV) into the first
longword of the iosb argument.

resultant-filespec

OpenVMS usage: char_string
type: character string
access: modify
mechanism: by descriptor--fixed length string descriptor

Resultant file specification that T3$WAITFR_FILE returns when
it has successfully parsed the specification in the filespec
argument. All concealed logical names will have been translated
in this expanded file specification.

iosb

OpenVMS usage: io_status_block
type: quadword (unsigned)
access: write only
mechanism: by 32-bit reference

I/O status block that is to receive the final completion status.
The iosb argument is the 32-bit address of the quadword I/O
status block.

T3$WAITFR_FILE sets the quadword to 0 upon request initiation.
Upon request completion, a condition value is returned to the
first longword; the second longword is reserved for future use.

Though this argument is optional, TIER3 strongly recommends that
you specify it, for the following reasons:

. If you are using an event flag to signal the completion of the
service, you can test the I/O status block for a condition value
to be sure that the event flag was not set by an event other
than service completion.

. If you are using the $SYNCH service to synchronize completion
of the service, the I/O status block is a required argument for
$SYNCH.

. The condition value returned in R0 and the condition value
returned in the I/O status block provide information about
different aspects of the call to the T3$WAITFR_FILE service.
The condition value returned in R0 gives you information about
the success or failure of the service call itself; the condition
value returned in the I/O status block gives you information
about the success or failure of the service operation.
Therefore, to accurately assess the success or failure of the
call to T3$WAITFR_FILE, you must check the condition values
returned in both R0 and the I/O status block.

astadr

OpenVMS usage: ast_procedure
type: procedure value
access: call without stack unwinding
mechanism: by 32-bit reference

AST service routine to be executed when T3$WAITFR_FILE
completes. The astadr argument is the 32-bit address of this
routine.

If you specify astadr, the AST routine executes at user mode
regardless of the access mode of the caller of the service.

astprm

OpenVMS usage: user_arg
type: longword (unsigned)
access: read only
mechanism: by value

AST parameter to be passed to the AST service routine specified
by the astadr argument. The astprm argument is the longword
parameter.

--------------------------------------------------------------------------------
DESCRIPTION
Control returns to your program immediately following successful
completion of this service. Once T3$WAITFR_FILE detects that
the file specified by filespec has been created, this service
will inform your program by setting the event flag specified by
efn and, if specified, delivering the AST.

For each VMS Process, there can only be one T3$WAITFR_FILE
service 'active' at any given time. Therefore it is not
currently possible to implement OR type waits with
T3$WAITFR_FILE. That is, you can not choose to wait for this
file OR that file.

Required Privileges

You must have read access to the disk and directory that you are
monitoring.

Related Services

T3$WAITFR_FILE_CANCEL



--------------------------------------------------------------------------------
CONDITION SS$_NORMAL The service completed successfully. The
VALUES directory watch has been initiated.
RETURNED SS$_INSFARG Insufficient call arguments specified.
Although astadr and astprm arguments are
optional they must still be specified as
zero.
SS$_ABORT A call to T3$WAITFR_FILE is already
active. Only one call to this service
per process, is permitted at any given
time. Review your code or cancel the
outstanding directory watch with
T3$WAITFR_FILE_CANCEL.
SS$_ACCVIO Filespec could not be read by the caller
or either the resultant-filespec or iosb
arguments could not be written to by the
caller.
SS$_BADPARAM One or more of the following conditions
is true: -
. filespec is longer than 255 characters
. filespec contained a wildcard * or %
. filespec contained a node name
. filespec contained a searchlist
SS$_UNSUPPORTED No FID could be obtained for the
directory specification or the specified
device is not a disk device.

Any errors returned by the following services:-
SYS$PARSE
SYS$ASSIGN
SYS$GETDVIW
SYS$ENQ

--------------------------------------------------------------------------------
CONDITION SS$_NORMAL The file that you have been waiting for
VALUES has been created.
RETURNED SS$_CANCEL The wait was cancelled by a call to
IN THE I/O T3$WAITFR_FILE_CANCEL
STATUS fab$l_stv Secondary RMS error status. This will
BLOCK only occur if T3$WAITFR_FILE failed
while attempting to call SYS$PARSE.

--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
T3$WAITFR_FILE_CANCEL

The cancel wait for file service will abort a file watch that
was previously set in motion by a call to T3$WAITFR_FILE.

--------------------------------------------------------------------------------
FORMAT T3$WAITFR_FILE_CANCEL

--------------------------------------------------------------------------------
ARGUMENTS
None.

--------------------------------------------------------------------------------
DESCRIPTION The cancel file wait service is used to prematurely terminate a
previous call to T3$WAITFR_FILE.

T3$WAITFR_FILE_CANCEL relinquishes all system resources
obtained by T3$WAITFR_FILE, sets the specified event flag, and
delivers the AST to the calling process, if one was requested.

In order to identify the abnormal termination of the wait for
file service T3$WAITFR_FILE_CANCEL sets the IOSB status to
SS$_CANCEL.

It is not a requirement of T3$WAITFR_FILE that this routine be
called. The equivalent processing, except for AST delivery and
event flag setting is implied at image exit.

Required Privileges

None.

Related Services

T3$WAITFR_FILE

--------------------------------------------------------------------------------
CONDITION SS$_NORMAL The service completed successfully.
VALUES
RETURNED

[Jim Geier_1]

This is an old thread, but one for which I would like to develop a solution, and as was stated in the original post, "a path leading to a solution in DCL would be perfect..."

I am running OpenVMS V8.3-1H1 on Integrity.

In one of the replies, it was stated:

A clean solution, but needs more programming:
put an ALARM ACE on the directory (SET SECURITY /ACL=(ALARM=SECURITY,ACCESS=WRITE+SUCCESS)),
enable ACL alarms in audit (SET AUDIT /ANABLE=ACL), create a mailbox for security alarms, SET AUDIT/LISTENER=thismailbox, write a program which reads this mailbox (can be DCL), and selects the messages for this specific ACE to trigger the actions You want.

Can someone give me some of the missing details for this solution. First, the reference to the command SET AUDIT /LISTENER=thismailbox. What is "thismailbox" or how does one get the name or identification of the mailbox? Would one do some thing like:

$ create/mailbox listener_mailbox
$ mailbox = f$trnlnm("listener_mailbox")
$ set audit/listener='mailbox'

When one reads the mailbox, how does one first filter out those that are not relevant, and second disect the messages? The suggestion is that this would be DCL, and that would be nice. As I recall, the messages are relatively complex.

I am assuming that one must read the mailbox. How does one handle (in DCL) the possible or even likely timeout in the read command (if no alarms are triggered)?

thanks,
--jim geier--

[abrsvc]

If a small change to the application creating the file is possible, why not try the following:

1) Create a mailbox for communication.
2) When the file has been created and closed, write the filename to the mailbox.
3) The "batch" job that does the FTP reads the mailbox.
4) Take the filename from the mailbox and "process" it.
5) Post a read to the mailbox. The job will be in LEF until the mailbox is written to.

Simple. No searching.

In order to handle the case where the system crashes or some other interruption occurs set it up with 2 directories as mentioned earlier. One to receive the files to be processed, and one to place them once processed so that the last step of processing is to rename the file to the "done" directory. This allows you to restart the processing side and have its initial function be clearing the directory of any pending files before processing the mailbox files.

Other than modifying the application to send the filename to the mailbox, the rest can be done with DCL.

Dan

[Jim Geier_1]

Dan,
There is no process doing an FTP in my case. I am trying, very simply, to have a DCL script know when a file is created in a directory, and when that happens, take some action. It appears that by adding a Security Alarm ACE on the directory, when a file is created in the directory, an Audit Alarm takes place. A DCL script can then create a mailbox, set the audit server listener to point to the mailbox, and then read the mailbox will get a record when any audit alarm takes place. Then the DCL will have to examine the record to determine (1) if the record is the result of a file creation or (2) some other alarm. If the record is the result of a file creation, it would be nice to know the file specification. I am looking for advice on how to do all of this in DCL, if possible, and particularly the correct setting of the ACE on the directory and how to parse the Audit Alarm record.

[John Gillings]

Jim,

>$ create/mailbox listener_mailbox
>$ mailbox = f$trnlnm("listener_mailbox")
>$ set audit/listener='mailbox'

This is basically correct, except that you'd be well advised to have a process reading the mailbox BEFORE declaring it as a listener device. It should be heavily protected and permanent to prevent DOS attacks. You should also SET AUDIT/EXCLUDE= to prevent deadlocks.

The risk is, if the listener process stops servicing the mailbox (or is too slow), and the audit server blocks trying to send audit messages, it will go into "paranoia" mode and start suspending processes that generate alarms. Maybe make the mailbox nice and fat to give yourself some extra headroom?

The messages are potentially large, binary and horribly complex, so interpreting them from DCL will be tricky, and slow. Making this work reliably for any system that has even a modest audit load will certainly be interesting.

Another option, which is uglier, but less of a system wide risk and notionally simpler to code... make sure audit messages are sent to OPCOM and poll the tail of the operator log file, looking for audits of interest - in your case I guess you're just looking for a match on the directory name?

A rough cut might be:

$ PIPE TYPE/TAIL/CONTINUOUS SYS$MANAGER:OPERATOR.LOG | @YourSearchProcedure

of course, you need some logic to detect log file rollover.

(might be better to start a new thread for further discussion)

[Robert Gezelter]

Jim,

Having written an Audit Server listener, I would not recommend the interface for this purpose.

As John Gillings has noted, this is a global interface. Thus, there can only be a single reader of the stream on a given OpenVMS node. Errors in the listener can effectively shut down the node, or worse. Setting up the interface requires privileges, which is a whole other conversation.

In short, it is an interface designed for system monitoring, not a particular application. It is by no means a message switch distributing system status information.

The approaches with F$SEARCH, or alternatively, checking when the directory has been accessed or modified however, so scale. They are strongly (emphasis STRONGLY) recommended.

- Bob Gezelter, http://www.rlgsc.com

P.S. ITRC collegiality note: IMHO, it would have been more appropriate to have this recent discussion as a separate ITRC thread, with a back reference to the earlier (2005) discussion.

[abrsvc]

Jim,

Perhaps I'm oversimplifying the case here, but you need to trigger actions based upon file creation correct? If so, I'm suggesting that the process that takes that action post a read on a permanent mailbox where the message is hte filename just created. The process(es) that create the file need to be modified to add the steps of opening that permanent mailbox for write and writing the filename to the mailbox. The messages will be small (filename only). This eliminates any need for filtering of messages or any processing overhead. The receiving process will be in a wait state until there is a message.

Did I miss something here?

Dan

[Hoff]

Based on the questions you're asking around mailboxes and events and asynchronous processing and on your general plan to try to use DCL to process event-driven binary data, well, you're headed for deep trouble.

DCL is a form of duct tape, and not known for its performance nor for its prowess with event handling nor for its capabilities with binary data.

You will want to (need to?) embrace the many limits that exist here, and simply poll for this directory data at an appropriate interval (an interval of a minute would cause negligible load on any OpenVMS box), or implement an application-defined scheme that triggers or that sends the notification along as the file or the data is promulgated, or a combination of these.

Or migrate to an application design based on web services, for instance. Something which would deal with this for you.

The whole concept of trying to optimize a DCL application in this fashion is arguably dissonant.

Knowing what little I do about DCL programming, I'd have some trouble with implementing your plan in DCL, and I wouldn't particularly want to hand the resulting implementation to a customer to maintain it. Polling at a reasonable interval would be far easier to implement, more maintainable, and with negligible overhead.

PleaseStartNewForumThreadNextTime++;