Directory Change Notification - ODS-2/ODS-5

[Ishan Oshadi Jayawarden]

Hi everyone!

I'm looking for a way to automatically run a DCL script when a file is created in a directory.

Our main application creates reports in a directory every hour, and right now I have a batch job that runs on the hour, waits for a bit for the file to be created, scans the directory using F$SEARCH() to get the real file name, and the processes it. But I suppose this situation must be a common enough scenario, that there must be a better solution... less coupling between the file creator and others external apps that need to monitor them. Specially as the hourly schedule may change in the future, and there may not be any elegant schedule at all...

Can anyone help? a path leading to a solution in DCL would be perfect... we don't have any compilers installed.

If I would have to poll the directory, would it be a noticeable hit on the system load to do frequent F$SEARCH() operations on a big directory, with say, 10,000 files in it? The system is a standard DS10 Alphaserver. VMS 7.3-2.

Thanks in advance...
/Oshadi

[Veli Körkkö]

Can you change the "main application" so that it would create the said hourly report onto a pretty much empty directory?

That way occasional scanning for new files would be less expensive.

One might be able to use search lists to have the new files created in one directory (empty mostly) and then the post-processing app would move them to the other (crowded) directory

with suitable code one could probaly have a suitable lock with BLOCKING AST against the said directory file but you would really need something more than plain DCL

_veli

[Marc Van den Broeck]

Hi,

we have several jobs scanning directories.
Most of them scan every 2 minutes and as far as I can see no big load is involved. (DS20).

I use:
IF F$SEARCH("IN_DIR:somefile.DAT;-0") .NES. ""

As you can see ;-0 is added at the filename to make sure the oldest version is processed first. This may also be important to you if several files are created in a short time period.

Rgds
Marc

[Antoniov.]

Hi Oshadi,
first of all Welcome to vms forum :-)

Sorry, I don't understand your real question. The F$SEARCH() function is not an heavy work for any vms machine. I use some procedure scanning ever few minutes on entry level machine (DS10). If you have some dubt, set priority very low (e.g. SET PROC /PRIO=1).

To avoid scheduling too many job in batch queue you have two different solutions:
1) Create a new batch queue just for this work
2) Create scanning process with RUN SYS$SYSTEM:LOGINOUT. In this case you can sleep process instead closing and resubmitting.

AFAIK there is no service can notify you any change on directory.

Antonio Vigliotti

[Joseph Huber_1]

As others said, reduce the number of files in the directory, where new files are created. (several 10 thousands are generally problematic, not just for this purpose: directory caching,window turns etc.).
Then some DCL polling is o.k.

A clean solution, but needs more programming:
put an ALARM ACE on the directory (SET SECURITY /ACL=(ALARM=SECURITY,ACCESS=WRITE+SUCCESS)),
enable ACL alarms in audit (SET AUDIT /ANABLE=ACL), create a mailbox for security alarms, SET AUDIT/LISTENER=thismailbox, write a program which reads this mailbox (can be DCL), and selects the messages for this specific ACE to trigger the actions You want.

[Ishan Oshadi Jayawarden]

Hello everyone,
Thanks for the quick responses. Search paths will work well, as already the app uses a logical to access this directory. I was just trying to find non-polling solution (like dnotify or *ChangeNotification/ReadDirectoryChanges apis in other OS's), with minimal changes to the running system - it's online and mission-critical, really :)

I'd like to go with a blocking AST in time...

Antoniov: Thanks!!! :)
I just went to the SysAdmin I course last March, after working with a 2 node + quorum node cluster for a year ;)

Looks like I'll stick with a low-priority F$SEARCH() approach for now...

[Joseph Huber_1]

>>ChangeNotification/ReadDirectoryChanges apis in other OS's

I think ALARM ACEs on the directory are the VMS equivalents.

[Bojan Nemec]

Oshadi,

If your application uses logical names to access the directory, you can redefine the logical name to point to two directories. You do this with a equivalence name list. The application will create the file in the first directory. The pool procedure can search this directory, process files and rename them to the second directory. With this technique you have only unprocessed files in the first directory.

The second thing is the number of files in the directory. If you have 10,000 files in one directory it is possible that you encounter the RMS-F-DME error when doing F$SEARCH() on this directory.

Bojan

[Rob Young_4]

Google is your friend.

Don't discount Usenet. After all, Usenet
predates this forum by many years and
many of these problems have been solved
before (there is nothing new under
the sun).

With that is an intro, 5 minutes in
groups.google.com lead to an article
in comp.os.vms that referenced the solution
in vmsnet.internals

This links to a C program that will
do what you are after:

http://tinyurl.com/durmx

Enjoy!

Rob

[Jan van den Ende]

Ishan,

lots of good advice already given.
(and I like the Alarm Ace mechanism by Joseph!).
However, whatever method you will be using, please realise that there _IS_ some chance that you will test for a file, and find it, but if you start processing the file it will not yet have finished being written...

So, please make sure that it already IS available for processing.
usually, F$FILE_AT(file,"LCK") will do, or F$FILE_ATT(file,"EOF") .ne. 0

We also have such applics, generating ca 400 files in clusters of few to some tens, and occasionally faced strange behavior.
We added this test, and just for good measure reported any such coincidences.
We hit one or two a month, and in such cases a 5 second delay solved all issues.

hth,

Proost.

Have one on me.

jpe

[Ishan Oshadi Jayawarden]

Rob: Thanks! It would probably need some sort of blessing from my app vendor. But yes, this is exactly what I was after... too bad some tool doesn't come with the standard VMS distribution that can do this...

Jan: Very good point! I'd totally forgotten this on the VMS end... the processing I mentioned was actually COPY/FTP'ing the file in question to a linux box. My app on the linux side does check for complete files, but it seems that so far, the only thing that's saved me is the five-second wait-retry loop in the DCL script.

The whole intent of this is to get some info from the cluster to a database, and then to a web server; even a one-minute delay is acceptable...

[John Gillings]

Beware!

The C code referenced will trigger whenever any process accesses the referenced directory in any way. That includes READ accesses. So, you'll get numerous false alarms. Make sure your code can cope with this.

Also, as Jan has pointed out, you'll get triggered when a file & consequent directory entry is *created*, not when it's closed. You need to work out a mechanism for synchronising reader and writer.

Note that exactly the same code (no mods required) can be used to block until a particular (non-directory) file is touched. That includes and read access, and also the creation of a new version. If your writer can be modified slightly - all it needs to do is open & close a specific file after it's generated the report, you can use activity on that file as the trigger for your reader.

[Uwe Zessin]

> F$FILE_AT(file,"LCK")

I don't see that LCK is a valid argument on V7.3-2 and "LOCKED" refers to 'deaccess locked' - a feature from RSX that locks the file when it was not properly closed.

You also need an error handler if you use:
F$FILE_ATT(file,"EOF")
because the lexical is aborted with an error status when the file is open for write.

$ open/write f file.dat
$ write sys$output f$file_attributes ("file.dat","eof")
%SYSTEM-W-ACCONFLICT, file access conflict
\file.dat\
$
$
$ close f ! don't forget this ;-)

[Ishan Oshadi Jayawarden]

F$FILE_ATTR(f, "EOF") can help...
In my case, the files I'm interested in have (very convenient) end-of-file magic numbers... say, a "--- 000 ---" string on a line by itself at the end of the file. I can run a search command, and check $SEVERITY...

[Antoniov.]

Hi Oshadi,
f$file() function abort if file is locked by another use and you are not able to get any information. Here there is a little piece of code I used to retrive file allocation
$ IF F$TYPE(FAL).NES."" THEN DELETE/SYMB FAL
$ SET NOON !Avoid trap of error
$ DEFINE SYS$ERROR NLA0: !Avoid terminal msg
$ DEFINE SYS$OUTPUT NLA0:
$ FAL=F$FILE(FPN,"EOF")
$ DEASS SYS$OUTPUT !Restore msg
$ DEASS SYS$ERROR
$ SET ON !Restore trap
$ IF F$TYPE(FAL).EQS."" THEN GOTO SD03
It's just an example.
You can also use CDT (creation date) to know last created files.

Antonio Vigliotti

[Peter Barkas]

Newer version of the program referenced by Rob can be found at:

http://www.eight-cubed.com/downloads.html#watchdir

[Richard W Hunt]

As a variant thought...

If your application code is in any way subject to change, you would do better to have it send you something to notify you. Is the program run inside a shell that could detect its exit? Even if only the shell could be changed, that might be enough.

One method is to have it set an event flag in a Common Event Flag block. That would work pretty well if they are on the same system. This won't work all of the time if there is a cluster and the two processes could possibly run on different nodes.

Another method would be to create a mailbox in your application that needs to know when the file is created and have the application write the name of the new file to the mailbox. This would work in a clustered environment.

A third possibility involves some complex lock code, taking out an interest lock on the directory but allowing anyone to write to it anyway. You would need some AST-level code to note that someone wanted to update the directory and use that code to notify you when someone actually does change something.

The problem I see with repeated scans of your folder has to do with file system caches, least-recently-used schemes, and resultant performance degradation, albeit by a small amount. Your frequent scan of the folder will force the file system to keep that folder in your directory cache when you might have better uses for that memory. Particularly if it is a large (heavily populated) folder.

Making the file creator notify you will obviate the need for any scanning, which is a GOOD thing with respect to file system caches on a busy system.

Just a variant thought on how to make it work faster.

[Ishan Oshadi Jayawarden]

All good points... I'll take a closer look at the writer; it's possible that it would create a mailbox... but this writer process is created only when needed (once an hour for now), and exits in a few seconds (after the report generation).

Another application level solution might be possible: When all is said and done (i.e. the report generated, and the file is closed), the writer logs this in a central logfile like so:
"Created File: xxxxxxxx.LIS"

If I could just do a blocking read on this logfile... in DCL...
Otherwise I'd be polling this file doing Type/Page commands. As far as I can tell, DCL's READ is out? It write-locks the file even when it's been opened just for reading.

[Ishan Oshadi Jayawarden]

> Type/Page

Drat, I meant Type/Tail :)

[Antoniov.]

Hi,
you can control sharing access with /SHARE option of open statement.
See follow lines:
$ OPEN/REA/SHARE=WRITE SRC
Previous statement tells vms open file for read without lock.
However, if another process open file without permission like follow dcl command
$ OPEN/APPEND ...
no any other process can read file.

Antonio Vigliotti

[Ishan Oshadi Jayawarden]

I knew I should've checked "help open"...

[Ishan Oshadi Jayawarden]

On first glance this seems to be capable of doing what I want... though polling, it'll be sleeping most of the time anyway...

$ open/read/share f logfile
$loop:
$ read/nolock/end=reboot f ln
$! write sys$output ln
$! process line
$ goto loop
$reboot:
$ wait 0:0:2
$ goto loop

[Richard W Hunt]

The thing that bothers me about this is that you are working from the wrong end of the data flow path. Looking at the destination for a change is not going to really help because the thing doing the listening with that OPEN/APPEND action might actively block the report producer from working.

Usually, as a matter of good programming practice, it is better when possible to have the acting program (producer) send the notification to the reacting program. This allows the reacting program to be passive and not affect system performance. It also allows the reacting program to respond very quickly when the right signal is given.

If the two programs are on the same cluster, you might even consider THIS mechanism... let the reacting program suspend itself and have the action program send a RESUME. You can do that cross-cluster with DCL SET PROCESS/RESUME /ID=xxxx or something similar to that. If it gets "hung" you can do a manual resume with the same command.

[Antoniov.]

Hi Oshadi,
I'm not sure get your real intention.
I guess writer process do something like this:
$ open/write/share=read f logfile
$loop:
$ write f
$ ...
In this case your example can work as you expected.

Antonio Vigliotti

[Ishan Oshadi Jayawarden]

Hi Antoniov... what I'm doing here is reading a central logfile; all report generating apps record their output filenames in this logfile; so if I can read this log file line by line, in close-to real-time, I'll be all set. The bit of script I posted lets me do that...

So in effect, I'm down to polling a single file, and only for the last line only, (can I call it non-blocking IO in DCL? :) instead of polling a big directory.

It's not the most elegant solution, but one major advantage is that I don't need to modify any writers... this is a good thing for me because this script should (in theory, at least) work without any changes through application upgrades, or even on different sites...