Directory Change Notification - ODS-2/ODS-5

[Ishan Oshadi Jayawarden]

Hi everyone!

I'm looking for a way to automatically run a DCL script when a file is created in a directory.

Our main application creates reports in a directory every hour, and right now I have a batch job that runs on the hour, waits for a bit for the file to be created, scans the directory using F$SEARCH() to get the real file name, and the processes it. But I suppose this situation must be a common enough scenario, that there must be a better solution... less coupling between the file creator and others external apps that need to monitor them. Specially as the hourly schedule may change in the future, and there may not be any elegant schedule at all...

Can anyone help? a path leading to a solution in DCL would be perfect... we don't have any compilers installed.

If I would have to poll the directory, would it be a noticeable hit on the system load to do frequent F$SEARCH() operations on a big directory, with say, 10,000 files in it? The system is a standard DS10 Alphaserver. VMS 7.3-2.

Thanks in advance...
/Oshadi

[Veli Körkkö]

Can you change the "main application" so that it would create the said hourly report onto a pretty much empty directory?

That way occasional scanning for new files would be less expensive.

One might be able to use search lists to have the new files created in one directory (empty mostly) and then the post-processing app would move them to the other (crowded) directory

with suitable code one could probaly have a suitable lock with BLOCKING AST against the said directory file but you would really need something more than plain DCL

_veli

[Marc Van den Broeck]

Hi,

we have several jobs scanning directories.
Most of them scan every 2 minutes and as far as I can see no big load is involved. (DS20).

I use:
IF F$SEARCH("IN_DIR:somefile.DAT;-0") .NES. ""

As you can see ;-0 is added at the filename to make sure the oldest version is processed first. This may also be important to you if several files are created in a short time period.

Rgds
Marc

[Antoniov.]

Hi Oshadi,
first of all Welcome to vms forum :-)

Sorry, I don't understand your real question. The F$SEARCH() function is not an heavy work for any vms machine. I use some procedure scanning ever few minutes on entry level machine (DS10). If you have some dubt, set priority very low (e.g. SET PROC /PRIO=1).

To avoid scheduling too many job in batch queue you have two different solutions:
1) Create a new batch queue just for this work
2) Create scanning process with RUN SYS$SYSTEM:LOGINOUT. In this case you can sleep process instead closing and resubmitting.

AFAIK there is no service can notify you any change on directory.

Antonio Vigliotti

[Joseph Huber_1]

As others said, reduce the number of files in the directory, where new files are created. (several 10 thousands are generally problematic, not just for this purpose: directory caching,window turns etc.).
Then some DCL polling is o.k.

A clean solution, but needs more programming:
put an ALARM ACE on the directory (SET SECURITY /ACL=(ALARM=SECURITY,ACCESS=WRITE+SUCCESS)),
enable ACL alarms in audit (SET AUDIT /ANABLE=ACL), create a mailbox for security alarms, SET AUDIT/LISTENER=thismailbox, write a program which reads this mailbox (can be DCL), and selects the messages for this specific ACE to trigger the actions You want.

[Ishan Oshadi Jayawarden]

Hello everyone,
Thanks for the quick responses. Search paths will work well, as already the app uses a logical to access this directory. I was just trying to find non-polling solution (like dnotify or *ChangeNotification/ReadDirectoryChanges apis in other OS's), with minimal changes to the running system - it's online and mission-critical, really :)

I'd like to go with a blocking AST in time...

Antoniov: Thanks!!! :)
I just went to the SysAdmin I course last March, after working with a 2 node + quorum node cluster for a year ;)

Looks like I'll stick with a low-priority F$SEARCH() approach for now...

[Joseph Huber_1]

>>ChangeNotification/ReadDirectoryChanges apis in other OS's

I think ALARM ACEs on the directory are the VMS equivalents.

[Bojan Nemec]

Oshadi,

If your application uses logical names to access the directory, you can redefine the logical name to point to two directories. You do this with a equivalence name list. The application will create the file in the first directory. The pool procedure can search this directory, process files and rename them to the second directory. With this technique you have only unprocessed files in the first directory.

The second thing is the number of files in the directory. If you have 10,000 files in one directory it is possible that you encounter the RMS-F-DME error when doing F$SEARCH() on this directory.

Bojan

[Rob Young_4]

Google is your friend.

Don't discount Usenet. After all, Usenet
predates this forum by many years and
many of these problems have been solved
before (there is nothing new under
the sun).

With that is an intro, 5 minutes in
groups.google.com lead to an article
in comp.os.vms that referenced the solution
in vmsnet.internals

This links to a C program that will
do what you are after:

http://tinyurl.com/durmx

Enjoy!

Rob

[Jan van den Ende]

Ishan,

lots of good advice already given.
(and I like the Alarm Ace mechanism by Joseph!).
However, whatever method you will be using, please realise that there _IS_ some chance that you will test for a file, and find it, but if you start processing the file it will not yet have finished being written...

So, please make sure that it already IS available for processing.
usually, F$FILE_AT(file,"LCK") will do, or F$FILE_ATT(file,"EOF") .ne. 0

We also have such applics, generating ca 400 files in clusters of few to some tens, and occasionally faced strange behavior.
We added this test, and just for good measure reported any such coincidences.
We hit one or two a month, and in such cases a 5 second delay solved all issues.

hth,

Proost.

Have one on me.

jpe