topic Re: Directory Change Notification - ODS-2/ODS-5 in Operating System - OpenVMS

Directory Change Notification - ODS-2/ODS-5

Ishan Oshadi Jayawarden — Thu, 14 Jul 2005 20:42:35 GMT

Hi everyone!

I'm looking for a way to automatically run a DCL script when a file is created in a directory.

Our main application creates reports in a directory every hour, and right now I have a batch job that runs on the hour, waits for a bit for the file to be created, scans the directory using F$SEARCH() to get the real file name, and the processes it. But I suppose this situation must be a common enough scenario, that there must be a better solution... less coupling between the file creator and others external apps that need to monitor them. Specially as the hourly schedule may change in the future, and there may not be any elegant schedule at all...

Can anyone help? a path leading to a solution in DCL would be perfect... we don't have any compilers installed.

If I would have to poll the directory, would it be a noticeable hit on the system load to do frequent F$SEARCH() operations on a big directory, with say, 10,000 files in it? The system is a standard DS10 Alphaserver. VMS 7.3-2.

Thanks in advance...
/Oshadi

Re: Directory Change Notification - ODS-2/ODS-5

Veli Körkkö — Thu, 14 Jul 2005 21:59:14 GMT

Can you change the "main application" so that it would create the said hourly report onto a pretty much empty directory?

That way occasional scanning for new files would be less expensive.

One might be able to use search lists to have the new files created in one directory (empty mostly) and then the post-processing app would move them to the other (crowded) directory

with suitable code one could probaly have a suitable lock with BLOCKING AST against the said directory file but you would really need something more than plain DCL

_veli

Re: Directory Change Notification - ODS-2/ODS-5

Marc Van den Broeck — Thu, 14 Jul 2005 22:18:25 GMT

Hi,

we have several jobs scanning directories.
Most of them scan every 2 minutes and as far as I can see no big load is involved. (DS20).

I use:
IF F$SEARCH("IN_DIR:somefile.DAT;-0") .NES. ""

As you can see ;-0 is added at the filename to make sure the oldest version is processed first. This may also be important to you if several files are created in a short time period.

Rgds
Marc

Re: Directory Change Notification - ODS-2/ODS-5

Antoniov. — Fri, 15 Jul 2005 00:08:38 GMT

Hi Oshadi,
first of all Welcome to vms forum :-)

Sorry, I don't understand your real question. The F$SEARCH() function is not an heavy work for any vms machine. I use some procedure scanning ever few minutes on entry level machine (DS10). If you have some dubt, set priority very low (e.g. SET PROC /PRIO=1).

To avoid scheduling too many job in batch queue you have two different solutions:
1) Create a new batch queue just for this work
2) Create scanning process with RUN SYS$SYSTEM:LOGINOUT. In this case you can sleep process instead closing and resubmitting.

AFAIK there is no service can notify you any change on directory.

Antonio Vigliotti

Re: Directory Change Notification - ODS-2/ODS-5

Joseph Huber_1 — Fri, 15 Jul 2005 00:43:40 GMT

As others said, reduce the number of files in the directory, where new files are created. (several 10 thousands are generally problematic, not just for this purpose: directory caching,window turns etc.).
Then some DCL polling is o.k.

A clean solution, but needs more programming:
put an ALARM ACE on the directory (SET SECURITY /ACL=(ALARM=SECURITY,ACCESS=WRITE+SUCCESS)),
enable ACL alarms in audit (SET AUDIT /ANABLE=ACL), create a mailbox for security alarms, SET AUDIT/LISTENER=thismailbox, write a program which reads this mailbox (can be DCL), and selects the messages for this specific ACE to trigger the actions You want.

Re: Directory Change Notification - ODS-2/ODS-5

Ishan Oshadi Jayawarden — Fri, 15 Jul 2005 00:45:11 GMT

Hello everyone,
Thanks for the quick responses. Search paths will work well, as already the app uses a logical to access this directory. I was just trying to find non-polling solution (like dnotify or *ChangeNotification/ReadDirectoryChanges apis in other OS's), with minimal changes to the running system - it's online and mission-critical, really :)

I'd like to go with a blocking AST in time...

Antoniov: Thanks!!! :)
I just went to the SysAdmin I course last March, after working with a 2 node + quorum node cluster for a year ;)

Looks like I'll stick with a low-priority F$SEARCH() approach for now...

Re: Directory Change Notification - ODS-2/ODS-5

Joseph Huber_1 — Fri, 15 Jul 2005 01:03:23 GMT

>>ChangeNotification/ReadDirectoryChanges apis in other OS's

I think ALARM ACEs on the directory are the VMS equivalents.

Re: Directory Change Notification - ODS-2/ODS-5

Bojan Nemec — Fri, 15 Jul 2005 01:06:06 GMT

Oshadi,

If your application uses logical names to access the directory, you can redefine the logical name to point to two directories. You do this with a equivalence name list. The application will create the file in the first directory. The pool procedure can search this directory, process files and rename them to the second directory. With this technique you have only unprocessed files in the first directory.

The second thing is the number of files in the directory. If you have 10,000 files in one directory it is possible that you encounter the RMS-F-DME error when doing F$SEARCH() on this directory.

Bojan

Re: Directory Change Notification - ODS-2/ODS-5

Rob Young_4 — Fri, 15 Jul 2005 09:19:09 GMT

Google is your friend.

Don't discount Usenet. After all, Usenet
predates this forum by many years and
many of these problems have been solved
before (there is nothing new under
the sun).

With that is an intro, 5 minutes in
groups.google.com lead to an article
in comp.os.vms that referenced the solution
in vmsnet.internals

This links to a C program that will
do what you are after:

http://tinyurl.com/durmx

Enjoy!

Rob

Re: Directory Change Notification - ODS-2/ODS-5

Jan van den Ende — Fri, 15 Jul 2005 09:29:37 GMT

Ishan,

lots of good advice already given.
(and I like the Alarm Ace mechanism by Joseph!).
However, whatever method you will be using, please realise that there _IS_ some chance that you will test for a file, and find it, but if you start processing the file it will not yet have finished being written...

So, please make sure that it already IS available for processing.
usually, F$FILE_AT(file,"LCK") will do, or F$FILE_ATT(file,"EOF") .ne. 0

We also have such applics, generating ca 400 files in clusters of few to some tens, and occasionally faced strange behavior.
We added this test, and just for good measure reported any such coincidences.
We hit one or two a month, and in such cases a 5 second delay solved all issues.

hth,

Proost.

Have one on me.

jpe

Re: Directory Change Notification - ODS-2/ODS-5

Ishan Oshadi Jayawarden — Fri, 15 Jul 2005 09:44:21 GMT

Rob: Thanks! It would probably need some sort of blessing from my app vendor. But yes, this is exactly what I was after... too bad some tool doesn't come with the standard VMS distribution that can do this...

Jan: Very good point! I'd totally forgotten this on the VMS end... the processing I mentioned was actually COPY/FTP'ing the file in question to a linux box. My app on the linux side does check for complete files, but it seems that so far, the only thing that's saved me is the five-second wait-retry loop in the DCL script.

The whole intent of this is to get some info from the cluster to a database, and then to a web server; even a one-minute delay is acceptable...

Re: Directory Change Notification - ODS-2/ODS-5

John Gillings — Sun, 17 Jul 2005 19:02:19 GMT

Beware!

The C code referenced will trigger whenever any process accesses the referenced directory in any way. That includes READ accesses. So, you'll get numerous false alarms. Make sure your code can cope with this.

Also, as Jan has pointed out, you'll get triggered when a file & consequent directory entry is *created*, not when it's closed. You need to work out a mechanism for synchronising reader and writer.

Note that exactly the same code (no mods required) can be used to block until a particular (non-directory) file is touched. That includes and read access, and also the creation of a new version. If your writer can be modified slightly - all it needs to do is open & close a specific file after it's generated the report, you can use activity on that file as the trigger for your reader.

Re: Directory Change Notification - ODS-2/ODS-5

Uwe Zessin — Sun, 17 Jul 2005 22:10:11 GMT

> F$FILE_AT(file,"LCK")

I don't see that LCK is a valid argument on V7.3-2 and "LOCKED" refers to 'deaccess locked' - a feature from RSX that locks the file when it was not properly closed.

You also need an error handler if you use:
F$FILE_ATT(file,"EOF")
because the lexical is aborted with an error status when the file is open for write.

$ open/write f file.dat
$ write sys$output f$file_attributes ("file.dat","eof")
%SYSTEM-W-ACCONFLICT, file access conflict
\file.dat\
$
$
$ close f ! don't forget this ;-)

Re: Directory Change Notification - ODS-2/ODS-5

Ishan Oshadi Jayawarden — Sun, 17 Jul 2005 23:01:01 GMT

F$FILE_ATTR(f, "EOF") can help...
In my case, the files I'm interested in have (very convenient) end-of-file magic numbers... say, a "--- 000 ---" string on a line by itself at the end of the file. I can run a search command, and check $SEVERITY...

Re: Directory Change Notification - ODS-2/ODS-5

Antoniov. — Mon, 18 Jul 2005 00:33:19 GMT

Hi Oshadi,
f$file() function abort if file is locked by another use and you are not able to get any information. Here there is a little piece of code I used to retrive file allocation
$ IF F$TYPE(FAL).NES."" THEN DELETE/SYMB FAL
$ SET NOON !Avoid trap of error
$ DEFINE SYS$ERROR NLA0: !Avoid terminal msg
$ DEFINE SYS$OUTPUT NLA0:
$ FAL=F$FILE(FPN,"EOF")
$ DEASS SYS$OUTPUT !Restore msg
$ DEASS SYS$ERROR
$ SET ON !Restore trap
$ IF F$TYPE(FAL).EQS."" THEN GOTO SD03
It's just an example.
You can also use CDT (creation date) to know last created files.

Antonio Vigliotti

Re: Directory Change Notification - ODS-2/ODS-5

Peter Barkas — Mon, 18 Jul 2005 05:57:49 GMT

Newer version of the program referenced by Rob can be found at:

http://www.eight-cubed.com/downloads.html#watchdir

Re: Directory Change Notification - ODS-2/ODS-5

Richard W Hunt — Mon, 18 Jul 2005 08:02:53 GMT

As a variant thought...

If your application code is in any way subject to change, you would do better to have it send you something to notify you. Is the program run inside a shell that could detect its exit? Even if only the shell could be changed, that might be enough.

One method is to have it set an event flag in a Common Event Flag block. That would work pretty well if they are on the same system. This won't work all of the time if there is a cluster and the two processes could possibly run on different nodes.

Another method would be to create a mailbox in your application that needs to know when the file is created and have the application write the name of the new file to the mailbox. This would work in a clustered environment.

A third possibility involves some complex lock code, taking out an interest lock on the directory but allowing anyone to write to it anyway. You would need some AST-level code to note that someone wanted to update the directory and use that code to notify you when someone actually does change something.

The problem I see with repeated scans of your folder has to do with file system caches, least-recently-used schemes, and resultant performance degradation, albeit by a small amount. Your frequent scan of the folder will force the file system to keep that folder in your directory cache when you might have better uses for that memory. Particularly if it is a large (heavily populated) folder.

Making the file creator notify you will obviate the need for any scanning, which is a GOOD thing with respect to file system caches on a busy system.

Just a variant thought on how to make it work faster.

Re: Directory Change Notification - ODS-2/ODS-5

Ishan Oshadi Jayawarden — Mon, 18 Jul 2005 08:17:47 GMT

All good points... I'll take a closer look at the writer; it's possible that it would create a mailbox... but this writer process is created only when needed (once an hour for now), and exits in a few seconds (after the report generation).

Another application level solution might be possible: When all is said and done (i.e. the report generated, and the file is closed), the writer logs this in a central logfile like so:
"Created File: xxxxxxxx.LIS"

If I could just do a blocking read on this logfile... in DCL...
Otherwise I'd be polling this file doing Type/Page commands. As far as I can tell, DCL's READ is out? It write-locks the file even when it's been opened just for reading.

Re: Directory Change Notification - ODS-2/ODS-5

Ishan Oshadi Jayawarden — Mon, 18 Jul 2005 08:20:41 GMT

> Type/Page

Drat, I meant Type/Tail :)

Re: Directory Change Notification - ODS-2/ODS-5

Antoniov. — Mon, 18 Jul 2005 08:35:35 GMT

Hi,
you can control sharing access with /SHARE option of open statement.
See follow lines:
$ OPEN/REA/SHARE=WRITE SRC
Previous statement tells vms open file for read without lock.
However, if another process open file without permission like follow dcl command
$ OPEN/APPEND ...
no any other process can read file.

Antonio Vigliotti