Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

Directory Change Notification - ODS-2/ODS-5

 
SOLVED
Go to solution

Directory Change Notification - ODS-2/ODS-5

Hi everyone!

I'm looking for a way to automatically run a DCL script when a file is created in a directory.

Our main application creates reports in a directory every hour, and right now I have a batch job that runs on the hour, waits for a bit for the file to be created, scans the directory using F$SEARCH() to get the real file name, and the processes it. But I suppose this situation must be a common enough scenario, that there must be a better solution... less coupling between the file creator and others external apps that need to monitor them. Specially as the hourly schedule may change in the future, and there may not be any elegant schedule at all...

Can anyone help? a path leading to a solution in DCL would be perfect... we don't have any compilers installed.

If I would have to poll the directory, would it be a noticeable hit on the system load to do frequent F$SEARCH() operations on a big directory, with say, 10,000 files in it? The system is a standard DS10 Alphaserver. VMS 7.3-2.

Thanks in advance...
/Oshadi

hey ho, what do you know...
62 REPLIES 62
Veli Körkkö
Trusted Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Can you change the "main application" so that it would create the said hourly report onto a pretty much empty directory?

That way occasional scanning for new files would be less expensive.

One might be able to use search lists to have the new files created in one directory (empty mostly) and then the post-processing app would move them to the other (crowded) directory

with suitable code one could probaly have a suitable lock with BLOCKING AST against the said directory file but you would really need something more than plain DCL

_veli
Marc Van den Broeck
Trusted Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Hi,

we have several jobs scanning directories.
Most of them scan every 2 minutes and as far as I can see no big load is involved. (DS20).

I use:
IF F$SEARCH("IN_DIR:somefile.DAT;-0") .NES. ""

As you can see ;-0 is added at the filename to make sure the oldest version is processed first. This may also be important to you if several files are created in a short time period.

Rgds
Marc
Antoniov.
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Hi Oshadi,
first of all Welcome to vms forum :-)

Sorry, I don't understand your real question. The F$SEARCH() function is not an heavy work for any vms machine. I use some procedure scanning ever few minutes on entry level machine (DS10). If you have some dubt, set priority very low (e.g. SET PROC /PRIO=1).

To avoid scheduling too many job in batch queue you have two different solutions:
1) Create a new batch queue just for this work
2) Create scanning process with RUN SYS$SYSTEM:LOGINOUT. In this case you can sleep process instead closing and resubmitting.

AFAIK there is no service can notify you any change on directory.

Antonio Vigliotti
Antonio Maria Vigliotti
Joseph Huber_1
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

As others said, reduce the number of files in the directory, where new files are created. (several 10 thousands are generally problematic, not just for this purpose: directory caching,window turns etc.).
Then some DCL polling is o.k.

A clean solution, but needs more programming:
put an ALARM ACE on the directory (SET SECURITY /ACL=(ALARM=SECURITY,ACCESS=WRITE+SUCCESS)),
enable ACL alarms in audit (SET AUDIT /ANABLE=ACL), create a mailbox for security alarms, SET AUDIT/LISTENER=thismailbox, write a program which reads this mailbox (can be DCL), and selects the messages for this specific ACE to trigger the actions You want.
http://www.mpp.mpg.de/~huber

Re: Directory Change Notification - ODS-2/ODS-5

Hello everyone,
Thanks for the quick responses. Search paths will work well, as already the app uses a logical to access this directory. I was just trying to find non-polling solution (like dnotify or *ChangeNotification/ReadDirectoryChanges apis in other OS's), with minimal changes to the running system - it's online and mission-critical, really :)

I'd like to go with a blocking AST in time...

Antoniov: Thanks!!! :)
I just went to the SysAdmin I course last March, after working with a 2 node + quorum node cluster for a year ;)

Looks like I'll stick with a low-priority F$SEARCH() approach for now...
hey ho, what do you know...
Joseph Huber_1
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

>>ChangeNotification/ReadDirectoryChanges apis in other OS's

I think ALARM ACEs on the directory are the VMS equivalents.
http://www.mpp.mpg.de/~huber
Bojan Nemec
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Oshadi,

If your application uses logical names to access the directory, you can redefine the logical name to point to two directories. You do this with a equivalence name list. The application will create the file in the first directory. The pool procedure can search this directory, process files and rename them to the second directory. With this technique you have only unprocessed files in the first directory.

The second thing is the number of files in the directory. If you have 10,000 files in one directory it is possible that you encounter the RMS-F-DME error when doing F$SEARCH() on this directory.

Bojan
Rob Young_4
Frequent Advisor

Re: Directory Change Notification - ODS-2/ODS-5


Google is your friend.

Don't discount Usenet. After all, Usenet
predates this forum by many years and
many of these problems have been solved
before (there is nothing new under
the sun).

With that is an intro, 5 minutes in
groups.google.com lead to an article
in comp.os.vms that referenced the solution
in vmsnet.internals

This links to a C program that will
do what you are after:

http://tinyurl.com/durmx

Enjoy!

Rob
Jan van den Ende
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Ishan,

lots of good advice already given.
(and I like the Alarm Ace mechanism by Joseph!).
However, whatever method you will be using, please realise that there _IS_ some chance that you will test for a file, and find it, but if you start processing the file it will not yet have finished being written...

So, please make sure that it already IS available for processing.
usually, F$FILE_AT(file,"LCK") will do, or F$FILE_ATT(file,"EOF") .ne. 0

We also have such applics, generating ca 400 files in clusters of few to some tens, and occasionally faced strange behavior.
We added this test, and just for good measure reported any such coincidences.
We hit one or two a month, and in such cases a 5 second delay solved all issues.

hth,

Proost.

Have one on me.

jpe
Don't rust yours pelled jacker to fine doll missed aches.

Re: Directory Change Notification - ODS-2/ODS-5

Rob: Thanks! It would probably need some sort of blessing from my app vendor. But yes, this is exactly what I was after... too bad some tool doesn't come with the standard VMS distribution that can do this...

Jan: Very good point! I'd totally forgotten this on the VMS end... the processing I mentioned was actually COPY/FTP'ing the file in question to a linux box. My app on the linux side does check for complete files, but it seems that so far, the only thing that's saved me is the five-second wait-retry loop in the DCL script.

The whole intent of this is to get some info from the cluster to a database, and then to a web server; even a one-minute delay is acceptable...
hey ho, what do you know...
John Gillings
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Beware!

The C code referenced will trigger whenever any process accesses the referenced directory in any way. That includes READ accesses. So, you'll get numerous false alarms. Make sure your code can cope with this.

Also, as Jan has pointed out, you'll get triggered when a file & consequent directory entry is *created*, not when it's closed. You need to work out a mechanism for synchronising reader and writer.

Note that exactly the same code (no mods required) can be used to block until a particular (non-directory) file is touched. That includes and read access, and also the creation of a new version. If your writer can be modified slightly - all it needs to do is open & close a specific file after it's generated the report, you can use activity on that file as the trigger for your reader.
A crucible of informative mistakes
Uwe Zessin
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

> F$FILE_AT(file,"LCK")

I don't see that LCK is a valid argument on V7.3-2 and "LOCKED" refers to 'deaccess locked' - a feature from RSX that locks the file when it was not properly closed.

You also need an error handler if you use:
F$FILE_ATT(file,"EOF")
because the lexical is aborted with an error status when the file is open for write.

$ open/write f file.dat
$ write sys$output f$file_attributes ("file.dat","eof")
%SYSTEM-W-ACCONFLICT, file access conflict
\file.dat\
$
$
$ close f ! don't forget this ;-)
.

Re: Directory Change Notification - ODS-2/ODS-5

F$FILE_ATTR(f, "EOF") can help...
In my case, the files I'm interested in have (very convenient) end-of-file magic numbers... say, a "--- 000 ---" string on a line by itself at the end of the file. I can run a search command, and check $SEVERITY...
hey ho, what do you know...
Antoniov.
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Hi Oshadi,
f$file() function abort if file is locked by another use and you are not able to get any information. Here there is a little piece of code I used to retrive file allocation
$ IF F$TYPE(FAL).NES."" THEN DELETE/SYMB FAL
$ SET NOON !Avoid trap of error
$ DEFINE SYS$ERROR NLA0: !Avoid terminal msg
$ DEFINE SYS$OUTPUT NLA0:
$ FAL=F$FILE(FPN,"EOF")
$ DEASS SYS$OUTPUT !Restore msg
$ DEASS SYS$ERROR
$ SET ON !Restore trap
$ IF F$TYPE(FAL).EQS."" THEN GOTO SD03
It's just an example.
You can also use CDT (creation date) to know last created files.

Antonio Vigliotti
Antonio Maria Vigliotti
Peter Barkas
Regular Advisor

Re: Directory Change Notification - ODS-2/ODS-5

Newer version of the program referenced by Rob can be found at:

http://www.eight-cubed.com/downloads.html#watchdir
Richard W Hunt
Valued Contributor

Re: Directory Change Notification - ODS-2/ODS-5

As a variant thought...

If your application code is in any way subject to change, you would do better to have it send you something to notify you. Is the program run inside a shell that could detect its exit? Even if only the shell could be changed, that might be enough.

One method is to have it set an event flag in a Common Event Flag block. That would work pretty well if they are on the same system. This won't work all of the time if there is a cluster and the two processes could possibly run on different nodes.

Another method would be to create a mailbox in your application that needs to know when the file is created and have the application write the name of the new file to the mailbox. This would work in a clustered environment.

A third possibility involves some complex lock code, taking out an interest lock on the directory but allowing anyone to write to it anyway. You would need some AST-level code to note that someone wanted to update the directory and use that code to notify you when someone actually does change something.

The problem I see with repeated scans of your folder has to do with file system caches, least-recently-used schemes, and resultant performance degradation, albeit by a small amount. Your frequent scan of the folder will force the file system to keep that folder in your directory cache when you might have better uses for that memory. Particularly if it is a large (heavily populated) folder.

Making the file creator notify you will obviate the need for any scanning, which is a GOOD thing with respect to file system caches on a busy system.

Just a variant thought on how to make it work faster.
Sr. Systems Janitor

Re: Directory Change Notification - ODS-2/ODS-5

All good points... I'll take a closer look at the writer; it's possible that it would create a mailbox... but this writer process is created only when needed (once an hour for now), and exits in a few seconds (after the report generation).

Another application level solution might be possible: When all is said and done (i.e. the report generated, and the file is closed), the writer logs this in a central logfile like so:
"Created File: xxxxxxxx.LIS"

If I could just do a blocking read on this logfile... in DCL...
Otherwise I'd be polling this file doing Type/Page commands. As far as I can tell, DCL's READ is out? It write-locks the file even when it's been opened just for reading.
hey ho, what do you know...

Re: Directory Change Notification - ODS-2/ODS-5

> Type/Page

Drat, I meant Type/Tail :)
hey ho, what do you know...
Antoniov.
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Hi,
you can control sharing access with /SHARE option of open statement.
See follow lines:
$ OPEN/REA/SHARE=WRITE SRC
Previous statement tells vms open file for read without lock.
However, if another process open file without permission like follow dcl command
$ OPEN/APPEND ...
no any other process can read file.

Antonio Vigliotti

Antonio Maria Vigliotti

Re: Directory Change Notification - ODS-2/ODS-5

I knew I should've checked "help open"...

hey ho, what do you know...

Re: Directory Change Notification - ODS-2/ODS-5

On first glance this seems to be capable of doing what I want... though polling, it'll be sleeping most of the time anyway...

$ open/read/share f logfile
$loop:
$ read/nolock/end=reboot f ln
$! write sys$output ln
$! process line
$ goto loop
$reboot:
$ wait 0:0:2
$ goto loop
hey ho, what do you know...
Richard W Hunt
Valued Contributor

Re: Directory Change Notification - ODS-2/ODS-5

The thing that bothers me about this is that you are working from the wrong end of the data flow path. Looking at the destination for a change is not going to really help because the thing doing the listening with that OPEN/APPEND action might actively block the report producer from working.

Usually, as a matter of good programming practice, it is better when possible to have the acting program (producer) send the notification to the reacting program. This allows the reacting program to be passive and not affect system performance. It also allows the reacting program to respond very quickly when the right signal is given.

If the two programs are on the same cluster, you might even consider THIS mechanism... let the reacting program suspend itself and have the action program send a RESUME. You can do that cross-cluster with DCL SET PROCESS/RESUME /ID=xxxx or something similar to that. If it gets "hung" you can do a manual resume with the same command.
Sr. Systems Janitor
Antoniov.
Honored Contributor

Re: Directory Change Notification - ODS-2/ODS-5

Hi Oshadi,
I'm not sure get your real intention.
I guess writer process do something like this:
$ open/write/share=read f logfile
$loop:
$ write f
$ ...
In this case your example can work as you expected.

Antonio Vigliotti
Antonio Maria Vigliotti

Re: Directory Change Notification - ODS-2/ODS-5

Hi Antoniov... what I'm doing here is reading a central logfile; all report generating apps record their output filenames in this logfile; so if I can read this log file line by line, in close-to real-time, I'll be all set. The bit of script I posted lets me do that...

So in effect, I'm down to polling a single file, and only for the last line only, (can I call it non-blocking IO in DCL? :) instead of polling a big directory.

It's not the most elegant solution, but one major advantage is that I don't need to modify any writers... this is a good thing for me because this script should (in theory, at least) work without any changes through application upgrades, or even on different sites...
hey ho, what do you know...