Monitoring a dir for new files with perl - some suggestions, pls

[so.nimda]

Hi,

I have to monitor a directory for new incoming files and thereafter to sftp this file to another server for processing.

The directory that I am monitoring is a Windows system and I plan on using perl to do the scripting.

There's no fixed schedule as to when the files arrive and there's no fixed file size - may be big or small.

The plan is to monitor a dir (c:\incoming) and once a file is put there, to then sftp this file to a 2nd server for processing. Once successfully sftp-ed over, this same file is then moved to another dir as archive (c:\archive). The incoming dir will always be empty except for new incoming files.

Here are some of my concerns :

1) How do I effectively process the files that may be coming in immediately one after another ?

2) Should the incoming file name be written to a "list" and processing (i.e. sftp to the 2nd server) done from this list?

3) If it is a big file, how do I ensure that the file has been completely uploaded and the filename written to the "list" before proceeding to sftp it to the 2nd server?

I'm not sure if using the "list" method is the correct or effective way, so I'm hoping that someone would suggest a better and more efficient way of doing this.

Sorry for the long post and thanks in advance.

[James R. Ferguson]

Hi:

> 1) How do I effectively process the files that may be coming in immediately one after another ?

You could create an outer 'while' loop with a 'sleep' of an appropriate period every cycle. I would use an inner 'while readdir' loop to examine your directory's contents following each sleep cycle. Since you noted that each successfully processed file is removed from the directory, the presence of a file following a sleep cycle means that it's a candidate for processing.

> 2) Should the incoming file name be written to a "list" and processing (i.e. sftp to the 2nd server) done from this list?

I would. This makes the view of the directory and its contents as atomic as possible. The overhead of a list or array is going to be negligible. If a file is 'sftp'ed and the 'sftp' is successful, archive the file in your archive directory.

> 3) If it is a big file, how do I ensure that the file has been completely uploaded and the filename written to the "list" before proceeding to sftp it to the 2nd server?

I would 'stat' the file (probably for it's last 'mtime'). If the file's modification time is more than n-seconds less than the current time, the file is probably not actively being built or added to. Another way to test this would be to 'stat' the file for its size; wait a short time; 'stat' the size again; and if the difference is zero then the file could be considered to be in a static (unchanging) state.

My experiences are with UNIX and not Windows, so these last comments may not be entirely accurate.

Regards!

...JRF...

[so.nimda]

Hi JRF,

Thanks for the reply and moving the post to the correct place. Sorry about that.

Your suggestions are most useful and helpful and I would take them into consideration.

Thanks !

[Fredrik.eriksson · Accepted solution]

Not sure how well this works since I don't really have anything to test it properly with (Only have openvms machines and that perl version doesn't support SFTP.pm).

But, something like this should work~:
# Include files.
use File::Copy;
use Net::SFTP;
# Set up some basic information.
$recv = $ARGV[1];
$filepath = $ARGV[0];
($user,$pass,$host,path) = split(/[@:]/, $recv);
# Begin the never ending loop.
while(true) {
# Get all the files in $filepath/
@files = <$filepath/*>;
# If it's empty, ignore doing anything.
if(@files) {
# Set up a SFTP connection.
my $sftp = Net::SFTP->new($host);
# Login
$sftp->login($user,$pass);
# Loop throu the files found.
foreach(@files) {
# SFTP the files first
$sftp->put($_, "$path/$_");
# Then move them to the new directory
move($_, $ARGV[2]);
}
# Disconnect
undef($sftp);
}
# Sleep for a minute then do it all over again.
sleep(60);
}


This script takes 3 arguments from the commandline.
1: Where to look for files.
2: Where to SFTP files in a standard SCP fashion: USER:PASS@HOST:~/path/
3: Where to store the files locally

There is some things that should be written into this. Like error handling :P

But this might give you a pointer atleast :)

ps. If you're using keys then I would recommend you reading Net::SSH login method (Net::SFTP uses the same implementation) to figure out what needs to be done. ds.

Best regards
Fredrik Eriksson

[so.nimda]

Hi Fredrik,

Thanks very much for the code snippet !

It is most helpful to get me started.

Thanks !!