Operating System - HP-UX
1748265 Members
3803 Online
108760 Solutions
New Discussion

Re: script to check files on landing directory

 
NDO
Super Advisor

script to check files on landing directory

Hi

 

I am trying to write a script to check for files that are arriving on a particular directory, but sometimes I get no output at all, because this files once they arrive they are picked up & processed therefore going to another directory. So how can I overcame this? This is what I wrote so far:

 

#!/bin/sh


echo " "
echo " Files arriving...."
ls -lrt /data/ICTPRD/bmd1/rating/data | tail -2
echo " "
echo " files processed..."
ls -lrt /data/ICTPRD/bmd1/rating/processed | tail -2

 

9 REPLIES 9
SoumitraC
Frequent Advisor

Re: script to check files on landing directory

Assuming a separate process is responsible for depositing the files and processing them, your script obviously has an inherent race condition.

 

What exactly is the requirement?

 

Assuming that you want to monitor/list the files that have "arrived" since you last checked and that the file names do not change as a result of the processing, you can do something similar to:

 

$ diff -r /data/ICTPRD/bmd1/rating/data          \
           /data/ICTPRD/bmd1/rating/processed | \
   sed -ne 's#Only in /data/ICTPRD/bmd1/rating/data: \(.*\)#\1#p'

 

Soumitra C
HP-UX Compilers
NDO
Super Advisor

Re: script to check files on landing directory

Hi
I dont quite understand your 3rd line, please can you explain, if yo dont mind

SoumitraC
Frequent Advisor

Re: script to check files on landing directory

The third line uses sed(1) to filter out the filenames from the recursive directory diff(1) that are present only in the "data" directory, as compared to the "processed" directory.

 

Since the assumption is that the "processing" is copying/moving files from the "data" directory to the "processed" directory, the directory diff would list out the files that have not been processed as yet.

 

Hope that explains.

Soumitra C
HP-UX Compilers
NDO
Super Advisor

Re: script to check files on landing directory

If that series of commands returns nothing means all files are the same on both directories?
Dennis Handly
Acclaimed Contributor

Re: script to check files on landing directory

>once they arrive they are picked up & processed therefore going to another directory

 

Why not fix the process that processes them to also give you a list of files?

NDO
Super Advisor

Re: script to check files on landing directory

Its an application that does that: pick up files on data dir, them take those files processes them & put them in processed dir
SoumitraC
Frequent Advisor

Re: script to check files on landing directory


@NDO wrote:
If that series of commands returns nothing means all files are the same on both directories?

That's right.

Soumitra C
HP-UX Compilers
bobjh
Advisor

Re: script to check files on landing directory

Does your process preserve file time stamps? If not, you could use perl to find all files modified since the last run.

Matti_Kurkela
Honored Contributor

Re: script to check files on landing directory

Can you change the directory where the files arrive initially?

 

For example, if you could make the files to arrive into /data/ICTPRD/bmd1/rating/incoming instead of /data/ICTPRD/bmd1/rating/data, then your script could check the incoming directory and move them to the data directory once your script has done everything it wants to do. Then the application could pick them up in the data directory exactly as it does now.

 

With this change, you could be certain that no files will be missed, because the application would not even see the files in the data directory before your script had moved them there. But your script would become a necessary part of the overall process: if your script would fail, the application would not get its files at all.

MK