File Manager Shell Script

[Ricardo Bassoi]

Hi,

I have a problem:
I need to check , each 5 minutes, the files names that are being generated at the dir /in
An example of the files names is acc12309889 and the number increase.
I need to check all the files acc in the /dir and execute a command like sciptname acc123XXXX for all the acc in the path /dir.
After 5 minutes we have more acc but I want to execute the script only for the new ones and not for the olds that I did before. I know that I need to send then to file with a list contents all file names but I??d like to implemented a better solution.

[Rodney Hills]

Could you create a folder called "/in_processed"? That way after you have processed a file, you could move it to that folder and it won't be re-processed again.

-- Rod Hills

[Mark Greene_1]

if you can modify your script (or add a wrap-around script) that tracks the last file handled that you can store somewhere, you can then use "find" to pick-up just the files you want:

find /in -newer $LAST_FILE -print |xargs -1 [your script] {}

where $LAST_FILE is the file name last handled by the script.

HTH
mark

[John Carr_2]

Hi

why not change the processing script to change the filename to filename.processed in the last line. Then when you execute the script execute it on all files except not containing the string .processed.

john.

[S.K. Chan]

Just my idea .. assumptions ..
1) new file that comes in has the "number" part of the filename always incremental.

First store the "ls|sort|wc" variable (call it $WC) to keep track of how many files you currently have, sorted.

After 5 minutes run "ls|sort|wc" again (store in $WCNEW). Take the value of (($WCNEW-$WC)) and if it's equal to zero (0) don't do anything because no new files have come in. If it's not equal to zero (ie 1 or more), then you would do "ls|sort|tail -${difference-between-$WC2-$WC}" to extract the filename of the new ones and only processed those.

[Thomas Schler_1]

Hi,

there are already good solutions the other authors gave to you.

Another possibility is to use the constraint that the number used to build the filename is increasing.

1) The 1st time your script runs, the filename of the last processed file should be stored somewhere in a reserved place, e.g. /etc/tmp/in/last_processed_file.

2) The 2nd time your script runs, a list of filenames should be created with e.g. 'ls -1t'. Process this list of filenames starting at the top, and stop the process when the actual position of the list of filenames shows the filename stored in /etc/tmp/in/last_processed_file.

3) Before ending the script, store the filename of the last processed file in /etc/tmp/in/last_processed_file.

Maybe, you want to choose this solution.

[Ricardo Bassoi]

Hi All,

Thanks by all the tips. I cannot change the file names extension and not copy to another dir.
In the attachment I have the list of file names that I collected just using:
ls acc* > processed
Now I need to execute a program:
scriptname accXXXXX
for each one of the file name in the list.
I need to check the last one and at the next time only execute the scipt again for the new ones

[Peter Kloetgen]

Hi Ricardo,

just a few more thoughts:

man touch

you can create a reference file with touch command which you can give the desired time stamp. After this you use the find- command to find newer files than your reference file.


man cut

if your file names are allways the same way, use the cut- command to get what you need: An increasing number...

echo filename | cut -c4-

cuts out only the characters beginning from 4 to the end. What you get is a number which you can use in a if- construction:

if [ number -gt reference_number ]
then
desired_action
fi

Allways stay on the bright side of life!

Peter

[Sachin Patel]

Hi Ricardo

Now you have file processed.

for i in `cat processed`
do
script ${i}
done

That will run script for all names in processed file

to get the last entry from processed
last -1 processed.

Sachin

[Sachin Patel]

Sorry it is
tail -1 processed (to get last entry).

using same loop you can move processed files in to different dir. or if you don't want to move files then

ls acc* |sort > processed
do the process on each file as I said in previous reply.

then second time
ls acc* | sort > processed1

diff processed processed1 > newprocessed

run script on newprocessed now.

before finishing join processed and newproced. now third time you will have processed is list of acc* which you pass on first on second phase.

Sachin

[Mark Greene_1 · Accepted solution]

you can do this without modifying either your existing script or any file or directory names:

run this script, which will call your existing one. this script will manage which file names to run your exising script with:

#!/bin/ksh
if [ -s /tmp/LAST_FILE ]; then
echo `ls -ltr |tail -1|tr -s " "||cut -d" " -f9` >/tmp/LAST_FILE
fi

LAST_FILE=`cat /tmp/LAST_FILE`

FILE_LIST=`find /in -newer $LAST_FILE -print`

for FILE_NAME in `echo $FILE_LIST`; do
your_script $FILE_NAME
done

echo $FILE_NAME >/tmp/LAST_FILE
exit


the "find" will isolate the files you've not done, and by running the lists to variable, this will eliminate any race-conditions between processing file and new files getting created.

HTH
mark