Script to move files

[Ron Kinner]

I have a compiled application for which there is no source code. I am writing a script to replace the application. Speed is not a problem so a script should do fine.

I have a separate application over which I have no control. At random intervals it opens up an FTP session and drops a file into the IN directory.

The process I am trying to replace does nothing more than look for files in the IN directory and move any files it finds from the IN directory to the OUT directory. Any file which appears in the Out directory will be worked on by another process (which I have already replaced with a script). The old process runs continually but my script will probably just run every 15 minutes or so.

I was told that the reason the first process didn't just drop files into the OUT directory was because they were afraid that the OUT process would start to work on a partial file before the FTP was complete. I don't know if this is true or not but it sounds reasonable.

Given that this is the case how can I write a script that doesn't move a file from the IN directory until the FTP is complete?

Obviously I can have the script look at netstat -a|grep ftp to make sure that there are no FTPs going on but is there a cleaner way or should I even worry about it?

Ron

[Sachin Patel]

Hi Ron,

Your script is running every 15 minuts so you can check timestemp of files(in IN directory) and move all files whose timestamp is older then current time. Considering that network is good.

for example you start your script at 1:15 then move all files whose time stamp is 1:12 or older.

Your idea is not bad either. you can add that as one more level checking.

Sachin

[Jacob D Levin]

Hi Ron,

what we've always done in this situation is to create the file with a temp name and then once it's on the system, rename it. So the NT process that dumps the file would name it file1.tmp and the last step of the process would rename it say file1.dat. then you would only move the files ending with .dat Hope this helps a little

Jake

[Patrick Wallek · Accepted solution]

You could also use fuser to see if there are any processes accessing the files. If fuser shows nothing then you are OK.

[A. Clay Stephenson]

Hi Ron:

The missing piece of your problem is a small script to determine if the file has been modified. Try this small Perl script, 'fileage.pl':

ls | while read FNAME
do
fileage.pl -m -s 600 ${FNAME}
STAT=$?
if [ ${STAT} -eq 1 ]
then
echo "File has been modified in the last 10 minutes"
else
echo "File has not been modified; safe to copy"
fi
done

If you like, you can add a -v argument and a 0 or 1 is printed to stdout to indicate whether the file has been modified. Normally, fileage.pl does its work silently and the exit status is used to indicate whether the file has been changed.

Fileage.pl -u will give full usage.

Regards, Clay

[Dave La Mar]

The ftp process could certainly rename the file after the put process allowing you to check for only the named files.
This would be a safe bet on large file transfers that could run more than 15 min.

just my .02.

Best of luck.
dl

[SHABU KHAN]

Ron,

Thinking along the lines of using netstat -a|grep ftp ... I would use lsof to do that kind of checking.

###If the VAR1 variable is not empty then this means that ftp is still in progress
VAR1=`lsof | grep | grep in.ftpd`


if [[ -n ${VAR1} ]]; then
print "FTP in progress, cannot move file"
else
print "NO FTP session in progress, safe to move file"
fi

If you know the username of who is doing the ftp then this logic will work.

I haven't tested this on a HP machine.

I also like Jacob's idea of renaming the file after an ftp is complete..

HOpe this helps !

-Shabu

[Ron Kinner]

Patrick gets the rabbit pelt and the full 10 points. I like his solution best since it appears to be easiest for me to work with. I just tested it and it works nicely and is much faster than my netstat -a|grep ftp idea.

The 7's are for solutions that would have worked if I weren't so lazy. (Don't have perl on this box and didn't really want to load it. Ditto for lsof. This is a special purpose box and it very basic without any frills.)

The 5's are for those who suggested changing the file name at the end of the FTP process. Remember I have no control over the process which ftps to me. I have to take what I get.

Thanks to all who replied tho.


Ron

[someone_4]

Hello
I was looking at your issue and thinking that the problem would be that the file keeps growing and you dont have to
Here is what I am doing.

1. testing the file size for file1 and giving it var filesize
2. sleeping for 5 secs to let the file grow.
3. testing the file size again and giving it var filesize1
4. going into a loop that will go testing on untill filesize1 is equal to filesize.
5. when the look is done and filesize1=filesize (meaning that the file has stopped growing) then it will perform an action. I did an echo .. you will do a mv.

#/usr/bin/sh
ll file1 | awk '{print $5}' | read filesize
sleep 5
ll file1 | awk '{print $5}' | read filesize1
until [ $filesize -eq $filesize1 ]
do
ll file1 | awk '{print $5}' | read filesize
sleep 5
ll file1 | awk '{print $5}' | read filesize1
echo "files are not the same"
done
echo " files are the same "
#_end

Richard

[Jordan Bean]

If you have lsof (http://hpux.cs.utah.edu/hppd/hpux/Sysadmin/lsof-4.61/) and speed is not crucial, then you can check to see if each file is open by another process. If so, skip; If not, move it. Let's assume that the incoming and outgoing directories are in the ftp pub directory, then do this (as root):

#!/usr/bin/sh
cd ~ftp/pub
find incoming -type f |while read file junk
do
if ! lsof $file >/dev/null 2>&1
then
mv $file outgoing
fi
done

This works only if lsof returns 0 if the file is open, and 1 if not or if it does not have enough permission to check.

[Ron Kinner]

Enough guys,

I like the fuser idea. It's short and sweet. I don't want to have to install stuff that's not there now and I'm not in a hurry. I have 48 hours to get the file processed so if a file is still being FTP'd I'll skip it and get it next time which will probably be in 15 minutes. The file are generally fairly short like under 10K so they don't take too long to transfer.

Ron

[CHRISTSHONDA CAMPBELL]

It's sounds like what you have in mind with checking for ftp would work fine. You could also use fuser -fu on the files and see if there is any output. Once you have checked for files in use, you can then copy the files over to the OUT directory.