RE: script to move ftp files once transfer complete

[Adam Noble]

Hi,

We have a problem. We have a script that picks up files from a holding area and moves them to another directory for processing.

The issue is this we need to ensure that we do not move files where the transfer is still taking place and therefore the file incomplete.

I thought fuser on the directory would be a good enough check but this is not the case as depending on the nature of the ftp connection it is not always persistant and we have moved files which are not complete.

Any idea how I can get round this.

Cheers

[Tingli]

If you run ftp, why cann't you remove the file just after the ftp is done with a successful call?

[Adam Noble]

problem is its not my company doing the ftp!!
I can't get them to change the process.

[James R. Ferguson]

Hi Adam:

You need to add some intelligence to your FTP script in any number of ways. In one way or another (and there are several ways), evaulate the success or failure of the transfer and upon success, rename/move the completed file.

Regards!

...JRF...

[Adam Noble]

I'm not doing the FTP thats the problem but I get the feeling your saying that its not possible to do what I want to do unless I am.

[OldSchool]

in your move process, you *could* do a long directory listing of the hold directory to a file...sleep for a bit...do a second listing to a second file.

using grep, throw out everything in the second file that doesn't match the first, which should give you a list of files that haven't changed during the period you were asleep.

you would only be depending on information you control and the frequency that you want to run it. you could vary the sleep period until you come up with an accurate list.

[Tingli]

Or:
du -s the holding directory.

[Steven Schweda]

Should I infer that some external entity is
using FTP to transfer files to your FTP
server? (I'm sure that it's all clear to
you, but some of us non-psychics can be (or
appear) rather dense.)

The right answer would be to change the
transfer protocol to include some
end-of-transfer indicator file, so that the
presence of, say, "fred.done" would imply
that "fred" was available. Or, wrap the
file(s) in a testable way (Zip, "tar", ...),
so that they could be checked for integrity
at the receiving end.

One might, for example, send a checksum file
after the real file, so that, say, "fred.md5"
would tell you thet "fred" was available, and
also let you check its integrity.

> I'm not doing the FTP thats the problem but
> I get the feeling your saying that its not
> possible to do what I want to do unless I
> am.

It's certainly easier to know when you're
done doing something when you're the one
who's doing it. More reliable, too. What
happens if the sender aborts a transfer
before a transfer is complete? How can the
receiver know the difference between a
complete transfer and an incomplete transfer?