DCL script to FTP files

[Craig A]

For those people who use COPY/FTP can I ask how you deal with potential network glitches that could cause the command to not complete?

How do you ensure that files are not left "hanging"?

Cheers

Craig

[Steven Schweda]

> How do you ensure that files are not left
> "hanging"?

What's a "hanging" file?

If the command hangs, then you can pretty
safely conclude that the transfer was not
complete. If the command completes, then its
status value should tell you whether the
command succeeded or failed. Right?

If you want a serious test, then suck the
file back in, and compare it with what you
sent. Or start using fancy checksums or
other data integrity tests. (A truncated
Zip archive, for example, will be identified
as corrupt by any reasonable UnZip program.)

Many things are possible. What's your goal?

[Craig A]

Steven

I take the view that that the network link will always fail at the most inopportune time.

Therefore, I wish to mitigate the effect of a transfer failing and, thus, potentially causing a remote trigger job from picking up a file fragment.

On that basis, I can think of no useful reason to use COPY/FTP rather than a script that has a degree of intelligence in it.

Craig

[Steven Schweda]

> Therefore, I wish to mitigate the effect of
> a transfer failing and, thus, potentially
> causing a remote trigger job from picking
> up a file fragment.

I always wonder how a "remote trigger job"
can distinguish between a partial transfer
and a complete transfer. Everything's
complicated.

Did the original inquiry here mention any
"remote trigger job"? Why work hard to solve
a problem which doesn't exist (here)?

> On that basis, I can think of no useful
> reason to use COPY/FTP rather than a script
> that has a degree of intelligence in it.

_I_ wasn't the one who was pushing COPY /FTP,
but it certainly "has a degree of
intelligence in it." On the other hand, to a
casual reader, it's not immediately obvious
where all the "intelligence" might be in the
scripts/fragments exhibited here, so far. I
can _imagine_ writing a fancy script which
would detect a hung transfer, but that script
might or might not use COPY /FTP.

If it does what you want, then use it. If
something else is better at doing what you
want, then use _that_. I don't care. In any
case, deciding exactly what to do often
requires knowing exactly what you want to do.
"How do I make FTP work" is not a detailed
enough problem specification to get me all
excited about the relative merits of using
COPY /FTP or some other FTP client.

[Jim_McKinney]

> I can think of no useful reason to use COPY/FTP

Don't know what the trigger is for your remote processing but it seems that if you want to drive it from the FTP client side you could just base the trigger decision on whether or not the COPY/FTP returned a successful status. Or, if you're driving the processing of the transferred file from the remote end you might choose to do something like transfer a FILE.TRIGGER (could even be zero blocks) upon a successful transfer status from the COPY/FTP of the FILE.DAT and have the remote end use the presence of that file as an indicator that the transfer was successful and as a trigger for processing.

[Craig A]

I developed a file trasnfer utility for one client and it ended up being used to transfer 10,000 files per day between both internal and external servers. (The latter via VPn links)

Each PUT was explicity checked for the expected output. The same goes for the RENAME command. I ASSUMED a problem and waited for evidence to demonstrate that the commands had been successful.

From time to time, an FTP would just hang. I modified the job to use a sub-process for the transfer while monitoring Direct I/O (iirc) - Anthing "stuck" was killed off and the job re-submitted.

We never once (in over 3 years) experienced a problem with remote jobs picking up incomplete files. They simply polled for the correct files and ignored everything else.

Using a flag/trigger file wouldn't be my solution. You would need to check that the flag was there and the file that it flagging on is there too. Also, what deletes the flag file? Remember that transfers will be going to many different types of servers and not all the SysAdmins of such boxes will be able to code solutions to deal with things like the housekeeping of flag files.

Personaly, I'd rather just get the main file to its ultimate destination safely, easily and reliably while being mindful of the fact that network glitches are a fact of life.

Craig.