Check return code via FTP

[Samuel_10]

Here's my current FTP script that I run from another script via a cron job every night.

cd /outgoing
ftp -n -v > logfile.log <<-EOF
open 127.0.0.1
user user mypassword
ascii
put $1 'FTPFILE'
bye
EOF

I would like to add a check to this so that if the put fails somthing else happens. If we get a return code of anythin not = to 0 then we want to do the following. Can this be done within the FTP or does another FTP have to be done? Need some help on this. PLease mention exact steps. Thanks.

ascii
put blankfile 'BLANKFILE'
quit

[A. Clay Stephenson]

Actually, FTP uses 2 as a good return. By far, the better method is to use Perl's Net::FTP module; you get error checking for free.

Try this Perl script:

ftpput.pl myfile1 myfile2
STAT=${?}
if [[ ${STAT} -eq 0 ]]
then
echo "All is well"
fi

It will login (change the host, user, and password to suit your needs) and then set ascii mode, next it will put the files passed on the command line and actually resend on failure (up to 5 times).

[Dave La Mar]

Samuel -
Actually, since you were smart enough to create the log file during the ftp process, your error checking can go against the log file.
We commonly do as you have done and check the log file for may different conditions, i.e. byte counts, not connected messages, etc.
You get the idea.

Best of luck.

Regards,

dl

[Samuel_10]

OK but I would have to do a separate FTP to implement the method of checking the logfile. Would I just have to do a grep on the file or something? I'd prefer to just use the FTP command directly.

[Steven E. Protter]

I have run a test.

ran a script that failed on a put but was able to login.

$? was still 0

Thats probably because the login didn't fail.

Even after a failed login the error code was zero.

A look at the help commands after an interactive ftp login reveals nothing that would seem to be helful

After the transfer though you can do this:

ls -la $1

At least then you'll get output to stdio that you can route to a file and process for errors later.

SEP

[Steven E. Protter]

While our Internet Connection was down I did some further research.

If you were transferring the file say via scp command line, you could check return codes after the transfer.

scp thisfile targethost://target_directory
rc=$?

if [ $rc -ne 0 ]
then
echo "Complain ... transfer failed"
fi

I have however noticed even on good transfers to a HP-9000 box having random seed problems that the return code has not always been zero.

I assume that once I resolve that issue my methodology will be reliable.

SEP

[A. Clay Stephenson]

You need to do a man ftpd for an explanation of replies sent to the ftp client upon receipt of commands. The actual replies considt of 3 digits but in almost all cases, all you need to do is examine the first (leftmost) digit and if it is a '2' then all is well. NOTE: This has NOTHING to do with the exit status of ftp.

[Samuel_10]

Ok I'm FTPing FROM HP to an MVS. So the return codes may not be the same on both. Looks like this is pretty tough to do.

[Samuel_10]

OK looks like unless I use Perl then there's no way "inside" the FTP script to do error handling.

[Dave La Mar]

Samuel -
We also ftp to and from a mainframe, but may of the error conditions are the same. In the ftp script, following the EOF, grep your log file for failure conditions or successful conditions.
If the grep is for failure, a RC=0 would idicate that particular failure was found. Using an if [ $RC = 0 ]
then
ux2dos logfile.log | uuencode logfile.txt | mailx -m -s "Failure in FTP" someone@somewhere.com
fi
That will email the log file as an attachment and notify an interested party as to the failure. The interested party will see the cause for failure in the log file.

On critical jobs we notify our operations department as well as others. If a call is necessary, ops calls the interested party.
In some cases we actually echo recoverin instruction to a file and then include it in the email as the body, i.e. (after the someon@somewhere.com) < recovery_instructions.file

I really believe you have all you need to perform what you desire.

Best regards,
dl

[Samuel_10]

OK I'll go back to my original email.

"PLease mention exact steps. Thanks."

This is what we want to do in response to a failure.

"ascii
put blankfile 'BLANKFILE'
quit"


Looks like the above need to be a separate FTP that needs to be run based on the result of a grep of the log. I was thinking of this before but I would have preferred to keep only the one FTP script. I see why someone mentioned Perl. I have no experience with PErl so that's not an option for this rigr now.

[Samuel_10]

OK so how would my if look if I was seeing if I got any results back from the grep. I'm going to search for a certain sequence in a line.

[A. Clay Stephenson]

I have a really strange question for you. If the put of the real file failed, what makes you think that a put of the blankfile will succeed?

This is a no-brainer even if you know no Perl because the changes are so easy.

1) On your HP-UX box, perl -v.
If that comes back with a version 5.6.1 or greater you are good to go as Net::FTP will be loaded for you otherwise should downlaod and install Perl from http://hpux.connect.org.uk/hppd/hpux/Languages/perl-5.8.3/.

2) I would now create a soft link for Perl in /usr/bin.
e.g. ln -s /opt/perl5/bin/perl /usr/bin/perl

3) The changes to the previously posted script are:

my $ftp = Net::FTP->new("remotehost",Debug => 0);
to

my $ftp = Net::FTP->new("127.0.0.1",Debug => 0);

------------------------------------
$ftp->login("cstephen","topsecret");
to
$ftp->login("user","mypassword");

and that's it.

If you insist upon trying to put the "blankfile" that is left as a dubious exercise. You could, of course, look at the exit status of the perl script and invoke a 2nd ftpput.pl blankfile and examine ${?} to see if that transmission was successful.

[Dave La Mar]

Here is a snip of the error checking in one of our ftp scripts.

grep 'Not connected' $FTP_LOG 1>/dev/null 2>/dev/null
echo $? | read RC
grep 'full' $FTP_LOG 1>/dev/null 2>/dev/null
echo $? | read RC1
grep 'fail' $FTP_LOG 1>/dev/null 2>/dev/null
echo $? | read RC2
grep 'Error writing' $FTP_LOG 1>/dev/null 2>/dev/null
echo $? | read RC3

if [ $RC -lt 1 -o $RC1 -lt 1 -o $RC2 -lt 1 -o $RC3 -lt 1 ]
then
echo "" >> $SCRIPT_HOME/messages
tail -5 $FTP_LOG >> $SCRIPT_HOME/messages
echo "" >> $SCRIPT_HOME/messages
echo "Problem with the CMSSVR server." >> $SCRIPT_HOME/messages
echo "" >> $SCRIPT_HOME/messages
echo "Gottschalks Operations ==> Review recovery procedures.">> $SCRIPT_HOME/messages
mailx -s "Nightly FTP failed for $SCRIPT_HOME" $MAIL_LIST < $SCRIPT_HOME/messages
exit

1. Of course grep -e may be used, but, in this case, I was interested in
the particular failure.
2. $MAIL_LIST is a variable (as may addresses as you'd like) like => export MAIL_LIST="someone@somewhere.com someone.else@somewhere.com"
3. $FTP_LOG is your ftplog.log
4. $SCRIPT_HOME is the path to your script directory.
5. tail -5 $FTP_LOG is generally all the error messages from the log file you will need in the body of the email to determine what happened.
6. In this example, there is no attachment as all is included in the email body.

Customize as you see fit.
Hope this helps.

Best regards,

dl

[Samuel_10]

The reason for the blank file would work is if the dataset on the MVS side ran out of space. The first file would fail partially but the blank file would work and therefore the other side would know, because of the presence of the blank file, that the first file was incomplete.

Yeah I got 5.6.1.

[A. Clay Stephenson]

But iff (iff and only if) the receiver ran out of space. The number of reasons for FTP failures is legion.

[Samuel_10]

Correct, but if the receiver doesn't receive the file that's ok. The problem we are trying to avoid is the reciever thinking they have received the complete file. They came up with the idea of us sending a blank file for them to check for.

If the FTP is successful I want to send another file. If it isn't, then I don't want to. Simple.

[Samuel_10]

So A. Clay,

OK so it seems the Perl method would be the "cleanest" maybe. However it looks like I would end up modifying the default script you sent to include another FTP to maintain that cleaness.

IF I use your method that means another FTP, which is basically like going with a GRREP method and based on the results doing another FTP.

[A. Clay Stephenson]

Not really because you could put the check in the same Perl script. It took me under a minute to make that change and I reduced the retries to only 2.

[A. Clay Stephenson]

I'm an idiot, I missed the attachment.

[Dave La Mar]

O.K. Samuel one last note on one of the method we use.

The destination knows when the file is complete when we send a "marker file". In one case the ftp jobs consists of sending up to 100 files. We cut out the file names sent from the log and send it as a recap.txt that they use as notification all is complete. Another job uses the byte count from the log file sent in another type of "marker file".
As stated, there are legions of errors that may be detected and legions of ways to let the destination know the ftp is completed. Some people send the ftp as one file name, then perform another ftp to rename the file when complete.

It's all a matter of what works for you and what your people are comfortable with. There is no real "best way". It is totally dependent on your environment, skill sets, and the users involved.

Again, best of luck.

dl

[Jos de Ruiter]

Hi Samuel,

Try this one:

export TMPLOG=/home/enz./.mixzftplog
export TRANSFER=/home/enz./.transfer.log

FTP=/home/operator/backup/ftpmixz.cmd
echo "user user passwd">$FTP
echo "lcd /home/enz.">>$FTP
echo "bin\nprompt\nmput filenaam">>$FTP
echo "quit">>$FTP
#
echo "\nBegin enz. " `date '+%d %b %Y %X` >>$TMPLOG
ftp -n -v ipnummer <$FTP 2>$TRANSFER 1>&2
cat $TRANSFER|grep "Transfer complete" >/dev/null
ERRORLEVEL_TRANSFER="$?"
if [ $ERRORLEVEL_TRANSFER -ne 0 ]; then
echo "\nTransfer FAILED: user probleem, connection probleem, enz " `date '+%d %b %Y %X` >>$TMPLOG
else
echo "\nTransfer complete " `date '+%d %b %Y %X` >>$TMPLOG
fi