How to setup an email alert notification for failed SFTP jobs?

[Jan Shu]

Hi,

My DBA setup a sftp cron job to upload data to a remote sftp server. How to setup an email alert notification for failed SFTP jobs? so my DBA can resend the data transfer. Can you please help? Thank you.

Regards,
Jan

[James R. Ferguson · Accepted solution]

Hi Jan:

If you SFTP script returns zero for successs and non-zero for failure, then you simply do something like:

# /home/jan/script || mailx -s "Failure in script" root < /dev/null

Regards!

...JRF...

[Jan Shu]

Dear James,
Thanks for the reply. I asked my DBA if her sftp script returns zero and non-zero for job status. DBA isn't sure, but she shows me the sftp log, it has only the verbes of the sftp connection. I think by default a script always return zero or non-zero for its result, right? Is there a way to test it?

Thanks.

Regards,
Jan

[OldSchool]

sftp, like ftp, appears to always return a zero, provided there isn't an error in sftp itself.

testing via the command line, and "echo $?" will show 0 regardless, you can prove this by trying to copy an non-existent file on the command line. the "script" will have to be written such that it scans to output log or takes some other action to confirm the file transfered completely / successfully. based on that action, the script should then return a zero or non-zero exit code.....

[Jan Shu]

Dear OldSchool,

Thanks for the information. I will let my DBA test this. Please wait for my update soon. Thanks again.

Regards,
Jan

[James R. Ferguson]

Hi (again):

As OldSchool notes, the return value of (S)FTP isn't going to help. You need to capture the session dialog in a file and parse it for success or failure.

If, however, your (S)FTP script were written in Perl, the Perl module implementations allow simple tests like:

...
$sftp->put( $localfile, $remotefile );
if ( $sftp->error ) {
warn "Can't put '$localfile' -> '$remotefile': " . $sftp->error . "\n";
...

Regards!

...JRF...

[Jan Shu]

Dear James,

I will check with my DBA if their sftp script is in Perl or not. I will search this forum website and hopefully I can find some sample sftp scripts in this forum website.

Thanks again.

Jan

[Steven E. Protter]

Shalom,

You could go as simple as this:

If the sftp job fails send an email.

If its a shell script.

sftp or scp command ...
rc=$?
#This gets you the return code.

if [ $rc -ne 0 ]
then
echo "File transfer error" | mailx -s "subject" email@yuor.net
fi

SEP

[Jan Shu]

Dear Stephen,

Thanks a lot, I will try this too. Please wait for my update.

Have a nice day.

Regards,
Jan

[James R. Ferguson]

Hi (again) Jan:

Re-read OldShool's post. A simple return code from SFTP in a shell script is _not_ going to work:

Consider:

# cat ./mysh
#!/usr/bin/sh
sftp somehost << EOF!
get /no/file /tmp/nofile
quit
EOF!
RC=$?
echo "rc=${RC}"

# ./mysh
Connecting to somehost...
sftp> get /no/file /tmp/nofile
Couldn't stat remote file: No such file or directory
File "/no/file" not found.
sftp> quit
rc=0

Regards!

...JRF...

[Jan Shu]

Hi James,

My DBA says their sftp script is a shell script not in Perl. If the return code won't work in the sftp shell script and I don't want to ask DBA to rewrite the script in Perl, what else should I try?

Thanks,
Jan

[OldSchool]

as noted previously:

"the "script" will have to be written such that it scans to output log or takes some other action to confirm the file transfered completely / successfully. based on that action, the script should then return a zero or non-zero exit code....."

In other words: somebody's going to have to write / rewrite something to make this work.

[Jan Shu]

Hi James,

OldSchool wrote: "sftp, like ftp, appears to always return a zero, PROVIDED there isn't an error in sftp itself.
" So if there is an error, then it will return non-zero. And it should work for my sftp shell script to send failure alert notification.

Jan

[Jan Shu]

Just got OldSchool's new update. Thanks.

[OldSchool]

"So if there is an error, then it will return non-zero. And it should work for my sftp shell script to send failure alert notification"

No..No.NO!

sftp ALWAYS returns a zero, even if the commands fed to it fail. the only time you would get a non-zero is if something caused "sftp" itself to fail, like no space in process table.

Read and understand JRF's example. The transfer *FAILED* because of missing file, but the return code was 0 (indicating success)

[James R. Ferguson]

Hi (again) Jan:

Here's a quick example of how you _might_ parse the SFTP session to determine success or failure:

# cat ./mysftp
#!/usr/bin/sh
typeset RC
typeset SFTPLOG=/tmp/sftplog.$$
trap 'rm ${SFTPLOG}' EXIT
exec 2> ${SFTPLOG}
sftp smhadt1 << EOF!
get /no/file /tmp/nofile
quit
EOF!
grep -iq "couldn't stat" ${SFTPLOG} && RC=1 || RC=0
echo "rc=${RC}"
exit ${RC}

...customize to your needs...

Regards!

...JRF...

[Jan Shu]

Hi James,

Thanks. Is this example a Perl script? Will it work with my DBA's shell script? e.g. I copy/paste the example commands to my DBA's script on a HPUX 11.23 Oracle 8i server?

Regards,
Jan

[James R. Ferguson]

Hi (again) Jan:

> Is this example a Perl script?

No, it is a SHELL script. The '#!/usr/bin/sh' (first) line signals what interpreter to use --- the HP-UX Posix shell in this case.

> Will it work with my DBA's shell script?

You can certainly integregate and embellish the concept I presented. I would make the hostname, the source and the target file names arguments to pass, too.

Regards!

...JRF...

[Jan Shu]

Thanks, James. I will try this, please wait.

Jan

[Jan Shu]

Hi James/OldSchool/SEP,

I got an idea. How about comparing the check sum of the data files before and after the sftp transfer? If they are the same, then the sftp upload is OK, if the check sum is different, then it is failed. What do you think?

Thanks.

Jan

[James R. Ferguson]

Hi (again) Jan:

> I got an idea. How about comparing the check sum of the data files before and after the sftp transfer? If they are the same, then the sftp upload is OK, if the check sum is different, then it is failed. What do you think?

That's OK _as_long_as_ your not moving a text file from UNIX to WINDOWS or vice versa using the ASCII mode of transfer. When you do that, a carriage return character will helpfully be added or subtracted from each "line" rendering the checksums different.

Regards!

...JRF...

[Jan Shu]

Hi James,

Thanks for the reply. The sftp server is a UX server (not sure Sun or HPUX) and I will make sure we use binary mode to upload. Thanks again.

Jan

[OldSchool]

JRF said: "That's OK _as_long_as_ your not moving a text file from UNIX to WINDOWS or vice versa using the ASCII mode of transfer. When you do that, a carriage return character will helpfully be added or subtracted from each "line" rendering the checksums different."


while correct for ascii mode transfers in conventional FTP, some of the SFTP stuff, especially early open-source, don't make to dos2ux / ux2dos conversions. I won't go into how I happen to know about this ;)

[Jan Shu]

Thank you all for your helpful inputs on this.

Jan