Shell script for multiple uploading

[rhansen]

Hello,
Below is the shell script I am using to upload files to a remote server using sftp. The shell script calls an expect script for this.
The problem I am facing is weird, the file is uploaded to the remote server on first attempt however the filesize is zero. It uploads fine on second attempt.

Can someone please help me in adding logic to the script so that I can try atleast 3 times to upload and check the filesize on the remote server before confirming it is a success. I have the expect script to check filesize but if someone could help me in the main shell script that would be appreciated.

#!/usr/bin/ksh
typeset host_id="server.test.org"
typeset port_number="/22"
typeset user_id="test"
typeset password="password"
typeset HOME="/home/test"
typeset TMP=$HOME/tmp


cd $HOME

# get all the IDs in a file id.conf
awk '{
if (NF > 0) {
c = substr ($1, 1, 1)
if (c != "#") { # skip comments
print $1
}
}
}' id.conf >$TMP/id.conf

while read line
do
# test
remote_directory=$HOME/submission/test/$line
source_directory=$HOME/toremote/test/$line
cd $source_directory
for filename in *
do
if [[ -f ${filename} ]]
then
filename_abs=$directory/$filename
$HOME/upload_file.exp $user_id $password $host_id $remote_directory $filename
if [[ $? -eq 0 ]]
then
echo "$(date): Upload successful
else
echo "${date}: Upload failed
fi
fi
done
done < $TMP/id.conf

[James R. Ferguson]

Hi:

> the file is uploaded to the remote server on first attempt however the filesize is zero. It uploads fine on second attempt.

It seems that you need to look at the called script to see why that would be the observation. When you run the calling shell script you might do so with 'set -x' to trace execution and the evaluated arguments. This might show something. For example, you do:

# filename_abs=$directory/$filename

...but never define 'directory' nor do you ever use 'filename_abs'.

I would also not use a variable named 'HOME' since this is normally defined by your login and exported into your environment.

> Can someone please help me in adding logic to the script so that I can try atleast 3 times to upload and check the filesize on the remote server before confirming it is a success. I have the expect script to check filesize but if someone could help me in the main shell script that would be appreciated.

I think you should concentrate on finding out _why_ two passes are required rather than adding code to "band-aid" a problem.

Checking file sizes of text files on different platforms may yield different sizes by virtue of the end-of-line controls (with and without a carriage-return).

Regards!

...JRF...

[rhansen]

Thanks for pointing out the flaws in the code. I have removed the line filename_abs=$directory/$filename. The error that I get after the first upload is Unable to stat destination after creation (no such file).

I can manually upload the file first time but not with the shell and expect script.
Any thoughts??

[James R. Ferguson]

Hi (again):

> The error that I get after the first upload is Unable to stat destination after creation (no such file). I can manually upload the file first time but not with the shell and expect script.

OK, so _exactly_ how do you "manually upload"? Please show the actual commands.

I assume that it is the 'expect' script that gives the error message, "Unable to stat...". It would be useful to see all of that code, too.

Regards!

...JRF...

[rhansen]

Here is the expect script

#!/opt/expect/expect
set argc [llength $argv]
set user_id [lindex $argv 0]
set Password [lindex $argv 1]
set host_id [lindex $argv 2]
set remote_directory [lindex $argv 3]
set source_directory [lindex $argv 4]
set directory [lindex $argv 5]
set filename [lindex $argv 6]


spawn /usr/local/bin/sftp2 $user_id@$host_id\#22
expect "Password:"
send "$Password\r"
expect "sftp>"
send "cd $remote_directory\r"
expect "sftp>"
send "lcd $source_directory\r"
expect "sftp>"
send "put $filename\r"
expect "sftp>"
send "quit\r"
exit


Note one change/typo in the main shell script.
$HOME/upload_file.exp $user_id $password $host_id $remote_directory $source_directory $filename.

I had missed the source_directory earlier.

Thanks for your help.

[rhansen]

Also I was wondering, if there is a way to just execute the uploading twice since it is working.

Thanks.

[James R. Ferguson]

Hi (again):

You pass six (6) arguments to your 'expect' script, but it expects (no pun intended) to receive seven (7).

Your shell script passes:

$user_id $password $host_id $remote_directory $source_directory $filename

Your 'expect' script expects to receive:

user_id, Password, host_id, remote_directory, source_directory, directory, filename

You failed to pass the 'directory' which leaves 'filename' undefined in the 'expect' script.

Regards!

...JRF...

[VK2COT]

Hello,

Firstly, you are looking for seven
command-line arguments (argv 0 to argv 6)!

Instead of reinventing the wheel, why don't
you use readily-available open-source tools:

a) rsync over ssh

b) Mscp: http://toic.org/projects/mscp/

c) lftp

and many more.

Cheers,

VK2COT

[rhansen]

James,

Thanks for the reply. Actually that was a typo again by me for directory. I am passing 6 arguments through shell as well as expect script. I am posting the relevant code again to make things clear.

set argc [llength $argv]
set user_id [lindex $argv 0]
set Password [lindex $argv 1]
set host_id [lindex $argv 2]
set remote_directory [lindex $argv 3]
set source_directory [lindex $argv 4]
set filename [lindex $argv 5]

Part of shell script.
$HOME/upload_file.exp $user_id $password $host_id $remote_directory $source_directory $filename

As I mentioned before the scripts does tranfers the file first time with 0 bytes and works fine on the second attempt (file size matches the file uploaded).

Thanks.

[James R. Ferguson]

HI (again) rhansen:

> The error that I get after the first upload is Unable to stat destination after creation (no such file).

This suggests that we still haven't seen all the code involved. What does your '/usr/local/bin/sftp2' look like?

In fact, for clarity, it would be nice to re-post all of the code with the modifications you have made to this point.

Your 'expect' script assumes that the remote and local directory changes succeed. For that matter, the shell script assumes the same. You should _test_ the results of these operations and take action if they fail.

It would seem much easier to build a shell script that uses 'scp' and public keys to eliminate passwords in code and eliminate all the interactive dialog. Therein you could simply specify the absolute paths of both the sending and receiving parts, simplifying both the logic and the evaluation of success or failure.

Regards!

...JRF...

[rhansen]

JRF,

I cannot use scp as the keys cannot be used. This is a limitation and SFTP using expect is the only option. Below, is the code that I have after the changes and as I mentioned a few times before this code uploads the file second time (matching file size). Also, the SFTP works perfectly fine when I do it manually on the SFTP prompt.

#!/usr/bin/ksh
typeset host_id="servername.test.org"
typeset user_id="test"
typeset password="password"
typeset HOME="/home/test"
typeset TMP=$HOME/tmp


cd $HOME
# get all the IDs in a file id.conf
awk '{
if (NF > 0) {
c = substr ($1, 1, 1)
if (c != "#") { # skip comments
print $1
}
}
}' id.conf >$TMP/id.conf

while read line
do
remote_directory=$line/submission/test
source_directory=$HOME/toremote/test/$line
cd $source_directory
for filename in *
do
if [[ -f ${filename} ]]
then
$HOME/upload_file.exp $user_id $password $host_id $remote_directory $source_directory $filename
if [[ $? -eq 0 ]]
then
echo "$(date): Upload successful
else
echo "${date}: Upload failed
fi
fi
done
done < $TMP/id.conf

The expect script that is called for the upload is:

#!/opt/expect/expect
set argc [llength $argv]
set user_id [lindex $argv 0]
set Password [lindex $argv 1]
set host_id [lindex $argv 2]
set remote_directory [lindex $argv 3]
set source_directory [lindex $argv 4]
set filename [lindex $argv 5]

spawn /usr/local/bin/sftp2 $user_id@$host_id
expect "Password:"
send "$Password\r"
expect "sftp>"
send "cd $remote_directory\r"
expect "sftp>"
send "lcd $source_directory\r"
expect "sftp>"
send "put $filename\r"
expect "sftp>"
send "quit\r"
exit

[James R. Ferguson]

Hi (again):

OK, the re-post of the code with corrections is appreciated. However, you still haven't answered some of the things discussed.

1: I had suggested that when you run the calling shell script you might do so with 'set -x' to trace execution and the evaluated arguments. Have you done that? Does it help uncover anything?

2: > The error that I get after the first upload is Unable to stat destination after creation (no such file). I can manually upload the file first time but not with the shell and expect script.

Once again, how do you "manually upload"? Is the 'expect' script used? Have you launched the 'expect' script from the commandline with valid arguments? And what happens then; does it work?

3: I still dislike the use of the HOME variable in assignments. I find this unnecessarily confusing.

4: As I noted before, your 'expect' script assumes that the remote and local directory changes succeed. For that matter, the shell script assumes the same. You should _test_ the results of these operations and take action if they fail. A failed change directory leaves the path where it was. Assume nothing.

Regards!

...JRF...

[OldSchool]

also, in the source and remote directory assignments, $line doesn't appear to be set anywhere prior to use.....

[James R. Ferguson · Accepted solution]

Hi (again):

> OldSchool: also, in the source and remote directory assignments, $line doesn't appear to be set anywhere prior to use.....

It is, in the 'while read line'.

However, in the original script the OP had:

remote_directory=$HOME/submission/test/$line
source_directory=$HOME/toremote/test/$line

...whereas in the last posted modifications the code is:

remote_directory=$line/submission/test
source_directory=$HOME/toremote/test/$line

Is this a change or was the '$line' lost?

Regards!

...JRF...

[OldSchool]

gak....

I parsed it as

while readline

I'd suggest that the he's probably not where he thinks he is when the "script" runs (as opposed to whatever he means by "manually".

as noted, "set -x" should help locate the issues, but I don't think he can "see the forest for the trees". The variable are getting set all over the place, and I'd bet it's not finding a path, but, witbout more information, who knows.......

[James R. Ferguson]

Hi:

> OldSchool: gak....I parsed it as while readline

I thought of that too. I think you did, because you know so much :-}

Regards!

...JRF...