Help with a script

[George Doller]

I am trying to write a script that I can setup in cron and copy/ftp files at a given time. I've tried the script 2 diffent ways. One way it didn't process either file if 2 or more files were created between the cron runs and the 2nd way (below) the ftp says there is already a file with that name and the process stops.
Here is what I am trying to run.
PATH=/sbin:/usr/sbin:/usr/bin:/opt/pgp-6.5.8;export PATH
WAND=/home/wand;export WAND
WANDSAVE=/home/wand/SAVE;export WANDSAVE
DATE=$(date '+%Y%m%d');export DATE
TIME=$(date '+%H%M%S');export TIME
echo $DATE
echo $TIME
export FTPSERVER=xxx.x.x.xxx

cd $WAND
mv people_error_*.txt.pgp /tmp/
mv people_error_*.txt /tmp/
/usr/bin/cp -p testfile* $WANDSAVE/

ls testfile* >/tmp/testfile.out
for file in '/tmp/testfile.out'
do
sleep 30
/usr/bin/mv $file people_error_"$DATE"_"$TIME".txt

/opt/pgp-6.5.8/pgp -e people_error_*.txt PGP-Key

/usr/bin/echo "`/usr/bin/date` Starting FTP process"

/usr/bin/echo "user xxxxxxx xxxxxxxx\nbin\nput people_error_*.txt.pgp\nbye" >/
p/ftpin_error.tmp
/usr/bin/ftp -n -i $FTPSERVER
/usr/bin/echo "`/usr/bin/date` ending FTP process"

rm /tmp/ftpin_error.tmp
done

Any help would be appreciated and points awarded.

[Steven Schweda]

Just as a starting point, what did you think
that this was doing?:

for file in '/tmp/testfile.out'
do

It's not _reading_ "/tmp/testfile.out".

td192> cat f1.sh
#!/bin/sh
for file in '/tmp/testfile.out'; do
echo $file
done

td192> ./f1.sh
/tmp/testfile.out

It is possible to read names from a file, but
this is not one of the ways.

For diagnostic/educational purposes, you may
find it helpful to run your script with "-v"
and/or "-x" options ("sh -x f1.sh").

[erics_1]

George,

I think your 'for' loop needs an adjustment. If you put an echo $file statement under your 'for/do' loop, you'll see that only /tmp/testfile.out gets returned and the loop exits.

Try:

for file in `cat /tmp/testfile.out`
do

done

Regards,
Eric

[George Doller]

You guys are correct, it's not putting out the files name as I expected. I tried adding the cat in front and I still only get /tmp/testfile.out. Not sure why....

[James R. Ferguson]

Hi George:

If you want to read a file do something like this:

#!/usr/bin/sh
while read LINE
do
echo ${LINE}
done < /etc/hosts


...This can be ammended to read and split a line into its components. For example, if you only want the first "word" (based on the current setting of 'IFS'):

while read FIRST REST
do
echo ${FIRST}
done < /etc/hosts


...or, change the 'IFS' (inter-field-seperator) to a colon:

OLDIFS=${IFS}
IFS=":"
while read FIRST REST
do
echo ${FIRST}
done < /etc/passwd
IFS=${OLDIFS}

...

Regards!

...JRF...

[George Doller]

The while read LINE worked well to get a listing of my file names but my next problem is how to rename each of them and ftp them individually. Any help would be great.

[James R. Ferguson]

Hi (again):

To move a file:

TIMESTAMP=$(date +%Y%m%d_%H%M%S)
while read FILE
do
mv ${FILE} ${FILE).${TIMESTAMP}
done < inputlist

...It is good technique to use the curly braces to surround your variables. This avoids ambuguity.

As for your FTP portion, you could reference your local and remote filenames as variables ( ${localfile} and ${remotefile} ). Then, you can pass what you need.

Regards!

...JRF...

[George Doller]

Your last reply was great, my only problem now is my users want the original files to be changed to people_error_{TIMESTAMP}.txt and since the 2 files are being processed together, I am only getting 1 renamed file. Is there a way to put a sleep in between the first mv and the 2nd mv? If not then I don't think what I am trying to do will work. since both output files will have the same name.

[James R. Ferguson]

Hi (again) George:

One problem with the ${TIMESTAMP} variable I built for you is that it is static.

You can do (better) with:

# mv ${FILE} ${FILE}.$(date +%Y%m%d_%H%M%S)

...as written. This causes the evaulation "on-the-fly" and if a second passes between moves, the file's name reflects that.

Of course, in a loop, you can also impose a wait with:

# sleep 1

Regards!

...JRF...

[Steven Schweda]

> I tried adding the cat in front and I still
> only get /tmp/testfile.out. Not sure
> why....

Probably because you didn't do exactly what
was suggested, and there's a difference
between "'" and "`".

td192> cat f3.dat
aaa
bbb
td192> cat f3.sh
#!/bin/sh
for file in ` cat f3.dat `; do
echo $file
done
td192> ./f3.sh
aaa
bbb

As usual, a precise and accurate description
of what you did, and what happened when you
did it, might be helpful.

> [...] my only problem now [...]

Oh, I doubt that.

[George Doller]

I tried using the date on the file instead of TIMESTAMP but I guess because they are same files they are happening at the same time and the 2nd file is over writing the 1st one. Thay for the suggestion. Can you explain how I might be able to try the loop?

[James R. Ferguson · Accepted solution]

Hi (again) George:

#!/usr/bin/sh
while read FILE
do
mv ${FILE} ${FILE}.$(date +%Y%m%d_%H%M%S)
sleep 1
done < inputlist

...will make sure that the timestamp extension is unique.

Regards!

...JRF...

[George Doller]

With the Sleep I was able to get my script to do everything I needed.
thank you very much.

[Dennis Handly]

>I tried adding the cat in front and I still only get /tmp/testfile.out.

(You shouldn't be using cat, that's evil. :-)

There are two basic ways to handle files with files. Either use while, as JRF said:
while file; do
...
done < file

Or if a "limited" number that will fit in a shell command line, you can use for (an improvement of Steven's case):
for file in $(< file); do
...
done

Note: No cats needed.

As a coding style, unless your other shell scripts need these variables, you shouldn't export them:
DATE=$(date '+%Y%m%d');export DATE
TIME=$(date '+%H%M%S');export TIME

And if you using a real shell, you can use the export command directly:
export FOO=bar

[James R. Ferguson]

Hi (again) George:

As Dennis noted, 'cat's can be evil.

Well-behaved (written) Unix filters will read files passed as commandline arguments, or take their input from STDIN as from a pipe's output.

Thus, when someone writes (variously):

# cat file | awk ...
# cat file | grep ...

...they are creating a needless extra process and wasting I/O. The 'cat' process reads a file and writes each line of output which is then *read* again by another process only to filter and *write* something. Instead, one should write:

# awk '{ ... }' file
# grep whatever file

Regards!

...JRF... [ who happens to like cats (as animals) ]

[Steven Schweda]

> [...] (an improvement of Steven's case) [...]

Hey. Don't blame me for that "cat". I was
only offering an illustration of how the
previous suggestion actually _does_ work
(with the right punctuation). As I said, "It
is possible to read names from a file, but
this is not one of the ways." The one of
those ways which I'd prefer is a while-read
like that suggested by Mr. Ferguson, which
allows more flexible processing more easily.

[trpjr1]

George,

I like to use "set -x" at the begining of scripts and in functions while testing. It helps to debug... Just thought it might help.

Tommy

[George Doller]

I was able to use the while command and with the sleep it solved all my needs. Thanks for all the quick responses.