need a perl script to read from text file and rename files

['chris']

hi

I know sometimes it's not easy, but I need a perl script rather urgently

I get 5 or more times a day, 3 files transfered : one info.txt and two data files:

info.txt
LA00001
LA00002

the name of info.txt never changed, but names of data files yes.
but the names in info.txt file (left) are always the same
like names of data files.

info.txt file looks:

----------------------------------

LA00001 DA20001

LA00002 DA20002

----------------------------------

the perl script should first open the info.txt file,
read the name of the first file (like left in info.txt),
rename the first file to DA20001 (like right in info.txt ),
transfer via ftp, wait 5 minutes,
rename the second file to DA20002 and transfer via ftp to the same server.
and should not transfer any files without info.txt

if someone can help me would be great, before I am lost !

kind regards
chris

[Sridhar Bhaskarla]

Hi Chris,

This is not a perl script. You can play with it until you get other responses. I assumed that if first file in info.txt doesn't exist the ftp will still continue. If that is not the case, you will need to make minor adjustments. Also it will create a directory called backup (modifyable) and puts the files in them so that you don't lose data.

You can test it out with some dummy files and see if it works.

-Sri

[H.Merijn Brand (procura · Accepted solution]

--8<---
#!/opt/perl/bin/perl

use strict;
use warnings;

use File::Copy;
use Net::FTP;
use Net::Netrc;

-f "info.txt" or die "No info.txt\n";

my $server = "my.ftp.server";
my $ftp = Net::FTP->new ($server, Debug => 3);
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
#$ftp->login ('user', 'password') or
#die "$_: cannot logon: " . $ftp->message;
chdir "/tmp" or die "/tmp: $!\n";
$ftp->cwd ("/tmp") or
die "$server: cannot chdir to /tmp: " . $ftp->message;

@ARGV = ("info.txt");
my %ren;
while (<>) {
m/^\s*(\S+)\s+(\S+)\s*$/ or next;
-f $1 or next;
-f $2 or die "Cannot rename $1 to $2: $2 already exists\n";
move ($1, $2);
$ftp->put ($2) or
die "$server: cannot put $2: " . $ftp->message;
sleep (5 * 60);
}
$ftp->quit;
-->8---

Completely off the top of my head. Nothing tested

Enjoy, Have FUN! H.Merijn

['chris']

thank you very much !
it helps a lot.

sorry, I did not write so clearly
I mean to wait for 5 minutes before the second file will be transfered.
this is very important.

and if I get only 1 or 2 data files without info.txt, I should get a mail.

sorry again for troubles, but how can I change this ?

kind regards
chris

[H.Merijn Brand (procura]

My script already does this!
it sleeps (5 * 60) seconds after each transfer.

Enjoy, Have FUN! H.Merijn

['chris']

hi procura

you are right, it's my mistake !
sorry about that.
I had only a a little bit problems with the script, but now it works well.
I did only some very little changes:

#!/usr/bin/perl
use strict;
use warnings;
use File::Copy;
use Net::FTP;
use Net::Netrc;

BEGIN
{
use CGI::Carp qw(carpout);
my $errorlog = "/home/test/logs/errorlog.txt";
open(LOG, ">$errorlog") or die("Unable to open $errorlog: $!\n");
print LOG "Errors:\n";
carpout(*LOG);
}

-f "/home/test/files/CDINFO" or die "No CDINFO NO TRANSFER\n";

chdir "/home/test/files" or die "/home/test/files: $!\n";

@ARGV = ("CDINFO");
my %ren;
while (<>) {
m/^\s*(\S+)\s+(\S+)\s*$/ ;
-f $1 ;
-f $2 ;
move ($1, $2);

my $server = "192.168.0.1";
my $ftp = Net::FTP->new ($server, Debug => 3);
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous') or
die "$_: cannot logon: " . $ftp->message;

$ftp->put ($2) or
die "$server: cannot put $2: " . $ftp->message;
sleep (1 * 10);
}
# $ftp->quit;

I can only not undestand, why $ftp->quit;
on the end is not working, if I put without #

My big problem at the moment is, that the remote ftp server
has a time out after 3 minutes and disconnects my connection,
before I can transfer the second file.
This procedure is a little bit complicated
and I can't send the second file earlier
or change the timeout of the ftp server.

I think I have to login to the remote server twice:
send the first file, close the ftp connection, wait for 5 minutes,
open the ftp session again
and transfer the second file.

I would not like to strain you thereby.
But if you have time and can help me, I'll be very happy.

kind regards
chris

[H.Merijn Brand (procura]

@ARGV = ("CDINFO");
my %ren;
while (<>) {

# You are now reading CDINFO, line by line

m/^\s*(\S+)\s+(\S+)\s*$/ ;

# this tests for the pattern
# word word
# => only two words on a single line. You cut out the 'or next' which should skip the current line if there is more or less than two words on this line

-f $1 ;
-f $2 ;

# skipped both warnings, making both lines void.
# -f $1 tests for existence of file 1 (the left word)
# I had this test because the move is useless if the file does not exist
# -f $2 tests for the existance of the second word as file. I had this test to be sure not to clobber it if it already existed

move ($1, $2);

# This statement moves the file nnamed by word one to the file named by word two, but with you having removed the validity checks, it is now *VERY* dangerous

my $server = "192.168.0.1";
my $ftp = Net::FTP->new ($server, Debug => 3);

# By moving above two statements *inside* the while loop, they are now lexical to the while loop, and thus only visible *inside* the loop, explaining why the quit does not work
# I had it before the loop, so you know the server connection throughout the script, and only connect once.
# With your modified code, you connect to the server on every line in the file, valid or not

$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous') or

# $ftp->login () probably also needs a password 'me@somewhere.com'

die "$_: cannot logon: " . $ftp->message;

$ftp->put ($2) or
die "$server: cannot put $2: " . $ftp->message;
sleep (1 * 10);

# again, for every line in the file

}
# $ftp->quit;

>I can only not undestand, why $ftp->quit;
>on the end is not working, if I put without #

because by now, $ftp is out of scope, and has closed itself in the destroy phase of the $ftp variable


>My big problem at the moment is, that the remote ftp server
>has a time out after 3 minutes and >disconnects my connection,
>before I can transfer the second file.

which indeed warrant the move of the connection to inside the loop.
man Net::FTP
would also show the timeout options of connect

>This procedure is a little bit complicated
>and I can't send the second file earlier
>or change the timeout of the ftp server.

Enjoy, Have FUN! H.Merijn [ who thinks that even thinking about running the changed code is dangerous ]

['chris']

hi procura

thanks for the comments.
this one works well, but how i said
I cannot sleep 5 minutes (5 * 60), because of the timeout from remote server.
------------------------------------------------------------
#!/usr/bin/perl
use strict;
use warnings;
use File::Copy;
use Net::FTP;
use Net::Netrc;

BEGIN
{
use CGI::Carp qw(carpout);
my $errorlog = "/home/test/logs/errorlog.txt";
open(LOG, ">$errorlog") or die("Unable to open $errorlog: $!\n");
print LOG "Errors:\n";
carpout(*LOG);
}

-f "/home/test/files/info.txt" or die "No info.txt NO TRANSFER\n";

chdir "/home/test/files" or die "/home/test/files: $!\n";

@ARGV = ("info.txt");
my %ren;
while (<>) {
m/^\s*(\S+)\s+(\S+)\s*$/ ;
-f $1 ;
-f $2 ;
move ($1, $2);

my $server = "192.168.0.1";
my $ftp = Net::FTP->new ($server, Debug => 3);
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous') or
die "$_: cannot logon: " . $ftp->message;

$ftp->put ($2) or
die "$server: cannot put $2: " . $ftp->message;
sleep (1 * 10);

$ftp->quit;
}
------------------------------------------------------------


I've tried something like this, but it doesn't work and I don't know how to fix it:
------------------------------------------------------------
#!/usr/bin/perl
use strict;
use warnings;
use File::Copy;
use Net::FTP;
use Net::Netrc;

BEGIN
{
use CGI::Carp qw(carpout);
my $errorlog = "/home/test/logs/errorlog.txt";
open(LOG, ">$errorlog") or die("Unable to open $errorlog: $!\n");
print LOG "Errors:\n";
carpout(*LOG);
}

-f "/home/test/files/info.txt" or die "No info.txt NO TRANSFER\n";

chdir "/home/test/files" or die "/home/test/files: $!\n";

@ARGV = ("info.txt");
my %ren;
while (<>) {
m/^\s*(\S+)\s+(\S+)\s*$/ ;
-f $1 ;
-f $2 ;
move ($1, $2);

my $server = "192.168.0.1";
my $ftp = Net::FTP->new ($server, Debug => 3);
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous') or
die "$_: cannot logon: " . $ftp->message;

$ftp->put ($1) or
die "$server: cannot put $1: " . $ftp->message;

$ftp->quit;

sleep (5 * 60);

my $server = "192.168.0.1";
my $ftp = Net::FTP->new ($server, Debug => 3);
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous') or
die "$_: cannot logon: " . $ftp->message;

$ftp->put ($2) or
die "$server: cannot put $2: " . $ftp->message;

$ftp->quit;
------------------------------------------------------------

and sorry again for troubles.

kind regards
chris

[H.Merijn Brand (procura]

#!/usr/bin/perl
use strict;
use warnings;
use File::Copy;
use Net::FTP;
use Net::Netrc;

BEGIN {
use CGI::Carp qw( carpout );
my $errorlog = "/home/test/logs/errorlog.txt";
open LOG, "> $errorlog" or die "Unable to open $errorlog: $!\n";
print LOG "Errors:\n";
carpout (*LOG);
}

chdir "/home/test/files" or die "/home/test/files: $!\n";
-f "info.txt" or die "No info.txt NO TRANSFER\n";

my $server = "192.168.0.1";
@ARGV = ("info.txt");
my %ren;
while (<>) { # for every line in info.txt
m/^\s*(\S+)\s+(\S+)\s*$/ or next; # That matches like "LA00001 DA20001"
-f $1 or next; # skip if file to rename does not exist
-f $2 or next; # skip if file to rename to already exists
move ($1, $2); # Rename the file

my $ftp = Net::FTP->new ($server, Timeout => 9000, Debug => 3); # Default timeout od 120 (2minutes) is too short
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous', 'some@body.net') or
die "$_: cannot logon: " . $ftp->message;

# Put file 2 (not 1) to the ftp server
$ftp->put ($2) or
die "$server: cannot put $2: " . $ftp->message;

$ftp->quit;

# And wait 5 minutes
sleep (5 * 60);
} # End of scope for reading info.txt: go to next line (and thus next file)
-->8---

- Do NOT remove the 'or next' parts. That is ultimately dangerous.
- once you've moved file $1 to file $2, $1 does not exist anymore, so there is no use in putting it to the server. A timeout is the desired operation :)
- remember that 'while (<>) {' reads *every* line in info.txt, not only the lines where the filenames are listed. You want to skip all lines that do not match that pattern

Enjoy, Have FUN! H.Merijn

['chris']

thanks it works very well.
great script !

I have only last question, but only if you really have time.
how can I stop ftp transfer, when one of these 2 files is missing ?

these files will be processed to the database and I cannot send only one file.
worst which can happen is to send the second file without the first one or if the first one will be transfered not completely,
for example the remote server crashed during the transfer.
But this, I have to find out self.
You did enough work.

thanks again and have a nice week
chris

[H.Merijn Brand (procura]

my %ren;
while (<>) { # for every line in info.txt
m/^\s*(\S+)\s+(\S+)\s*$/ or next; # That matches like "LA00001 DA20001"
-f $1 or next; # skip if file to rename does not exist
-f $2 or next; # skip if file to rename to already exists
$ren{$1} = $2;
}

unless (2 == scalar keys %ren) {
die "there are not exactly 2 files in info.txt\n";
# extend to your feeling
}

foreach my $f1 (keys %ren) {
$f2 = $ren{$f1};
move ($f1, $f2); # Rename the file

my $ftp = Net::FTP->new ($server, Timeout => 9000, Debug => 3); # Default timeout od 120 (2minutes) is too short
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous', 'some@body.net') or
die "$_: cannot logon: " . $ftp->message;

# Put file 2 (not 1) to the ftp server
$ftp->put ($f2) or
die "$server: cannot put $f2: " . $ftp->message;

$ftp->quit;

# And wait 5 minutes
sleep (5 * 60);
}
-->8---

Note that $1 and $2 now changed to $f1 and $f2 because they are saved in the hash on the first pass

Note also that this method does NOT guarantee that file1 is sent BEFORE file2, but since we now know for sure that they are BOTH sent, I guess that does not matter. It's not har to preserve sequence, but probably unneccacary

Enjoy, Have FUN! H.Merijn

['chris']

hi

thanks,

but if I try this:

#!/usr/bin/perl
use strict;
use warnings;
use File::Copy;
use Net::FTP;
use Net::Netrc;

BEGIN
{
use CGI::Carp qw(carpout);
my $errorlog = "/home/test/logs/errorlog.txt";
open(LOG, ">$errorlog") or die("Unable to open $errorlog: $!\n");
print LOG "Errors:\n";
carpout(*LOG);
}

chdir "/home/test/files" or die "/home/test/files: $!\n";
-f "/home/test/files/info.txt" or die "No info.txt NO TRANSFER !\n";

my $server = "192.168.0.1";
@ARGV = ("info.txt");
my %ren;
while (<>) { # for every line in info.txt
m/^\s*(\S+)\s+(\S+)\s*$/ or next; # That matches like "LA00001 DA20001"
-f $1 or next; # skip if file to rename does not exist
-f $2 or next; # skip if file to rename to already exists
$ren{$1} = $2;
}

unless (2 == scalar keys %ren) {
die "there are not exactly 2 files in info.txt\n";
# extend to your feeling
}

foreach my $f1 (keys %ren) {
$f2 = $ren{$f1};
move ($f1, $f2); # Rename the file

my $ftp = Net::FTP->new ($server, Timeout => 9000, Debug => 3); # Default timeout od 120 (2minutes) is too short
$ftp or die "$server: cannot connect: $@";
# If you don't use ~/.netrc
$ftp->login ('anonymous', 'some@body.net') or
die "$_: cannot logon: " . $ftp->message;

# Put file 2 (not 1) to the ftp server
$ftp->put ($f2) or
die "$server: cannot put $f2: " . $ftp->message;

$ftp->quit;

# And wait 5 minutes
sleep (5 * 60);
}


these 2 files wont be renamed and transfered.
I tried but couldn't find the error.

I get in the log:

[Mon Feb 23 13:39:37 2004] ftp3.cgi: Global symbol "$f2" requires explicit package name at ftp3.cgi line 36.
[Mon Feb 23 13:39:37 2004] ftp3.cgi: Global symbol "$f2" requires explicit package name at ftp3.cgi line 37.
[Mon Feb 23 13:39:37 2004] ftp3.cgi: Global symbol "$f2" requires explicit package name at ftp3.cgi line 46.
[Mon Feb 23 13:39:37 2004] ftp3.cgi: Global symbol "$f2" requires explicit package name at ftp3.cgi line 47.
[Mon Feb 23 13:39:37 2004] ftp3.cgi: Execution of ftp3.cgi aborted due to compilation errors.
Errors:

greetings
chris

[H.Merijn Brand (procura]

Sorry. change it to

foreach my $f1 (keys %ren) {
my $f2 = $ren{$f1};
move ($f1, $f2); # Rename the file

in other words: I forgot to make $f2 lexical in the transition for $2 to $f2

Should do the job

Enjoy, Have FUN! H.Merijn [ Who hopes you picked up some Perl scripting on the way ]

['chris']

I am missing the words to thank you for the very competent support.

Now it works great and checks correctly if files are not complete.
I'm very happy.

What I have to do like before, is to take out
"or next" from

-f $2 ;

otherwise is not working and the log says:

[Mon Feb 23 23:38:19 2004] ftp3.cgi: there are not exactly 2 files in info.txt
Errors:

that was the reason why I've done before
and you asked me.
But if I put away script works correctly.

Only last thing and really last if I can ask you
how to move all 3 files to other directory but only if the transfer was successfully ?

greetings
chris

[H.Merijn Brand (procura]

Good to see happy ITRC members :)

> What I have to do like before, is to take out "or next" from
>
> -f $2 ;

Or you could change it to 'and next', what it should have been in the first place :/

> how to move all 3 files to other directory > but only if the transfer was successfully ?

All three??? You're talking about TWO files all the time.

Now that this code

# Put file 2 (not 1) to the ftp server
$ftp->put ($f2) or
die "$server: cannot put $f2: " . $ftp->message;

*die*s if the put is unsuccessful, you *know* that at the end of the script, all transfers succeeded. Now just choose a target directory, and go on with the flow:

# And wait 5 minutes
sleep (5 * 60);
}
foreach my $f2 (values %ren) {
move $f2, "/your/new/dir/$f2";
}

Enjoy, Have FUN! H.Merijn

['chris']

thank you again,

it works works excelent !

there are 3 files, info.txt included, because info.txt changes all the time

I added:

move $f2, "/home/test/out/$f2";
move "info.txt", "/home/test/out/info.txt";
move "/home/test/logs/errorlog.txt", "/home/test/out/errorlog.txt";

and it works well

I need the log file also, because it's be overwritten each time.

last problem which I have is each time after transfer to create in /home/test/out/
the new underdirectory with date and time like 20040224
and put all files there, because will be overwritten each time.
that will be better solution
and I can control a little bit the transfers.

but I have to find out self.
You did really enough for me.

greetings
chris

[H.Merijn Brand (procura]

Hint:


my @dt = localtime;
my $subfolder_name = ((1900 + $dt[5]) * 100 + 1 + $dt[4]) * 100 + $dt[3];
mkdir "/home/test/out/$subfolder_name" or die "$subfolder_name: $!";

Enjoy, Have Fun! H.Merijn

['chris']

thanks for your time, quick reaktion and patience with me

it works perfectly !

subdirectory will be created.
only last little question:

if I put:

move $f2, "/home/test/out/$subfolder_name/$f2";
move "info.txt", "/home/test/out/$subfolder_name/info.txt";
move "/home/test/logs/out/$subfolder_name/errorlog.txt", "/home/test/out/errorlog.txt";

then all files will be moved correctly
to the new subdirectory,
but not deleted at their old place.
I don't know really what's wrong and try
to find out.
they were moved before and deleted

thanks again and have a nice evening
chris

[H.Merijn Brand (procura]

That should not happen.
move () should move, and thus remove the file from the old location.
copy () should copy, and thus leave the original file intact.

You could change

move ($f2, "/blahdieblah/$f2");

with

system "mv $f2 /blahdieblah/$f2";

but now I'm realy interested in what went wrong (or not as good as it should).

man File::Copy

should give you some info on move ()'s return codes. Maybe that'll give you some insight in the cause

Enjoy, Have FUN! H.Merijn

['chris']

with:

system "mv $f2 /home/test/out/$subfolder_name/$f2";

it works perfectly now and all files were removed !

I don't have any questions.

thanks for the great and competent support.

greetings
chris

['chris']

you were interested, what was wrong
and I know more

the problem is when the subfolder already exists, then script wont to overwrite it
and all files stayed at their old location

there was the reason, why "move" doesn't work.

when I tried first time with "system mv" ,
then I've removed subfolder already manually.
second time I had the same problem like with "move" before.

so "move" and "system mv" do the same job.

maybe will be the solution to put the time to the subfolder like 200402200915 ?
then each time the new subfolder will be created and all files moved into.

greetings
chris

['chris']

I tried this and it works:

my @dt = localtime;
my $subfolder_name = ((1900 + $dt[5]) * 100 + 1 + $dt[4]) * 100 + $dt[3];

if($dt[2] < 10){
$dt[2] = "0" . $dt[2];
}
if($dt[1] < 10){
$dt[1] = "0" . $dt[1];
}
$subfolder_name .= "$dt[2]$dt[1]";

mkdir "/home/test/out/$subfolder_name" or die "$subfolder_name: $!";

the subfolder 200402250115 was created
and all files moved into.

greetings
chris

[H.Merijn Brand (procura]

or easier

my @dt = localtime;
my $subfolder_name = ((((1900 + $dt[5]) * 100 + 1 + $dt[4]) * 100 + $dt[3]) * 100 + $dt[2]) * 100 + $dt[1];

Enjoy, Have FUN! H.Merijn

['chris']

thanks.

your solution is better and easier and I've changed.

I have only a problem, which prepares big headaches for me.
I can access ftp only via vpn client
and what happens when the first file
was already transfered successfully,
then when the script is waiting for 5 minutes,
the vpn connection goes down ?

is it possible try to send the second file
2 times again in period of 15 minutes
and if the file could not be sent,
to send a mail ?

open(MAIL, "|/usr/sbin/sendmail -t");
print MAIL "to:$mail\n";
print MAIL "from:$linux\n";
print MAIL "subject:ftp transfer was NOT successfully \n";

print MAIL "------------------------------------------------------------------\n";

print MAIL "ftp transfer was NOT successfully\n\n";

print MAIL "Time: ". localtime(time) ." \n";

print MAIL "------------------------------------------------------------------\n";

close(MAIL);
}

but only if you really have time.

kind regards
chris

[H.Merijn Brand (procura]

# Put file 2 (not 1) to the ftp server
# VPN might go down, so retry twice
foreach my $try (0 .. 2) {
$ftp->put ($f2) and last; # Success
if ($try == 2) {
open my $mail, "| /usr/sbin/sendmail -t" or die "Cannot send mail: $!\n";
print $mail "To: $mail\n",
"From: $linux\n",
"Subject: ftp transfer was NOT successfull\n",
"\n------------------------------------------------------------------\n",
"ftp transfer was NOT successfully\n\n";
"Time: ". scalar localtime ."\n",
"------------------------------------------------------------------\n.\n";
close $mail;
die "$server: cannot put $f2: " . $ftp->message;
}
print STDERR "Trasfer of $f2 failed: ",$ftp->message, ", retry in 3 minutes\n";
sleep (3 * 60);
}
$ftp->quit;

# And wait 5 minutes
sleep (5 * 60);
}

Change to your needs

BTW Mail::Sendmail would make that mailing stuff even more readable and maintainable

Enjoy, Have FUN! H.Merijn [ Who doubts the number of braces in above code snippet ;) ]