problem in selecting files as per timestamp while ftp

[hpuserhk]

Hello Folks,
I written shell script for processing some data,one of the requirement is to ftp files from remote server, problem is i want to ftp
files of from server only of yesterday ,and filename do not have any string for timestamp,so from file creation date on remote server i need to pick up files of yesterday,
when i go in ftp session i dont have any option to select yesterdays files ,so i create list of files and copy on local computer and try to filter name of yesterday files from list,but created list also not in proper format,filename and date on seperate lines,pl adive,
thanks in advance
best regards

[Steven E. Protter]

Shalom,

Here is an example ftp script.

ftp -i -v -n ftp.somwhere.com << END_FTP >> someuser mypasswordbinary
lcd /scripts/download
cd /scripts
get auto_ftp_xfer.ksh
bye
END_FTP

you can do an ls -la instead, connect the data and use whatever criteria you want to actually get files with a second connection.

SEP

[hpuserhk]

Thanks steven for reply,

i already tried that way also,but issues are after ftp files from remote to local machine
file creation date is reset to todays date of all the files,so timestamps of files of remote machine is lost,so its impossible to fileter yesterdays files,other issues there is very large number of files on remote machine and this process run everyday so performance wise problem.
Now i am thinking of getting list of all files from remote server(using ftp session since i have authentication of ftp only)using command
dir fielnames* outputfile.txt
so outputfile.txt contain list of all files
but when i checked the outputfile.txt list contains filename and timestamps on seperate lines ,
example
filename1 owner
dateoffile
filename2 owner
dateoffile
so advice how to filter

[Steven Schweda]

It may not solve all your problems, but you
might wish to look into using wget as your
FTP client. It can read and interpret FTP
server file listings, and set the date-time
of a fetched file to match the date-time it
had on the server (within the obvious limits
of the lame date-time format used in a
typical UNIX-like "ls -l" report). It
doesn't know what "yesterday" is, but it does
have options like

-N, --timestamping don't re-retrieve files unless newer than local.

which might be useful.

http://www.gnu.org/software/wget/wget.html

[James R. Ferguson]

Hi:

> ...but created list also not in proper format,filename and date on seperate lines,please adisve...

This is easy to fix. Given something like:

# cat myfile
filename1
09-21-2008 08:01
filename2
09-21-2008 09:00
filename3
09-21-2008 09:00

Do:

# paste -d " " - - < myfile > myfile.new

# cat myfile.new
filename1 09-21-2008 08:01
filename2 09-21-2008 09:00
filename3 09-21-2008 09:00

Now you can more easily parse the file for yesterday's file names.

Regards!

...JRF...

[Dennis Handly]

Didn't you already ask this question before:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1263027

assigned points to 5 of 31 responses
Also please read the following about assigning points:
http://forums.itrc.hp.com/service/forums/helptips.do?#33
http://forums.itrc.hp.com/service/forums/pageList.do?userId=CA1472826&listType=unassigned&forumId=1

[hpuserhk]

Thanks JRF
but my file listing is bit complicated since the remote server is open vms and i am running this program from unix box
dir listing created has many columns like filename version,filesize,timestamp,owner,privileges
and same file information of this type is split on 2 lines
like
filename1
filesize timestamp owner privileges
filename2
filesize timestamp owner privileges
filename3
filesize timestamp owner privileges



i am using dir command after ftp session to list files
pl advice

Best Regards

[Steven Schweda]

> [...] the remote server is open vms [...]

Well. The secret is out at last.

If the FTP server runs on a VMS system, then
you haven't specified which one it is, as
there is more than one IP package possible,
and probably more than one possible FTP
server on each. I'd still suggest looking at
wget to see if it helps, but you'd probably
do better with the more VMS-tolerant code in
my wget kit:

http://antinode.info/dec/sw/wget.html


> dateoffile

This, by the way, was a particularly
unhelpful way to describe what you get in the
file listing. If you had shown the actual
data instead of hiding the useful details,
some important things would have been clearer
sooner.

> dir listing created has many columns like
> filename
> version,filesize,timestamp,owner,privileges
> and same file information of this type is
> split on 2 lines
> like

I wouldn't say that the file version is in a
separate column, either. In any case, if my
wget code has any trouble dealing with a VMS
FTP server, please let me know. (If, that
is, you can provide a complete and useful
problem report instead of one like this.)

[hpuserhk]

following is sample file contents:
FILE1_29223080.DAT;1
0/0 21-SEP-2008 04:05:39 [CSL,PCMIBAT]
(RWED,RWED,RWED,RE)
FILE2_29223080.DAT;1 0/0 21-SEP-2008 04:05:39 [CSL,PCMIBAT]
(RWED,RWED,RWED,RE)
FILE3_29223080.DAT;1 0/0 21-SEP-2008 04:05:39 [CSL,PCMIBAT]
(RWED,RWED,RWED,RE)

Please note 2nd and 3rd line in file is one line so each record has 2 lines,
pl advise how to make this line continuos or concat the files

Best Regards

[Steven Schweda]

> following is sample file contents:
> [...]

Yup, that's a typical VMS FTP server listing.

> Please note 2nd and 3rd line in file is one
> line so each record has 2 lines,
> pl advise how to make this line continuos
> or concat the files

In a listing like this, I'd expect to see all
the items on one line if the name is short,
and to see the name on its own line if the
name is long. It's not obvious from the
Forum-mutilated text shown here, but the file
names start in column 1 of their lines, while
the lines which don't include the file name
start with some white space. This gives you
an easy way to distinguish a continuation
line from a file name. I'd need to think a
while if I wanted some sed commands to do a
join when a line starts with a space, but I
suspect that there's a way to do that.

Personally, I'd try very hard to find a way
to persuade wget to do the work, because I
believe that it (well, my version of "it")
has all the code needed to interpret a
listing like this. It's certainly willing to
fetch only files which have been updated on
the server since they were last downloaded:

-N, --timestamping don't re-retrieve files unless newer than
local.

With a little fooling around, I'd bet that it
could be modified to fetch files newer than
some single, user-specified date-time, too,
but I don't think that it has an option for
exactly that behavior now.

[hpuserhk]

Thanks Steven

I can understand wget is easy way to do this task,but problem is i cant ask client to use 3rd party software ,so i had to resolve this by unix /scripting ,i am trying but getting directory listing first then do some shell script coding to read alternate line match the string and create file of required ftp files,but here again files generated by ftp dir command contain timestamp in DD-MMM-yyyy HHMISS format where
is yesterday date of by unix command is in dd-mm-yyyy format ,so is there any direct command to change format of date or i handle this also by coding

Best Regards

[Dennis Handly]

>files generated by ftp dir command contain timestamp in DD-MMM-yyyy HHMMSS format where as yesterday date of by unix command is in dd-mm-yyyy format

You can't get yesterday's day unless you save it, use perl or a C application.

>is there any direct command to change format of date

date(1) allows you to use any format you like:
date "+%d-%b-%Y"

>I handle this also by coding

Getting "yesterday" may be the toughest.

[James R. Ferguson]

Hi:

We could have been of more help from the beginning had you posted some *actual* samples of your file's data.

As Dennis deduced, this is also a continutation of your earlier "Urgent- Problem in FTP" dated August 27 [where you failed to score the feedback given there]. It would have been clearer to this whole discussion had you continued this there.

Either use 'paste' to join multiple lines of your directory listing as I suggested above, or write a simple shell read loop to join (or not) successive lines based on their content.

All this said, use Perl to compute and format "yesterday's" date.

# DATE=$(perl -MPOSIX -le 'print strftime "%d-%b-%Y",localtime(time-86400)')
# echo $DATE
21-Sep-2008

Regards!

...JRF...

[Steven Schweda]

> You can't get yesterday's day unless you
> save it, use perl or a C application.

A programming problem which can be solved
only by using these two languages?
Interesting, but unlikely.

There's already C code in (my) wget to parse
a VMS FTP server listing (which, I claim, is
much easier to do than parsing a UNIX-like
"ls -l" listing), so you can use it, adapt
it, or ignore it and do it all yourself (in
whichever language you find appropriate).
Having already done this work once, I'm not
particularly interested in doing it again,
but you can do what you wish.

> [...] but problem is [...]

Actually, the _big_ problem here is that you
seem to be unable to present a usefully
complete description of the problem you're
trying to solve, and the constraints on the
solution, which pretty much sucks all the fun
out of trying to help you with it.

[hpuserhk]

Thanks all for your support
Once i free from work i will make necessary assignment for help

I am doing to code it in shell scripting
actually i was avoiding to write more code
and find some way to do it effectively in one command
but this can be handle with shell scripting
by reading and manipulating file containing dir listing
Best Regards