problem in selecting files as per timestamp while ftp

[hpuserhk]

Hello Folks,
I written shell script for processing some data,one of the requirement is to ftp files from remote server, problem is i want to ftp
files of from server only of yesterday ,and filename do not have any string for timestamp,so from file creation date on remote server i need to pick up files of yesterday,
when i go in ftp session i dont have any option to select yesterdays files ,so i create list of files and copy on local computer and try to filter name of yesterday files from list,but created list also not in proper format,filename and date on seperate lines,pl adive,
thanks in advance
best regards

[Steven E. Protter]

Shalom,

Here is an example ftp script.

ftp -i -v -n ftp.somwhere.com << END_FTP >> someuser mypasswordbinary
lcd /scripts/download
cd /scripts
get auto_ftp_xfer.ksh
bye
END_FTP

you can do an ls -la instead, connect the data and use whatever criteria you want to actually get files with a second connection.

SEP

[hpuserhk]

Thanks steven for reply,

i already tried that way also,but issues are after ftp files from remote to local machine
file creation date is reset to todays date of all the files,so timestamps of files of remote machine is lost,so its impossible to fileter yesterdays files,other issues there is very large number of files on remote machine and this process run everyday so performance wise problem.
Now i am thinking of getting list of all files from remote server(using ftp session since i have authentication of ftp only)using command
dir fielnames* outputfile.txt
so outputfile.txt contain list of all files
but when i checked the outputfile.txt list contains filename and timestamps on seperate lines ,
example
filename1 owner
dateoffile
filename2 owner
dateoffile
so advice how to filter

[Steven Schweda]

It may not solve all your problems, but you
might wish to look into using wget as your
FTP client. It can read and interpret FTP
server file listings, and set the date-time
of a fetched file to match the date-time it
had on the server (within the obvious limits
of the lame date-time format used in a
typical UNIX-like "ls -l" report). It
doesn't know what "yesterday" is, but it does
have options like

-N, --timestamping don't re-retrieve files unless newer than local.

which might be useful.

http://www.gnu.org/software/wget/wget.html

[James R. Ferguson]

Hi:

> ...but created list also not in proper format,filename and date on seperate lines,please adisve...

This is easy to fix. Given something like:

# cat myfile
filename1
09-21-2008 08:01
filename2
09-21-2008 09:00
filename3
09-21-2008 09:00

Do:

# paste -d " " - - < myfile > myfile.new

# cat myfile.new
filename1 09-21-2008 08:01
filename2 09-21-2008 09:00
filename3 09-21-2008 09:00

Now you can more easily parse the file for yesterday's file names.

Regards!

...JRF...

[Dennis Handly]

Didn't you already ask this question before:
http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1263027

assigned points to 5 of 31 responses
Also please read the following about assigning points:
http://forums.itrc.hp.com/service/forums/helptips.do?#33
http://forums.itrc.hp.com/service/forums/pageList.do?userId=CA1472826&listType=unassigned&forumId=1

[hpuserhk]

Thanks JRF
but my file listing is bit complicated since the remote server is open vms and i am running this program from unix box
dir listing created has many columns like filename version,filesize,timestamp,owner,privileges
and same file information of this type is split on 2 lines
like
filename1
filesize timestamp owner privileges
filename2
filesize timestamp owner privileges
filename3
filesize timestamp owner privileges



i am using dir command after ftp session to list files
pl advice

Best Regards

[Steven Schweda]

> [...] the remote server is open vms [...]

Well. The secret is out at last.

If the FTP server runs on a VMS system, then
you haven't specified which one it is, as
there is more than one IP package possible,
and probably more than one possible FTP
server on each. I'd still suggest looking at
wget to see if it helps, but you'd probably
do better with the more VMS-tolerant code in
my wget kit:

http://antinode.info/dec/sw/wget.html


> dateoffile

This, by the way, was a particularly
unhelpful way to describe what you get in the
file listing. If you had shown the actual
data instead of hiding the useful details,
some important things would have been clearer
sooner.

> dir listing created has many columns like
> filename
> version,filesize,timestamp,owner,privileges
> and same file information of this type is
> split on 2 lines
> like

I wouldn't say that the file version is in a
separate column, either. In any case, if my
wget code has any trouble dealing with a VMS
FTP server, please let me know. (If, that
is, you can provide a complete and useful
problem report instead of one like this.)

[hpuserhk]

following is sample file contents:
FILE1_29223080.DAT;1
0/0 21-SEP-2008 04:05:39 [CSL,PCMIBAT]
(RWED,RWED,RWED,RE)
FILE2_29223080.DAT;1 0/0 21-SEP-2008 04:05:39 [CSL,PCMIBAT]
(RWED,RWED,RWED,RE)
FILE3_29223080.DAT;1 0/0 21-SEP-2008 04:05:39 [CSL,PCMIBAT]
(RWED,RWED,RWED,RE)

Please note 2nd and 3rd line in file is one line so each record has 2 lines,
pl advise how to make this line continuos or concat the files

Best Regards

[Steven Schweda]

> following is sample file contents:
> [...]

Yup, that's a typical VMS FTP server listing.

> Please note 2nd and 3rd line in file is one
> line so each record has 2 lines,
> pl advise how to make this line continuos
> or concat the files

In a listing like this, I'd expect to see all
the items on one line if the name is short,
and to see the name on its own line if the
name is long. It's not obvious from the
Forum-mutilated text shown here, but the file
names start in column 1 of their lines, while
the lines which don't include the file name
start with some white space. This gives you
an easy way to distinguish a continuation
line from a file name. I'd need to think a
while if I wanted some sed commands to do a
join when a line starts with a space, but I
suspect that there's a way to do that.

Personally, I'd try very hard to find a way
to persuade wget to do the work, because I
believe that it (well, my version of "it")
has all the code needed to interpret a
listing like this. It's certainly willing to
fetch only files which have been updated on
the server since they were last downloaded:

-N, --timestamping don't re-retrieve files unless newer than
local.

With a little fooling around, I'd bet that it
could be modified to fetch files newer than
some single, user-specified date-time, too,
but I don't think that it has an option for
exactly that behavior now.