using curl command to get list of files in directory via ftps

[John Kittel]

I'm trying to use curl in a shell script to get a list of file names from an ftps site, save the names in a shell variable, then use the file names in another curl command to get the specific file(s) by name. I am getting the file names back, but they seem to have weird characters in them, so that they aren't useable in the next step to specify the file I want to get. Also, for debug, I don't know how to see, in hex for example, what are the contents of the variables.

Here is a fragment of a script where I get the directory list back, and then print the variable names:

FILES=`curl --ftp-ssl --silent --use-ascii --ftp-method nocwd --list-only --user user:passwd ftp://ftp.vendorname.com/`
for F in ${FILES}
do
echo "File = ${F}...\n"
done

And the output of that looks weird, like this:

...e = edidata

...e = edidata.prior


Whereas, what I expect the output should look like would be this:

File = edidata...

File = edidata.prior...

It's like the variable ${F} contains characters that cause the output of the echo to be garbled. I only put the "..." characters in the echo statement by chance while I was trying to debug why the variable(s) ${F} couldn't be used in a subsequent curl command to specify a specific file to get. When I try to use it that way, I get back a response from the ftps site saying there is no such file. If I hard code into my script the filename "edidata", then the get succeeds.

3 questions:

1) Do you agree it seems I'm getting weird characters back as part of the file names?
2) If yes, why?
3) How can I see what exactly are the characters in the variable ${F}

Thanks...

[Steven Schweda]

> 3) How can I see what exactly are the
> characters in the variable ${F}

Don't store them in a shell variable?

Why not run the cURL command, and direct its
output to a file? Worst case, a program like
"od" should be able to show you what's
happening.

[Michael Mike Reaser]

For (3), try

echo $F | xd -v -tx1c

This should display the contents of "F" in hex, 1 character at a time.

Maybe curl is putting something funky, like a carriage return or newline, embedded directly in the string being returned?

[James R. Ferguson]

Hi:

You could use 'vis' to expose unprintable characters:

# F="ab\007cdef"
# echo "File = '${F}'|vis

Regards!

...JRF...

[John Kittel]

That worked great. Thanks again.

The directory list I get back from the curl command uses DOS/Windows end of line sequence, CRLF.

I am using the curl --use-ascii switch, but for whatever reason it doesn't work, isn't applicable, in this case / context.

I will find a workaround.

[James R. Ferguson · Accepted solution]

Hi (again) John:

As originally pointed out, you could cirect the output to a file ( '--output ' ) and then filter out the carriage-return characters with 'dos2ux'; OR:

# echo "${F}|perl -ple 's{\r$}{}'

...will remove the carriage-return too.

Regards!

...JRF...

[Steven Schweda]

> I am using the curl --use-ascii switch, but
> for whatever reason it doesn't work, isn't
> applicable, in this case / context.

I don't know what cURL does, but when wget
does ASCII FTP, it tells the server to do an
ASCII transfer, but it does not re-jigger the
line endings on what it receives. (Well,
that's what it does until I modify it. _My_
wget puts out locally appropriate line
endings on UNIX and VMS.) The FTP standard
(I believe) is to use CR+LF for ASCII
transfers, and a directory listing should be
done as an ASCII transfer, which may account
for what you see.

I suspect that, like wget, cURL has some
optional diagnostics, which you could use to
see exactly what it does in a case like this,
that is, which FTP commands it sends to the
FTP server. (You'd still need to look at the
code to see how it writes the output file.)

[John Kittel]

James - Thanks. The perl program is exactly what I need.

Steven - I think you're right about why it works that way. I'm not going to pursue it.



- John