topic Re: Curl and --crlf in Operating System - OpenVMS

Curl and --crlf

James T Horn — Wed, 11 Oct 2006 13:21:38 GMT

When using Curl (Version 7.15.3) to try and transfer a Fixed length 250 byte record, Carriage Return Carriage control to a FTPSERVE IBM FTP CS V1R6. They seem to need CrLf and Curl always get's an Partial_file error on any format other than Stream_LF.
Any help would be greatly appreciated.

Re: Curl and --crlf

Steven Schweda — Wed, 11 Oct 2006 14:53:33 GMT

As usual, it might help to see the actual
command you used, and its output.

What's an "FTPSERVE IBM FTP CS V1R6"?

Are you trying to do a binary transfer or an
ASCII transfer? You're sending from the VMS
system, not getting on the "FTPSERVE" thing?

> They seem to need CrLf [...]

Who are "they"?

> Curl always [gets a] Partial_file error
> on any format other than Stream_LF.

I haven't looked at the "curl" code in this
neighborhood, but it would not be unusual for
a UNIX-oriented program to be confused about
file size on VMS, when the file is not
Stream_LF or fixed-512 (Record attributes:
None).

A binary FTP transfer should get all the
bytes to the other end, but if there are line
endings which need to be adjusted, then an
ASCII transfer may be needed. If that
doesn't give you the results you want, then
you may need to fiddle with the file before
sending it, so that it gets reconstituted as
you'd like at the other end.

It might help to see a DIRE /FULL on the
file, and a DUMP of the first couple of
records, and some idea of how the file is
supposed to look at the other end.

Also, for an outbound FTP transfer, I'd
probably try COPY /FTP before I resorted to
a loose program like "curl" (or "wput").

Re: Curl and --crlf

James T Horn — Wed, 11 Oct 2006 15:19:59 GMT

The Texas state Controller's office is wanting a Secure transfer using ssl of some files. They recommend BlueZone but we are needing to transfer from OpenVMS.

I've tried the --crlf parameter which does not seem to make a difference. I convert the file from a Fixed Length, 250 byte record, Carriage Return Carriage Control to a Stream_LF, Carriage Return Carriage Control.

Re: Curl and --crlf

Steven Schweda — Wed, 11 Oct 2006 17:12:14 GMT

> As usual, it might help to see the actual
> command you used, and its output.

Still true.

> The Texas state Controller's office is
> wanting a Secure transfer using ssl of
> some files. They recommend BlueZone but
> we are needing to transfer from OpenVMS.

If the file is in some unusual VMS-specific
format, how much sense will it make at the
destination?

> I've tried the --crlf parameter which does
> not seem to make a difference.

I know nothing, but if you're trying to use
the "ftps" scheme, I wouldn't bet on it
knowing the difference between ASCII and
binary the way plain-old FTP does. So curl's
--crlf option might have no effect with ftps.

> I convert the file from a Fixed Length, 250
> byte record, Carriage Return Carriage
> Control to a Stream_LF, Carriage Return
> Carriage Control.

Convert using CONVERT? How?

> It might help to see a DIRE /FULL on the
> file, and a DUMP of the first couple of
> records, and some idea of how the file is
> supposed to look at the other end.

Still true.

Re: Curl and --crlf

James T Horn — Thu, 12 Oct 2006 08:43:49 GMT

> As usual, it might help to see the actual
> command you used, and its output.

$ convert/fdl=fdl$source:ucssftp.fdl 'filename' ucswork:'f$parse(filename,,,"name")'.curl
$ curl --config ucscom:usas06rmt.cfg --user 'ftpuser':'site_password' --upload-file ucswork:'f$parse(filename,,,"name")'.curl ftp://'site':9021/"''remote_filename'"

The fdl is:
SYSTEM
SOURCE OpenVMS

FILE
ORGANIZATION SEQUENTIAL

RECORD
BLOCK_SPAN yes
CARRIAGE_CONTROL carriage_return
FORMAT stream_LF
SIZE 0

The Curl Config file:
--anyauth
--disable-epsv
--dump-header ucslog:usas06r.dmp
--ftp-method singlecwd
--ftp-ssl
--insecure
--progress-bar
#--trace ucslog:usas06r.trc
--trace-ascii ucslog:usas06r.trc
--use-ascii

> It might help to see a DIRE /FULL on the
> file, and a DUMP of the first couple of
> records, and some idea of how the file is
> supposed to look at the other end.
Record format: Fixed length 250 byte records
Record attributes: Carriage return carriage control

1075300000099061009
12C200709012006 0608 DIRECTOR, CONTRACTS & GRANTS DIRECTOR, CONTRACTS & GRANTS

Re: Curl and --crlf

Steven Schweda — Thu, 12 Oct 2006 09:23:09 GMT

I'm beyond my range of experience (and too
lazy to inspect the code), but guesswork
would say that with "--ftp-ssl", your
"ftp://" URL will act like an "ftps://" URL,
which you probably want. Further, I'll
venture that with ftps, "--use-ascii" and
"--crlf" have no effect, as it's not really
(normal) FTP any more.

You're already using CONVERT on the file to
make it Stream_LF. If you want CR-LF line
endings, I'd expect " FORMAT stream" (instead
of "FORMAT stream_lf") to do that.

> 1075300000099061009
> 12C200709012006 0608 DIRECTOR, CONTRACTS & > GRANTS DIRECTOR, CONTRACTS & GRANTS
> [...]

By "DUMP", I meant "DUMP", as in "DUMP
file_name", but it might not tell me much,
anyway. You might wish to see if the
CONVERT (any of them) actually leaves you
with fixed-length records.

If "curl" emits a "Partial_file error" with
the STREAM format file, and/or if the file at
the destination is actually truncated, then
the "curl" code may need some work.
Alternatively, it may help to ditch the
"--use-ascii" option, as once the file data
include CR-LF line endings, it may do more
harm than good.

Re: Curl and --crlf

GuentherF — Thu, 12 Oct 2006 16:21:59 GMT

It is quite possible that the input files already has the right whatever_LFs embedded. And if the record format from DIR/FULL shows fixed then doing a CONVERT to whatever_LF does add characters to the output file.

Without a DIR/FULL and DUMP/BLOCK=COUNT:1 output I can't tell. If you want give this a try:

$ COPY org test
$ SET FILE/ATTRIBUTE=RFM:STM test
$ curl test

/Guenther