cURL file upload error with content-length mismatch

[Andreas Kaempfe]

On VMS-Alpha:
When using an http-file-upload the file is not sent to the Server with the
correct content-length.
Sending a file with 511 or less bytes, content-length 512 is used.
Sending a file with 513 - 1023 bytes, content-length 1024 is used.
Files with a length of a multiple of 512 Bytes show the correct
content-length. Only these files work for upload.

File test.doc has the size 21.504 (multiple of 512)
File test.ras has the size 5.686 (not a multiple of 512).

Using test.doc everything works.
Using test.ras the Content-Length in the LOG-File differs from the content
length on Linux/Windows.
The receiving program (in my testcase "upload.pl") says that I have a
post-data mismatch.

VMS shows:
002c: Content-Disposition: form-data; name="myfile"; filename="test.ra
006c: s"
0070: Content-Type: application/octet-stream
0098:
=> Send data, 6144 bytes (0x1800)

Linux + Windows show:
002c: Content-Disposition: form-data; name="myfile"; filename="test.ra
006c: s"
0070: Content-Type: application/octet-stream
0098:
=> Send data, 5686 bytes (0x1636)

My workaround: I wrote a perl-program to add some bytes to the file to have
a multiple of 512 Bytes. The CGI-Program (upload.pl) then reduces the
filesize back to the origin.

This workaround works for all files with all sizes - but the error in curl
still remains.

Now I have to say, that I'm not a C-Programmer, so I kindly ask for help.

Used CURL-Version on all systems: 7.18.2.

[Duncan Morris]

Andreas,

what is the format of your source file on the VMS system?

Can you post a DIR/FULL for test.ras ?

Duncan

[Harald Schwarz]

Hello Duncan,
Andreas first opened this point for me - now I think it's better that I answer - because I have the problem.

$ dir /full test.ras

Directory DRA3:[USER]

TEST.RAS;1 File ID: (168209,132,0)
Size: 12/118 Owner: [USER]
Created: 18-AUG-2008 15:32:26.01
Revised: 18-AUG-2008 15:32:26.03 (1)
Expires:
Backup: 18-AUG-2008 21:37:59.55
Effective:
Recording:
Accessed:
Attributes:
Modified:
Linkcount: 1
File organization: Sequential
Shelved state: Online
Caching attribute: Writethrough
File attributes: Allocation: 118, Extend: 0, Global buffer count: 0
No version limit
Record format: Fixed length 512 byte records
Record attributes: None
RMS attributes: None
Journaling enabled: None
File protection: System:RWED, Owner:RWED, Group:E, World:E
Access Cntrl List: None
Client attributes: None

Due to your hint I tested following:
Use a file (test.ras) that is stored with
Record format: Fixed length 512 byte records
and a file (trace.log) that is stored with
Record format: Stream_LF, maximum 0 bytes, longest 32767 bytes
Uploading test.ras, curl uses the wrong file-size as described.
Uploading trace.log, curl uses the correct file-size.

But: I need a solution to upload any file. I have binary files - max. size could be 20 MByte or more. And I do not have the chance to set the record format for every file during creation, because the files come from different systems (VMS-VAX, Unix, Windows, per ftp-client, ftp-server).

What would you recomend?

[Duncan Morris]

Harald,

I see that this has already been logged as a bug with cURL!

This is fairly typical behaviour for software designed for *nix style filesystems.

As you have already found, using STREAM_LF will produce the best results.

Would it be possible for your upload script to use CONVERT/FDL to convert the source file to STREAM_LF format immediately prior to invoking cURL?

PS: I see that cURL 7.19.0 is now available

[x2084]

Did you say the file has fixed length records of 512 bytes and the file size is not a multiple of 512?

[Duncan Morris]

my last reply got chopped by the cursed itrc software!

I should have also said to try

SET FILE/ATTRIB=RFM=STMLF test.ras

Depending upon the original source of the file, this may be all that is needed, rather than having to use CONVERT

Duncan

[Steven Schweda]

> Did you say the file has fixed length
> records of 512 bytes and the file size is
> not a multiple of 512?

It can happen. Start with an odd-sized file
on a UNIX(-like) system, and use binary FTP
in a VMS FTP client program to transfer it to
a VMS system. The VMS FTP client typically
creates a fixed-512 file. The FFB should be
set, so that an application can still know
where the original end-of-file is, but the
"Record format: Fixed length 512 byte
records" stuff could fool someone.

ra# uname -a
SunOS ra 5.10 Generic_127127-11 sun4u sparc SUNW,UltraSPARC-IIi-cEngine

ra# ls -l fred.zip
-rw-r--r-- 1 root root 6995 Jul 31 09:40 fred.zip


alp $ tcpip show version

HP TCP/IP Services for OpenVMS Alpha Version V5.4 - ECO 7
on a COMPAQ Professional Workstation XP1000 running OpenVMS V7.3-2

alp $ copy /ftp /binary ra"root passwd"::fred.zip []
alp $ dire /full fred.zip

Directory ALP$DKA0:[SMS.FTPTEST]

FRED.ZIP;1 File ID: (34824,11519,0)
Size: 14/69 Owner: [SMS]
[...]
Record format: Fixed length 512 byte records
[...]

So you might think that its size is 14* 512
bytes = 7168 bytes, but it's not, really:

alp $ write sys$output f$file_attributes( "fred.zip", "ffb")
339

Or:

alp $ bash
bash$ ls -l fred.zip
-rwxr-x--- 1 SMS 40 6995 Sep 3 23:27 fred.zip


Using my VMS adaptation of wget with its
--ftp-stmlf ("Use Stream_LF format for all
binary FTP files") option might help in some
cases, by avoiding the fixed-512 file format,
but that's a fetch-only program, not a
sender.

If these files are being created on the VMS
system, then wget would seem not to help
here, but if the files are being fetched from
a UNIX system, then "wget --ftp-stmlf" might
be useful.


> [...] rather than having to use CONVERT

I'd expect CONVERT to mess with the data in
undesired ways. "SET FILE/ATTRIB=RFM=STMLF"
sounds better to me.

It might help to know how these files got to
be fixed-512 in the first place.

Does sound like a cURL bug, though, if it's
ignoring the FFB (EOF marker) when it reads a
fixed-512 file.

[x2084]

I know that it can happen, I just wanted to point out the "incompatibility".

>>>
Does sound like a cURL bug, though, if it's
ignoring the FFB (EOF marker) when it reads a
fixed-512 file.

Yes, it looks like some software is reading record by record without checking if the last read record was complete.

I would set the RFM to UDF rather than to STM_LF. Stream LF imposes a record structure based on the content. There was no record structure or there were fixed length records. The STM_LF doesn't match that

Whatever tool created the file (even if it is a copy/ftp/bin) the RFM looks wrong or at least confusing. If you do a binary ftp-get from a Unix system, why should the result have any VMS record format at all?

[Steven Schweda]

> [...] STM_LF doesn't match that

True, but it works well with the C RTL.

> [...] why should the result have any VMS
> record format at all?

A fair question. I assume that fixed-512 is
a very old convention which (almost) everyone
seems to follow. Using anything else might
break a whole lot of existing software.

[Harald Schwarz]

I tested your suggestion:
SET FILE/ATTRIB=RFM=STMLF test.ras
and used curl for upload again - it worked.
Now I'm really happy :-))

Thank you all for your support - it saved me a lot of time.

To complete the information:
Some files come from a CAD-system on VMS-VAX. The system is quite old and stores the files always with 512 Bytes fixed record length. Due to the mass of interfaces and connected systems I cannot demand that I get no 512 Byte fixed record length.

What next:
The whole thing seems to be a BUG of CURL - and I think that the tool ignores the FFB (EOF marker) - as some of you already mentioned. I will add a comment to the curl bug tracking list.

[Andreas Kaempfe]

Thread closed because solution was found.