FTP vs. SFTP behavior differences

[pcseunix]

We have some files that have a RFM:FIXED and a record length of 512. However, these files have a FFB (first free byte) attribute with a non-zero value. The data contained in the files is in an undefined (to VMS/RMS) format, but rather the format is special to the application that writes and reads it. As long as the file stays on OpenVMS, our standard tools all work correctly with it. But, in transferring one of these files to a PC causes problems. Using FTP from TCPIP V5.4 works correctly. That is, the resulting file on the PC is of the correct length. Using SFTP from the same version TCPIP results in a file that is some number of bytes longer than by using FTP. The number of bytes longer is the difference between 512 and the FFB attribute value.

Therefore, it appears that FTP honors the FFB attribute in the file header, but SFTP transfers the entire last 512 byte record. While the last few bytes are nulls (0x00h), the fact that the file is now longer causes problems with the tools on the PC.

Now, I can see that both tools (FTP and SFTP) are doing the Right Thing (FTP honors the FFB attribute, while SFTP says "the file header says 512 byte/fixed records, so that's what I'm sending").

If we could simply use FTP, we'd be all set (and I wouldn't be posting this), but we need to run this over an SSH connection (hence the SFTP).

So, how do we get SFTP to transfer the correct number of bytes in the file -- that is, recognize the FFB attribute and only send that many bytes from the last record?

Thanks!

[Thomas Ritter]

The number one rule for using SFTP from VMS to non-vms host, is to convert all files to stream_lf (unix format).
The good news is that it is easy to do on VMS.

Consider
$ convert/fdl=sys$input: sftp_dir.tmp sftp_dir.cmd
RECORD; FORMAT STREAM_LF

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1185836

[pcseunix]

Thanks for the quick reply.

Converting the file to Stream_LF using the CONVERT utility increases the filesize (as seen on OpenVMS), and this will cause the (locally written) OpenVMS tools to fail. Furthermore, the resulting file on the PC (after both FTP and SFTP) is larger than the original.

Which leads me to, how 'bout just setting the record format to Stream_LF. But, that doesn't work either, in that the resulting file will have an extra LF at the end of the file on the PC.

One thing we tried was setting RFM:UDF, and this seems to work with SFTP, but now the OpenVMS tools don't work

So far, it's coming down to how to get SFTP to only send the FFB bytes from the last block or record.

[Thomas Ritter]

Well that's bad news. Having just deployed a number of SFTP based file transfer systems I only repeat my advise. I suggest you have your tools upgraded or modified. Why don't they work ? COBOL programs ? The changes required maybe very small.

[Steven Schweda]

> So, how do we get SFTP to transfer the
> correct number of bytes in the file [...]?

Complain to HP?

If it's not too much trouble, you might try
using Zip on the VMS side, and UnZip (of some
sort) on the Windows side. I don't guarantee
that it will work, but it should hide the
actual record attributes from SFTP.

> One thing we tried was setting RFM:UDF, and
> this seems to work with SFTP, but now the
> OpenVMS tools don't work

What are "the OpenVMS tools" which don't
work?

Of course, the Zip scheme is probably more
work than setting the file attributes, doing
the SFTP, and resetting them.

[John Gillings]

As you've said, RFM:FIXED 512 and FFB are a contradiction. However, depending on what your PC is expecting, conversion to stream might not be correct, as it will insert an LF character at the end of every record.

Obviously you need to convert the file into something that SFTP will transfer in a form that your PCs understand. What if the extra bytes were all nulls? A simple way to make that happen might be to enable highwater marking on the source volume.

If the files are small enough (<32K), you might get away with setting the record length to the TOTAL size of the file.

$ SET FILE/ATTR=(RFM:FIXED,MRS:total,LRL:total)

(I can never remember if it's MRS or LRL, to set a fixed length so I just set both)

If it's >23K, perhaps find a factor of the total size <32K and set the fixed length to that size. Of course, if it's a prime number >32K you're out of luck ;-)

Another option might be to write a little utility on the PC which you can invoke via SSH to truncate the file to the "correct" size after copying. Since you know the FFB, you can send it in the SSH command.

BTW, on V8.3 and above it's even easier to convert:

$ convert/fdl="RECORD; FORMAT STREAM_LF
" sftp_dir.tmp sftp_dir.cmd

[John Gillings]

Collisions!

> One thing we tried was setting RFM:UDF, and
> this seems to work with SFTP, but now the
> OpenVMS tools don't work

OK, so you set the file to UDF, copy it, to the PC, then set it back to FIX for the OpenVMS tools?

You could report this to HP, BUT keep in mind your comment about RFM:FIXED and FFB. If this were accepted as a bug report, I'd expect the most likely result would be to "fix" FTP which is arguably the incorrect behaviour. RFM:FIXED is exactly what it says!

[Hoff]

This report reminds me of the odd usage that Windows had within some files. For reasons I never figured out, Windows stored data or metadata after the claimed EOF. Yes, data after the EOF. And this meant that you had to store the last block (garbage or otherwise) to account for this usage. CMS got nailed by this, IIRC.

I'd suggest moving to a transfer format that is more portable (XML, XML-encoded or encapsulated binary data, or your own wad of binary data) and particularly to an approach rather less dependent on the vagaries of either Windows or of OpenVMS. (But then you very likely know that is the best approach, and have an installed base you have to deal with.)

IIRC, there have been some fixes in this area over the years within RMS, too. LRL has seen a few tweaks, and FFB and EOF have seen changes.

From the FAQ: "The number of bytes in a file can be determined by multiplying the number of blocks allocated for the file times the number of bytes in a block. For sequential files (only), the FFB (XAB$W_FFB, in the File Header XAB) value can be used to find out how much of the last (XAB$L_EBK) block is used. FFB and EBK are meaningful only for sequential files, and only in a limited context-partial block allocations are not permitted. For other file formats, the EOF marker is not meaningful."

The RMS manual specifically says that the FFB field is only meaningful for sequential files. Which one could infer as indicating that this usage is unsupported, at best.

Stephen Hoffman
HoffmanLabs LLC

[Pim van Velzen]

An alternative might be to use ftp over a secure (tunnel) ssh session.

[Joseph Huber_1]

If there is no solution for the file format or size problem, then a possible solution could be to establish an STUNNEL for FTP: this should fulfill the requirement for encryption as well as sftp does.
STUNNEL is available for VMS (see the open source tools page on HPs VMS site), and Unix type systems, just don't know if and how for Windows.