FTP between Windows & HPUX

[ITeam]

Now this is a bit bizarre…

In FTP, when using a mix of Windows and Unix systems then the FTP should be set use "ASCII Mode".
In this mode end-of-line sequences should be handled correctly (namely, from LINEFEED on Unix, to CARRIAGE_RETURN/LINEFEED on Windows).

However, what we see when using this mode with the HP box is that we are getting the interpretation of LF to CR/LF seemingly done by the HP box, not by the FTP process. As a result, when we transfer a file of, say, 100 records it will be 100 data bytes short - though actually exactly the same number of bytes overall. What has happened is that each record is padded with the CR character, and real data is truncated off the end.

One way to fix this, seemingly, is to set Comlink to use "BINARY" mode. This should bring over the file "as is", without conversions. But instead, in this case, we get the correct file, with CR/LF and the right number of total bytes.

Since this doesn't happen with Sun and DG boxes, my guess is that it is somewhere in the setup of the FTP "daemon" on the HP box. Alternatively, the Comlink front-end is jiggered (getting things the wrong way round) - but if that was the case, I'm sure we'd know about it already!

Thanks,

Tony.

[Allan Bowman]

From my experiences with mixed system FTP, I would say to always ignore the rules and never expect known results! I usually set things up the way they "should" work and then start tweaking. If you have found a way to transfer the file and get the results you want, then you should probably make a note for future reference for that specific transfer type. Of course that doesn't mean that some patch or other OS revision on either end isn't going to change the behavior some time in the future...

Allan

[A. Clay Stephenson]

Since you haven't bothered to identify the OS version, it's not possible to be specific but, for example, under 11.11 there is an ftp/ftpd patch. Search the patch database; this is always a good starting point although I must say I have never seen the symptoms you describe. I would see how the vanilla Windows FTP client behaves. You should also put ftp in verbose mode.

[ITeam]

Thanks. It is running 11.23 (Itanium). Will check the patch status.

Tony.

[Stephen Keane]

You can use the "cr" command in ftp on HPUX to toggle CR/LF stripping. The default is to strip CR from CR/LF pairs (when in ASCII mode).

You might experiment with this command and post your results?

[Steven E. Protter]

I have seen this behavior.

Its not an HP-UX problem.

the Windows ftp client is a piece of junk.

1) It defaults to ascii and leaves it up the the user to switch to binary.

2)It is not smart enough to automatically determine the filetype as sftp does.

Your best solution is to stop using ftp all together. Buy yourself a little free sftp client and install secure shell for free on the hpux side.

If you must use FTP then get a better client. There are decent free one's available on the web.

There is nothing that need be done to the ftpd server on HP-UX. It is for security reasons an excellent idea to download and install the patches.

Some patches are first set up binary only but eventually they are all released in depot format.

SEP

[Bill Hassell]

ftp is probably the most universal industry standard for file transfer that there is. To transfer a file exactly, you use BINARY. When you use ASCII mode, a *BIG* translation machine starts up. Here's how it works:

1. On the local machine, each ASCII record is read in a format-neutral form. The exact length of the record is known to ftp and will be sent to the remote system with nothing on the front of back except the length informtion.

2. On the remote computer, the record is then converted to the native format for that system. For Unix, each line has a line termination character of LF (nothing else) and a null is cteated at the end of the ASCII file signifying the end. On a Windows box, the record is terminated with 2 chars: CR+LF and the end of the file has a CTRL-Z. For mainframes and business computers, the record may be converted to fixed length, or it may have a record length code inserted in front of each record, or a multitude of possible file format features.

So ASCII will almost always corrupt a binary file. What happens is that on the source side, a binary sequence of data might have the equivalent of an LF imbedded in the middle but it isn't an end-of-line indicator. ftp doesn't know this because you lied and said it was ASCII, so the special codes are added as defined by the ASCII translation defined in ftp.

So, NEVER transfer a file in ASCII unless it is really an ASCII file. And if you do transfer an ASCII file using BINARY mode, the source system's internal code will be transferred too and this will confuse the destination system.

So for Windows, a Unix ASCII file trasnferred in BINARY will be displayed by Notepad with little black squares at the end of each line, but open that file in Word or Wordpad and it sort-of looks OK. The LF (which Unix uses) is actually a soft-CR code for Windows word processors.

And for Unix, transfer an ASCII file in BINARY and in vi, you'll see ^M at the end of each line. The reason is that the CR+LF is seen as CR and the end of the line. CR is ^M on the keyboard.

You can filter files like this in HP-UX with dos2ux and ux2dos (see the manp page). NOTE: This is also what happens when sharing files using NFS or SAMBA. The shared filesystem is not translated so the files are effectively seen in BINARU mode.

[A. Clay Stephenson]

Binary mode should bring the file over absolutely intact. I begin to question what you are using to actually view the file. For example, on UNIX you can use od (or xd) to view the file and see all the ASCII characters as well as the control characters. I would be very interested in knowing the before amd after results displayed in a known good tools using binary transfer.

[Arturo Galbiati]

Hi,
this is an excerpt from man ftp:

cr Toggle carriage return stripping during ascii type file
retrieval. Records are denoted by a carriage-return/line-feed
sequence during ascii type file transfer. When cr is on (the
default), carriage returns are stripped from this sequence to
conform with the UNIX single line-feed record delimiter. Records
on non-UNIX remote systems may contain single line-feeds; when an
ascii type transfer is made, these line-feeds can be
distinguished from a record delimiter only when cr is off.

HTH,
Art

[ITeam]

Thanks for all your help.

Managed to solve the problem by tweaking the Windows Comlink.

Tony.