ftp stripping end-of-line characters

[Alan Casey]

Hi,
I am ftpinf from HP-UX 11.0 to an IBM mainframe. And when I do the end of line characters from the file are missing!
Transfer type is ascii

any ideas?

[Peter Godron]

Alan,
have you tried binary transfer?
Or it may be a EBCIDIC (sp?) problem?

[Alan Casey]

I tried binary, but the file is not readable on the mainframe in this format

[Luk Vandenbussche]

Hi alan,

The 3270 emulation software normally has an option to send or receive a file from the mainframe.

Have you tried this option yet?

[James R. Ferguson]

Hi Alan:

I believe that you need to toggle EBCDIC mode.

Regards!

...JRF...

[Alan Casey]

unfortunitely in EBCDIC mode nothing transfers at all! no errors just no data gets transferred

[James R. Ferguson]

Hi Alan:

Is the file on the mainframe a fixed size record-oriented one?

Regards!

...JRF...

[Alan Casey]

The problem seems to be due to the fact that the file is an XML file.
Other files transfer fine.

[Peter Godron]

Alan,
thanks for this additional info.
There is a problem with ftping XML files. The problem is the translation of the end-of-line character(X13X10) to X85.
Quick solution appears to be a NFS mount?!
Other solution may be available via a friendly search engine of your choice.

[Dave La Mar]

Alan -
A lot depends on the file type sent and the received.
As James pointed out, if the file is going to sequential disk then a Blocksize, Record Size and Record Format must be specified. If the file is going to a mainframe library this is not necessary.
If going into an ESDS mainframe file, the Record Size must be specified, as well as the optional Reuse parameter.
FTP is not recommended for mainframe KSDS files as the keys do not translate properly.
We perform hundreds of these per day most of which are FB Sequential Disk files.
Most common mode of transfer in our shop is ascii.

Perhaps, if you gave us a bit more information on the file and maybe a sample of the first 10 lines, we could be more helpful.

Best of luck.

Regards,

dl

[rick jones]

any chance the file could be gzipped prior to transfer and then unzipped as a workaround to the EOL manipulations?

[Dave La Mar]

Alan-
Rick brings up a good point.
The problem is the "mainframe" you mention. Speaking in IBM-ese, OS390 and VSE deal with EBCDIC characters and, thusly, their translation.
Aa well as not all shops have zip/unzip capability in OS390 and VSE.

While we perform zips and unzips in VSE I think your problem will lie in the application accessing this file once it is received by your "mainframe".

Zipping is certainly worth a try.

I really want to see a solution on this one.

Best of luck.

Regards,

dl

[Steve Post]

Here's a guess. The end of line character is not getting stripped off because it NEVER EXISTED? Perhaps you method of viewing the original file automatically formated the file (aka inserted the line breaks), as you viewed it? So nothing is being stripped? Only the viewer you use after it is moved does not have this pre-formatting?

It's interesting to note that I get an xml file via ftp. It also has no line breaks. But I have a tilde (~) at the end of every line. After I get the file, I do this:
cat myfile.txt | tr "~" | "\012" > myfile2.txt

[Peter Godron]

Alan,
have these answers helped?
Can you please update.

[BUPA IS]

hello alan ,
are you transfering the file to a "normal " mainframe file or directly to a uss file if it is normal please include details of the lrecl and recfm valuse of the target file and in any case the output of quote stat
the ibm normal datasets have no concept of an eof /eol charcter it is all done by logical record length if this is too short you may loose the ends of lines

[Steve Post]

01#!/usr/bin/perl
02# file hex85_rtn.pl
03# replace hex value 85 with a hard return.
04# file going in is stdin
05# file going out is stdout
06#
07while( <> ){ # read the whole big 45k line
08 @array=unpack("C*", $_); # convert it into a really big array
09 foreach $x ( @array ){ # go through each char in array
10 if ( $x == 133 ){ # if it's ascii value 133
11 print "\n"; # print a carriage return
12 } else { # otherwise
13 printf ("%c", $x); # print the Char of that ascii value
14 } # end of if
15 } # end of looping array
16} # end of loop each line (ONE big line though)

I don't know if there would be any problems with this program. Feel free to blow holes in it. I know one problem is I the file has one line of 45,000 characters. I would rather pull out 10 characters at a time, not 45,000.

steve

[Alan Casey]

Hi guys, thanks for the replies.
I worked out a simple solution for this.

Editing the data file before ftp'ing as follows:

fold -w80 [input file] > [output file]

[Steve Post · Accepted solution]

Ah but fold will only work for fixed length lines. I was interested in your problem because I had the same issue. But in my case, the records are variable length.

A coworker gave me a fix the bug in MY perl script.

$sep_tmp="$/;
$/ = "\x85";
while (<>)
{ just like before. }
$/ = $sep_tmp;

My little dilemma was I didn't want to read one line of 45,000 characters. The $/ is the perl name for line separator. By using "\x85", the script automatically took the file in chunks broken up at each hex85. It termed these "chunks" to be lines.

I used $sep_tmp to hold the original value of perl's line separator.

Still.... I never knew about that fold command.