topic Re: Unknow characters in Operating System - HP-UX

Unknow characters

Hunki — Tue, 06 Feb 2007 16:41:04 GMT

We are getting a file from mainframe for our unix box through the connect direct software which has special characters. My question is how to deal with these special characters ( ^@ ) while the file is being transfered ( at the time of transfer ) over from mainfraime to our box through connect:direct ( NDM ). The special character is not visible during "vi" but is visible while doing a "less filename"

Re: Unknow characters

A. Clay Stephenson — Tue, 06 Feb 2007 16:58:43 GMT

Data conversion can be very data dependent. Are you certain that these characters are not needed? If so, you can setup a trival filter using tr to delete all non-printable characters.

tr -cd "[ -~\012]" < infile > outfile

This will delete all characters not in the range through tilde and not a LF.

If these special characters are binary data then you will have to craft a custom conversion. Perl is very good at doig selective EBCDIC to ASCII conversion and can convert one binary data format to another.

Re: Unknow characters

Hein van den Heuvel — Tue, 06 Feb 2007 17:02:41 GMT

^@ id most likely a representation of ascii NUL.

Is the data supposed to be readoable ascii, or perhaps ebcdic or binary/raw data

Control- being 64 less than the character. See for example: http://www.asciitable.com/

The best way to analyze the situation is to use 'od' or 'xd' in some flavor ( -c ).

It the data is supposed to be text already, then you may need to write a little post-transfer filter (perl? tr -d \0000 ?).

For better help please consider a reply with
an .txt attachment with output from
head -10 | xd - c > xd.txt

hth,
Hein.

Re: Unknow characters

Hein van den Heuvel — Thu, 08 Feb 2007 09:35:28 GMT

I'm curious. Was this resolved?
What was the outcome?
Regards,
Hein.

Re: Unknow characters

Hunki — Tue, 13 Feb 2007 17:43:27 GMT

Attaching the file for your reference. Any help would be greatly appreciated.

thanks,
hunki

Re: Unknow characters

Hunki — Tue, 13 Feb 2007 17:45:35 GMT

There are two types of files here ...file 1 is binary/date file and file 2 is ascii text file.

Re: Unknow characters

Hunki — Tue, 13 Feb 2007 17:47:14 GMT

Need to find out on how to translate those "null" characters into spaces either during the Connect:Direct transmission or upon receipt of the file.

Re: Unknow characters

Hein van den Heuvel — Tue, 13 Feb 2007 18:34:13 GMT

Well, you did not pie through 'xd' as requested, so now the .txt file, or my notepad may have munged funny characters.

I do see the first head command being followed by and empty line (edit error?) and then a 'funny' character with valye 0xE2 (226 decimal). I was guessing it might be a line length, but that appears to be 101 (decimal)

If you want to fix whatever the first char is to become a space, then you could probaly use (untested):

perl -pe "s/^./ /" >

This just tells perl to loop through , substitute the first char in a line with a space, and print onto

Hein.

Re: Unknow characters

Bill Hassell — Tue, 13 Feb 2007 19:18:41 GMT

The transferred data looks like it contains JCL codes and is likely a very specialized file on the mainframe. Mainframes have hundreds of file formats and in fact, a plain old ASCII file is quite unusual on a mainframe. To transfer a useful file from any non-Unix computer will require discussions with the mainframe system administrators. Translating the one character isn't going to be the end of your work.

I would display the file in hex+ASCII so you can see everything else that needs fixing:

xd -xc mainframe.file | more

Note that if you are transferring the file first from mainframe to a PC and then PC to HP-UX, there other precautions you need to take to preserve the data.