topic Re: How to convert ebcdic data? in Operating System - HP-UX

How to convert ebcdic data?

Robert Fisher_1 — Tue, 22 Jul 2003 19:47:17 GMT

Hello experts,

I have some legacy data that comes in from an IBM mainframe. All the string data is in ebcdic format and I need to convert it to ascii. My problem is that when I use dd to convert the data all the strings are ok but the binary data is garbage. If I don't use dd, the binary data is perfect but the strings are garbage. Is there anyway to make dd convert only the strings and leave the binary parts of each record alone.

TIA,
Bob

Re: How to convert ebcdic data?

A. Clay Stephenson — Tue, 22 Jul 2003 19:50:29 GMT

Sorry Bob. That dog won't hunt. Dd is all or nothing in this context. It's time to use C or Perl. You can use the Convert::EBCDIC module from www.perl.org/CPAN to do the conversion and use the unpack() and pack() Perl functions to manipulate the data. You will need to know the exact data layout of each record.

Re: How to convert ebcdic data?

Paul Sperry — Tue, 22 Jul 2003 19:55:43 GMT

dd conv=ascii Convert EBCDIC to ASCII.

Re: How to convert ebcdic data?

Elena Leontieva — Tue, 22 Jul 2003 19:57:20 GMT

Robert,

My company purchased a FilePort for Unix product from SyncSort specifically for this purpose. The product is good and easy to use.

http://www.syncsort.com/products.htm

Elena.

Re: How to convert ebcdic data?

A. Clay Stephenson — Tue, 22 Jul 2003 20:20:10 GMT

Actually, if you know just a little Perl this is really, really easy. In your case, you don't need to worry about the individual binary fields because you can simply lump them together - as long as they are adjacent. It makes this conversion quite simple if you know that your binary data imports okay.

Re: How to convert ebcdic data?

Robert Fisher_1 — Tue, 22 Jul 2003 20:41:20 GMT

Hi Clay,

I have several files to convert. The simplest one has just a few fields:
Offset Length Description
0 4 Binary integer
4 20 ebcdic string
24 2 Binary integer
26 20 ebcdic string
46 60 ebcdic string
106 48 array of 4 12-byte BCD integers
------
154 bytes total

Can you show me a perl example and then I should be able to handle the other files. Please, please??

TIA,
Bob

Re: How to convert ebcdic data?

A. Clay Stephenson — Tue, 22 Jul 2003 21:07:20 GMT

Okay Bob,

This looks like duck soup. Here's my untested (even for syntax) example based on your layout:

--------------------------------------

#!/usr/bin/perl -w

use strict;
use Convert::EBCDIC;

use constant TYPEDEF => 'a4 a20 a2 a20 a60 a48';
# declare each field as binary data, 'a' format follwed by length
# note that we can treat the 4 12-byte BCD's as one field because we
# are not converting them

my $recsz = length(pack(TYPEDEF,()));
my $b;
my $n = read(STDIN,$b,$recsz);
while ($n == $recsz)
{
# unpack fields specified using TYPEDEF in array
my @a = unpack(TYPEDEF,$b);
# convert each EBCDIC string to ASCII $a[0] = 1st field, $a[1] = 2nd, ...
my $s1 = Convert::EBCDIC::ebcdic2ascii($a[1]);
my $s2 = Convert::EBCDIC::ebcdic2ascii($a[3]);
my $s3 = Convert::EBCDIC::ebcdic2ascii($a[4]);
# pack data using unchanged fields from array for binary and
# converted EBCDIC strings where appropriate
my $out = pack(TYPEDEF,$a[0],$s1,$a[2],$s2,$s3,$a[5]);
print $out;
$n = read(STDIN,$b,$recsz);
}
-------------------------------------------

Note that this script functions as a filter, so

convert.pl < oldfile > newfile

Again, you will need to download and install the Convert::EBCDIC module. Installing is easy. After downloading, gunzip it, untar it, and view the README file. It will guide you through the rest of the module install. Again, check this very carefully because I typed in 'on the fly' -- and I ain't exactly known for my good typing.

Re: How to convert ebcdic data?

Elena Leontieva — Wed, 23 Jul 2003 12:35:18 GMT

Bob,

We used FilePort to migrate from mainframe to UNIX, a lot of data. So, if you need to convert just several files, you do not want to purchase this software, but you probably can get a demo/trial version for free.

Elena.

Re: How to convert ebcdic data?

Robert Fisher_1 — Wed, 23 Jul 2003 13:14:37 GMT

Hello,

The perl script actually worked. I just cut and pasted it and ran it. We have been thinking that it would make it easier to process on the HP end if we added a linefeed character to each record. Can we just add a '\n' to the print command? Also, is there some documentation on the pack and unpack procedures?

Thanks,
Bob

Re: How to convert ebcdic data?

Massimo Bianchi — Wed, 23 Jul 2003 13:16:31 GMT

Clay is Clay. dot.

[please no points]

Re: How to convert ebcdic data?

A. Clay Stephenson — Wed, 23 Jul 2003 13:23:18 GMT

Hi Bob:

I'm glad my advanced one-finger hunt-and-peck typing actually worked. To add a LF, all you need to do is change print $out; to print $out,"\n"; However, THIS IS DUMB. You still need to process it as fixed length records because in your binary data you could easily have a 0x0a (LF) byte that would be perfectly valid data but not be the end of the record.

To get more documentatio, do a man perlfunc. All the standard Perl functions are lumped into this one man page so it's not possible to get jusy the pack and unpack functions. It's a rather big man page so I would print it. viz man perlfunc | lp.