Coverting csv file to binary

[Henry Chua]

Hi Merijn,

I gotten this when I try to run the script..
"Can't locate Text/CSV_XS.pm in @INC (@INC contains: /usr/lib/perl5/5.8.0/i386-l.
BEGIN failed--compilation aborted at ./test line 7."

do you know what i did wrong? how can i get the csv_xs.pm package?

best regards
Henry

[Henry Chua]

Hi guys,

Manage to fix the problem and make the module work. However, I was wondering whether i could modify the script to convert binary to csv as well..

can be done?

Best regards
Henry

[H.Merijn Brand (procura]

If you pass out some points to reflect the appreciation of the answers, I could give you the hint to use unpack or split and Text::CSV_XS to do the reverse. I could even bshow you some code.

Enjoy, Have FUN! H.Merijn

[Henry Chua]

No problem... ^_^

[H.Merijn Brand (procura]

OK, here's a real-life script that would convert almost anything text-like to csv or M$-Excel. You don't have (nor can get) PROCURA::Diac, so you need to find other methods to deal with Unicode conversions (Encode is a good start)

Spreadsheet::WriteExcel and Date::Calc are both included in my perl builds and are available on CPAN. The advantage of writing xls yourself, instead of having M$-Excel do it on import or open is that you have control over the dates, instead of having M$ fuck-up all valid dates to be forced to braindead US format

Also attached to have your script retain format. Hey, I even have a usage sub :)
--8<--- append_csv
#!/pro/bin/perl

use strict;
use warnings;

sub usage ()
{
print STDERR
"usage: $0 [-c utf8|iso8859] [-i ] [-o | -ms] [out_file]\n
",
" $0 -x [-c utf8|iso8859] [-d ] [-w ] o
utfile\n";
exit;
} # usage

use Getopt::Long qw( :config bundling nopermute);
my $codep = $ENV{UTF8_OK} // 0 ? "utf8" : "iso8859-1";
my $i_sep = qr/\s*\|\s?/;
my $o_sep = ",";
my $pfx = "";
my $xls = 0;
my $dtf = "dd-mm-yyyy"; # Excel date format
my $wdt = 4; # Default minimal column width
GetOptions (
"c|codep=s" => \$codep,
# CSV stuff
#!/pro/bin/perl

use strict;
use warnings;

sub usage ()
{
print STDERR
"usage: $0 [-c utf8|iso8859] [-i ] [-o | -ms] [out_file]\n
",
" $0 -x [-c utf8|iso8859] [-d ] [-w ] o
utfile\n";
exit;
} # usage

use Getopt::Long qw( :config bundling nopermute);
my $codep = $ENV{UTF8_OK} // 0 ? "utf8" : "iso8859-1";
my $i_sep = qr/\s*\|\s?/;
my $o_sep = ",";
my $pfx = "";
my $xls = 0;
my $dtf = "dd-mm-yyyy"; # Excel date format
my $wdt = 4; # Default minimal column width
GetOptions (
"c|codep=s" => \$codep,
# CSV stuff
"i=s" => sub { $i_sep = qr{$_[1]} },
"o|csv=s" => \$o_sep,
"m|ms" => sub { $o_sep = ";" }, # Micro$hit expects semi-colon
# Excel stuff
"x|xls|excel" => \$xls,
"d|date=s" => \$dtf,
"w|cwidth=i" => \$wdt,
) or usage;

use Text::CSV_XS;
use Date::Calc qw( Delta_Days );
use Spreadsheet::WriteExcel;
use PROCURA::Diac;

my $mode = $0 =~ m/clr/ ? "> " : ">> ";
if (@ARGV) {
my $ofile = shift;
if ($xls) {
($xls = $ofile) =~ s/(\.xls)?$/.xls/i;
}
else {
open STDOUT, "$mode $ofile" or die "$ofile: $!\n";
}
}
elsif ($xls) {
die "Excel formaat kan niet naar stadaard uit.\n";
}

my ($h, $w, $csv, $wbk, $wks, %fmt, @w) = (0, 1);
if ($xls) {
$wbk = Spreadsheet::WriteExcel->new ($xls);
$wks = $wbk->add_worksheet ();
%fmt = (
date => $wbk->add_format (align => "center", num_format => $dtf),
rest => $wbk->add_format (align => "left"),
);
}
else {
$csv = Text::CSV_XS->new ({
binary => 1,
sep_char => $o_sep,
always_quote => 1,
});
}
my $p_tex = $codep =~ m/^u(tf-?8|nicode)$/i ? \&TexUnicode : # utf-8, utf8, unic
ode
$codep =~ m/8859\D9|latin5/i ? \&Tex8859_9 : # iso8859-9, latin5
$codep =~ m/none|strip|cs7/i ? \&TexNone : # none, strip, cs7
\&Tex8859_1; # * (iso8859-1, lat
in1)
$p_tex == \&TexUnicode and binmode STDOUT, ":utf8";


while (<>) {
chomp;
m/\S/ or next;
s/^\s+//;
my @row = split $i_sep, $_, -1;
if ($xls) {
@row > $w and push @w, ($wdt) x (($w = @row) - @w);
foreach my $c (0 .. $#row) {
(my $val = $row[$c]) =~ s/^\s+//;
my $l = length $val;
$l > $w[$c] and $w[$c] = $l;
my @d = (0, 0, 0); # Y, M, D
$val =~ m/^(\d{4})(\d{2})(\d{2})$/ and @d = ($1, $2, $3);
$val =~ m/^(\d{2})-(\d{2})-(\d{4})$/ and @d = ($3, $2, $1);
if ($d[1] >= 1 && $d[1] <= 12 && $d[0] >= 1900) {
my $dt = 2 + Delta_Days (1900, 1, 1, @d);
$wks->write ($h, $c, $dt, $fmt{date});
next;
}
if ($val =~ m/^[\d.]{10,}$/) {
# Excel will convert long numericals to float
# 1000101222101220 ...
# It's in this case much more likely that it is a string
$wks->write_string ($h, $c, $val);
}
else {
$wks->write ($h, $c, $val);
}
}
}
else {
unless ($csv->combine (@row)) {
print STDERR "Data error: ", $csv->error_input, "\n";
exit;
}
print $p_tex->($csv->string), "\r\n";
}
++$h % 100 or printf STDERR "%6d x %6d\r", $w, $h;
}
if ($xls) {
$wks->set_column ($_, $_, $w[$_]) for 0 .. $#w;
$wbk->close ();
}
close STDOUT;
-->8---

Enjoy, Have FUN! H.Merijn

[Henry Chua]

Hi Merijn,

Many thanks for the script. I am in the midst of digesting - quite a big for my level.

One last thing I would like to know is whether converting binary to text is possible? if so can I used CSV_XS to achieve this?

I am terribly sorry for constant queries. Thanks for your invaluable assistance.

Best regards
Henry

[H.Merijn Brand (procura]

Binary to text is to vague a goal to solve.

Do you consider UTF8 binary? Or Unicode? Or even iso-8859-1?

If your defenition of "Text" is plain US-ASCII 7 bit, yes, it's probably possible, most likely possible even. But your conversion will not be lossless

What is "Binary" and what is "Text". That is the crucial question in your quest. Once you can answer that unambigouously, my answer would probably be "yes"

If your def of Binary is "fixed width records" that should be converted to html or other text based data files, do

# man perlpacktut

and teach yourself unpack

Enjoy, Have FUN! H.Merijn

[Henry Chua]

Hi Merijn,

From a look at a binary file is it possible to tell which coding it is using? For example the binary file I provided in the begining of the query, can you tell which format is it using?

And many thanks, You have been really been a great help, sorry if my question look weird to u..

Regards
Henry

[H.Merijn Brand (procura]

Your .null file is a LIF file, which is pretty much a binary format :)

a5:/tmp 102 > file *.null
226232.null: lif file
a5:/tmp 103 > lifls *.null | head
WS_FILE

a5:/tmp 104 >

There are several LIF commands:

lifcp, lifinit, lifls, lifrename, and lifrm

a5:/tmp 108 > lifls -l *null
volume HFSLIF data size 1052 directory size 1
filename type start size implement created
===============================================================
WS_FILE -5791 2 121 20200080 00/00/00 00:00:00
a5:/tmp 109 >

Now, maybe, you might mean that you want to know the format of WS_FILE

a5:/tmp 110 > lifcp 226232.null:WS_FILE .
a5:/tmp 111 > file WS_FILE
WS_FILE: data
a5:/tmp 112 >

Looking at the hex-dump of WS_FILE, it looks very parsable and very binary (about Volt, Ampere, and Ohm). Since I have no idea what it is, I cannot give you any guidelines here

00000100 00 00 00 06 52 36 39 34 30 32 00 00 00 06 52 36 ....R69402....R6
00000110 39 34 30 32 00 00 00 07 53 55 42 53 49 54 45 20 9402....SUBSITE
00000120 00 00 00 07 53 55 42 53 49 54 45 20 00 00 00 08 ....SUBSITE ....
00000130 4E 56 74 68 69 28 31 29 00 00 00 07 4E 49 64 73 NVthi(1)....NIds
00000140 28 31 29 20 00 00 00 08 4E 56 74 68 69 28 32 29 (1) ....NVthi(2)
00000150 00 00 00 08 4E 49 6F 66 66 28 32 29 00 00 00 07 ....NIoff(2)....

00000890 00 00 00 00 00 03 49 44 53 20 00 00 00 07 53 55 ......IDS ....SU
000008A0 42 53 49 54 45 20 00 00 00 04 56 54 48 46 00 00 BSITE ....VTHF..
000008B0 00 03 49 44 53 20 00 00 00 04 56 54 48 46 00 00 ..IDS ....VTHF..
000008C0 00 06 49 44 53 4F 46 46 00 00 00 03 49 44 53 20 ..IDSOFF....IDS
000008D0 00 00 00 05 42 56 44 53 53 20 00 00 00 08 56 54 ....BVDSS ....VT
000008E0 48 4C 43 44 32 46 00 00 00 03 49 44 53 20 00 00 HLCD2F....IDS ..
000008F0 00 08 56 54 48 4C 43 44 32 46 00 00 00 06 49 44 ..VTHLCD2F....ID

00000EC0 00 00 00 00 00 00 00 00 00 01 41 20 00 00 00 01 ..........A ....
00000ED0 41 20 00 00 00 01 56 20 00 00 00 01 41 20 00 00 A ....V ....A ..
00000EE0 00 01 56 20 00 00 00 01 41 20 00 00 00 01 41 20 ..V ....A ....A
00000EF0 00 00 00 01 56 20 00 00 00 01 56 20 00 00 00 01 ....V ....V ....
00000F00 41 20 00 00 00 01 56 20 00 00 00 01 41 20 00 00 A ....V ....A ..
00000F10 00 01 41 20 00 00 00 01 41 20 00 00 00 01 56 20 ..A ....A ....V
00000F20 00 00 00 01 56 20 00 00 00 01 41 20 00 00 00 01 ....V ....A ....
00000F30 41 20 00 00 00 01 41 20 00 00 00 01 56 20 00 00 A ....A ....V ..
00000F40 00 01 56 20 00 00 00 01 41 20 00 00 00 01 56 20 ..V ....A ....V
00000F50 00 00 00 01 56 20 00 00 00 06 4F 68 6D 2F 5B 5D ....V ....Ohm/[]
00000F60 00 00 00 06 4F 68 6D 2F 5B 5D 00 00 00 06 4F 68 ....Ohm/[]....Oh
00000F70 6D 2F 5B 5D 00 00 00 01 41 20 00 00 00 01 56 20 m/[]....A ....V

000016A0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000016B0 00 01 33 20 00 00 00 01 32 20 00 00 00 01 34 20 ..3 ....2 ....4
000016C0 00 00 00 01 31 20 00 00 00 00 00 00 00 00 00 00 ....1 ..........
000016D0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000016E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
000016F0 00 00 00 00 00 00 00 00 00 01 33 20 00 00 00 01 ..........3 ....
00001700 32 20 00 00 00 01 34 20 00 00 00 01 31 20 00 00 2 ....4 ....1 ..
00001710 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
00001720 00 01 33 20 00 00 00 01 32 20 00 00 00 01 34 20 ..3 ....2 ....4
00001730 00 00 00 01 31 20 00 00 00 00 00 00 00 00 00 00 ....1 ..........
00001740 00 00 00 00 00 00 00 00 00 01 35 20 00 00 00 01 ..........5 ....


Enjoy, Have FUN! H.Merijn

[samsonic221]

What converter or code did you use to achieve this?