Help with a script!

[MAD_2]

I would like to know if some of you can provide some ideas on how to start with a script here:

Situation example (see attached file) -- Description:
1. Each two lines include date, time, a queue name, a fax ID number, a caseID number, the user ID, and the case type.
2. The file is a log and each data row is actually split into two lines each (there is an end of line character separating each row)
3. Most data rows contain the information as described in 1, however some (Errors) exclude parts of this information.

Objective:

1. To strip the file of the end of line characters separating each row (odd lines), placing each row of data into one line each.
2. To extract some of the data and put it into a raw file separated by any type of delimiter (pipes or commas is OK). Data required is date, time (only hour+minutes+seconds), the faxID number, caseID number, UserID, and case type.

As noted, some of the rows contain an "Error", in which case there is only a caseID, User, and Case type (no faxID).

Ideas please... Thanks.

[MAD_2]

Oops, forgot the attachment, here it is:

[Leif Halvarsson_2]

Hi
Let us start with the first, merge two lines into one.
awk { $1="CaseId" { line1=$0 ; getline line2 ; print line1 line2} }

I dont understand the lines with errors , the caseId after the error why is this not on a new line ?

[Leif Halvarsson_2]

Sorry
You should include the ":" in the pattern. My example should look like:

awk { $1="CaseId:" { line1=$0 ; getline line2 ; print line1 line2} }

[MAD_2]

Sorry for not replying earlier Leif, but here are the errors I get when trying your sample code:
syntax error The source line is 1.
The error context is
>>> { <<<
awk: The statement cannot be correctly parsed.
The source line is 1.
awk: There is a missing } character.
/FaxLog_strip.sh[2]: getline: not found

I also believe you misunderstand my scenario. Each two lines represent one row, i.e. the following are two rows (4 lines in the sample file) -- the beginning of each row is an odd line in the file:

<11/06 12:18:04.3786> Fax1.P5ixRCCC(377): FaxId: 2428
CaseId: 9956167 User: jc CaseType: LEAD
<11/06 14:55:30.4643> Fax1.P5ixRCCC(383): **Error** CaseId: 9956168 User: RF23
CaseType: COMPLAINT

[harry d brown jr · Accepted solution]

#!/usr/contrib/bin/perl
$newrow=1;
while () {
chop;
tr/ \t/ /s;
if ($newrow == 1) {
$saverow = $_;
$newrow = 0;
} else {
$outrow = $saverow . " " . $_;
$newrow=1;
$f_datetime = $f_control = $f_faxid = $f_caseid = $f_user = $f_casetype = "";
if ($outrow =~ /(<.*>) (.*:) (\*\*Error\*\*) (CaseId:) (.*) (User:) (.*) (CaseType:) (.*)/) {$f_datetime = $1; $f_control = $2; $f_faxid = $3; $f_caseid = $5; $f_user = $7; $f_casetype = $9;}
else {
if ($outrow =~ /(<.*>) (.*:) (FaxId:) (.*) (CaseId:) (.*) (User:) (.*) (CaseType:) (.*)$/) {$f_datetime = $1; $f_control = $2; $f_faxid = $4; $f_caseid = $6; $f_user = $8; $f_casetype = $10;}
}
if ($f_datetime =~ /<(..)\/(.*) (..):(..):(..)\.(.*)>/) {$f_day=$1; $f_month=$2;
$f_hour=$3; $f_min=$4; $f_sec=$5;} printf("%s,%s,%s,%s,%s,%s,%s,%s,%s\n",$f_day,$f_month,$f_hour,$f_min,$f_sec,$f_faxid,$f_caseid,$f_user,$f_casetype);
}
}


live free or die
harry

[harry d brown jr]

In case that's too hard to read, I appended it and added some comments/


live free or die
harry

[harry d brown jr]

I think I need some more coffee, let me try that again


live free or die
harry

[MAD_2]

Thanks Harry, where is the input (log) file specified?

[MAD_2]

Thanks for the help so far; although I have not gotten very much ahead yet I must go today (I've been here since 4:30 AM) But will keep pondering at this tomorrow morning...

[harry d brown jr]

Sorry


cat inputfile | perlscript


live free or die
harry

[harry d brown jr]

Save the script I gave you,

check the path of perl,

change permissions:

chmod +x scriptname

run program by:

cat inputfile | scriptname


live free or die
harry

[harry d brown jr]

here's the output I got with your input:

# cat input_file1 | ./input_exe3
29,05,12,15,36,1737 ,9956152,jc,COMPLAINT
30,05,14,13,07,2075 ,9956153,jc,LEAD
06,06,08,15,04,2419 ,9956165,KSHEN,COMPLAINT
11,06,11,49,14,2427 ,9956166,jc,LEAD
11,06,12,18,04,2428 ,9956167,jc,LEAD
11,06,14,55,30,**Error**,9956168,RF23,COMPLAINT
11,06,14,58,05,**Error**,9956169,RF23,COMPLAINT
11,06,14,59,23,**Error**,9956170,RF23,COMPLAINT
11,06,15,00,10,**Error**,9956171,RF23,COMPLAINT
11,06,15,01,12,**Error**,9956172,RF23,COMPLAINT
#

live free or die
harry

[Leif Halvarsson_2]

Hi
Yes I misunderstod you.

First I should remove some unwanted characters, the example begins with two tabs.
cat orgfile |tr -d "\011" >tmpfile
mv tmpfile orgfile

then supress repeating spaces
cat orgfile |tr -s " " >tmpfile
mv tmpfile orgfile

When testing with awk it is better writing a awk script file and run awk with the -f option:
Example test.awk file:

$1 ~ /^
Run this with
awk -f test.awk orgfile >tmpfile

Now you can write a new awk script which extract your information ftom the tmpfile. As the fields mot appers in the same column in evry line you must write one awk line for evry

example (I think you understand it):

$5 == "Faxid:" {print $7 $8 $9}
$6 == "Faxid:" {print $8 $9 $10}

When you get all this working you can write a shell sript which do all this with one command.

[MAD_2]

Harry, you basically produced a script I can use. There were a couple of lines where I encountered some problems in the original file I am trying to clean, but I still need to look more into it.

Although I know nothing about Perl (except that maybe I should learn it!), I will attempt to read the script you gave me and try to make modifications to the code (if I can understand) to see if the problem can be fixed. Thanks so much, the output looks much more like what I want. And Leif, I appreciate your help, I still need to look into your last solution.

[MAD_2]

Harry, can I get a little more help? If you have the time. This is what is happening:

1. I made a few modifications to your original code (you can see the modified version attached), basically the output I would like to have is: date (DD/MO), time (HH:MM), FaxID, CaseID, UserID, and ComplaintType.

2. I do not know anything about Perl, so how to separate an output value like in the code below is a guessing trip for me:
# pattern match the time and break it down
if ($f_time =~ /(..):(..):(..)\.(.*)/) {
$f_time=$1;}

How do I make it look like $f_time = $1:$2? (to get HH:MM)
And for the date, how do I get rid of the "<" (to get DD/MO), I guess I would have to use something like:
if ($f_date =~ /(..)/(.*)/)
{$f_date= ????} -- to get $1/$2... I don't know.

Attached you will see the modified code, and in the next I will provide a newer and more complete version of the log (since there are still some lines that concatenate when running)...

[MAD_2]

This is a longer version of the log... I am just starting to like Perl!

[MAD_2]

And this is how the output looks like, using the current script you gave me, for the log I just attached (too bad I could not include all three files at once)

[harry d brown jr]

Here's the new prog attached


live free or die
harry

[H.Merijn Brand (procura]

If you are starting to like perl, please, please, please, upgrade to at least perl-5.6.1

Not only do you get a much more featured perl, but you also get many more people that are able to help you.

I know I am biased, but IMHO perl4 is dead. Dead as a doornail.

Recent perl binaries can be fetched on my ITRC page https:/www.beepz.com/personal/merijn (compressed tar archive) or (just a teeny bit older from the HP porting centers as software depots). If you have any HP-UX application CD's newer than September 2001, you can install HP's perl-5.6.1 from the last CD of the set. Be aware though that this perl is not able to talk to Oracle (but starting as a perl newbee, you are not to be worried about it :)

If you already know awk or sed, it's easy to learn how to solve the same problem in perl, since it ships with 'a2p' (awk2perl) and 's2p' (sed2perl) and loads of on-line documentation.

If you are starting with perl, please don't learn perl4.

perl4 is available as /usr/contrib/bin. Perl5 will be installed on /opt/perl

Enjoy, have FUN!

[harry d brown jr]

Could I be any dumber this morning? Man I like to submit!

try this:

live free or die
harry

[harry d brown jr]

Adam,

I changed the first line to the newer perl as procura suggested, so make sure

/opt/perl/bin/perl

exists before executing.


also just change the PRINT STATEMENT near the end of the "program" to output what you want. IE:

printf("%s,%s,%s,%s,%s,%s,%s,%s,%s\n",
$f_day,$f_month,$f_hour,$f_min,$f_sec,$f_faxid,$f_caseid,$f_user,$f_casetyp
e);

to (remember ro remove $f_sec and corresponding %s)

printf("%s/%s,%s:%s,%s,%s,%s,%s\n",
$f_day,$f_month,$f_hour,$f_min,$f_faxid,$f_caseid,$f_user,$f_casetyp
e);


live free or die
harry

[MAD_2]

You two rock! Procura, I do have ActivePerl 5.6.1.627, so I guess I am OK with that, and I just bought me a Perl book online. Harry, just to bug you one more time:

1. I did figure out I did not need the additional output when I noticed a few extra commas at the end of each line, so I removed the unecessary fields (pretty good for someone who has never touched perl)

2. What I would like for the date and time woul be complete DD:MO and HH:MM without separating these fields with a comma each, something like DD:MO, HH:MM, not DD, MO, HH, MO...

Thanks,

[H.Merijn Brand (procura]

I've made some notes and changes to Harry's script to be just a little bit more modern perl compliant.

Harry, please don't see this as 'your programs are bad', but as help to have a newbee jump into the right part of the swimming pool.

If I would have created this from scratch in order to help him, I would not have helped him, but probably just scared him off with much too obfuscated perl code. Your's is straitforward and understandable (though you should never use '' again, but '' or '<>')

[H.Merijn Brand (procura]

I've made some notes and changes to Harry's script to be just a little bit more modern perl compliant.

Harry, please don't see this as 'your programs are bad', but as help to have a newbee jump into the right part of the swimming pool.

If I would have created this from scratch in order to help him, I would not have helped him, but probably just scared him off with much too obfuscated perl code. Your's is straitforward and understandable (though you should never use '' again, but '' or '<>')