topic Re: Help with a script! in Operating System - HP-UX

Help with a script!

MAD_2 — Sun, 25 Aug 2002 15:07:56 GMT

I would like to know if some of you can provide some ideas on how to start with a script here:

Situation example (see attached file) -- Description:
1. Each two lines include date, time, a queue name, a fax ID number, a caseID number, the user ID, and the case type.
2. The file is a log and each data row is actually split into two lines each (there is an end of line character separating each row)
3. Most data rows contain the information as described in 1, however some (Errors) exclude parts of this information.

Objective:

1. To strip the file of the end of line characters separating each row (odd lines), placing each row of data into one line each.
2. To extract some of the data and put it into a raw file separated by any type of delimiter (pipes or commas is OK). Data required is date, time (only hour+minutes+seconds), the faxID number, caseID number, UserID, and case type.

As noted, some of the rows contain an "Error", in which case there is only a caseID, User, and Case type (no faxID).

Ideas please... Thanks.

Re: Help with a script!

MAD_2 — Sun, 25 Aug 2002 15:08:33 GMT

Oops, forgot the attachment, here it is:

Re: Help with a script!

Leif Halvarsson_2 — Sun, 25 Aug 2002 16:06:27 GMT

Hi
Let us start with the first, merge two lines into one.
awk { $1="CaseId" { line1=$0 ; getline line2 ; print line1 line2} }

I dont understand the lines with errors , the caseId after the error why is this not on a new line ?

Re: Help with a script!

Leif Halvarsson_2 — Sun, 25 Aug 2002 17:24:27 GMT

Sorry
You should include the ":" in the pattern. My example should look like:

awk { $1="CaseId:" { line1=$0 ; getline line2 ; print line1 line2} }

Re: Help with a script!

MAD_2 — Sun, 25 Aug 2002 19:40:14 GMT

Sorry for not replying earlier Leif, but here are the errors I get when trying your sample code:
syntax error The source line is 1.
The error context is
>>> { <<<
awk: The statement cannot be correctly parsed.
The source line is 1.
awk: There is a missing } character.
/FaxLog_strip.sh[2]: getline: not found

I also believe you misunderstand my scenario. Each two lines represent one row, i.e. the following are two rows (4 lines in the sample file) -- the beginning of each row is an odd line in the file:

<11/06 12:18:04.3786> Fax1.P5ixRCCC(377): FaxId: 2428
CaseId: 9956167 User: jc CaseType: LEAD
<11/06 14:55:30.4643> Fax1.P5ixRCCC(383): **Error** CaseId: 9956168 User: RF23
CaseType: COMPLAINT

Re: Help with a script!

harry d brown jr — Sun, 25 Aug 2002 19:54:54 GMT

#!/usr/contrib/bin/perl
$newrow=1;
while () {
chop;
tr/ \t/ /s;
if ($newrow == 1) {
$saverow = $_;
$newrow = 0;
} else {
$outrow = $saverow . " " . $_;
$newrow=1;
$f_datetime = $f_control = $f_faxid = $f_caseid = $f_user = $f_casetype = "";
if ($outrow =~ /(<.*>) (.*:) (\*\*Error\*\*) (CaseId:) (.*) (User:) (.*) (CaseType:) (.*)/) {$f_datetime = $1; $f_control = $2; $f_faxid = $3; $f_caseid = $5; $f_user = $7; $f_casetype = $9;}
else {
if ($outrow =~ /(<.*>) (.*:) (FaxId:) (.*) (CaseId:) (.*) (User:) (.*) (CaseType:) (.*)$/) {$f_datetime = $1; $f_control = $2; $f_faxid = $4; $f_caseid = $6; $f_user = $8; $f_casetype = $10;}
}
if ($f_datetime =~ /<(..)\/(.*) (..):(..):(..)\.(.*)>/) {$f_day=$1; $f_month=$2;
$f_hour=$3; $f_min=$4; $f_sec=$5;} printf("%s,%s,%s,%s,%s,%s,%s,%s,%s\n",$f_day,$f_month,$f_hour,$f_min,$f_sec,$f_faxid,$f_caseid,$f_user,$f_casetype);
}
}

live free or die
harry

Re: Help with a script!

harry d brown jr — Sun, 25 Aug 2002 20:06:48 GMT

In case that's too hard to read, I appended it and added some comments/

live free or die
harry

Re: Help with a script!

harry d brown jr — Sun, 25 Aug 2002 20:08:22 GMT

I think I need some more coffee, let me try that again

live free or die
harry

Re: Help with a script!

MAD_2 — Sun, 25 Aug 2002 20:10:11 GMT

Thanks Harry, where is the input (log) file specified?

Re: Help with a script!

MAD_2 — Sun, 25 Aug 2002 20:15:29 GMT

Thanks for the help so far; although I have not gotten very much ahead yet I must go today (I've been here since 4:30 AM) But will keep pondering at this tomorrow morning...

Re: Help with a script!

harry d brown jr — Sun, 25 Aug 2002 20:16:49 GMT

Sorry

cat inputfile | perlscript

live free or die
harry

Re: Help with a script!

harry d brown jr — Sun, 25 Aug 2002 20:18:37 GMT

Save the script I gave you,

check the path of perl,

change permissions:

chmod +x scriptname

run program by:

cat inputfile | scriptname

live free or die
harry

Re: Help with a script!

harry d brown jr — Sun, 25 Aug 2002 20:22:08 GMT

here's the output I got with your input:

# cat input_file1 | ./input_exe3
29,05,12,15,36,1737 ,9956152,jc,COMPLAINT
30,05,14,13,07,2075 ,9956153,jc,LEAD
06,06,08,15,04,2419 ,9956165,KSHEN,COMPLAINT
11,06,11,49,14,2427 ,9956166,jc,LEAD
11,06,12,18,04,2428 ,9956167,jc,LEAD
11,06,14,55,30,**Error**,9956168,RF23,COMPLAINT
11,06,14,58,05,**Error**,9956169,RF23,COMPLAINT
11,06,14,59,23,**Error**,9956170,RF23,COMPLAINT
11,06,15,00,10,**Error**,9956171,RF23,COMPLAINT
11,06,15,01,12,**Error**,9956172,RF23,COMPLAINT
#

live free or die
harry

Re: Help with a script!

Leif Halvarsson_2 — Mon, 26 Aug 2002 06:54:26 GMT

Hi
Yes I misunderstod you.

First I should remove some unwanted characters, the example begins with two tabs.
cat orgfile |tr -d "\011" >tmpfile
mv tmpfile orgfile

then supress repeating spaces
cat orgfile |tr -s " " >tmpfile
mv tmpfile orgfile

When testing with awk it is better writing a awk script file and run awk with the -f option:
Example test.awk file:

$1 ~ /^
Run this with
awk -f test.awk orgfile >tmpfile

Now you can write a new awk script which extract your information ftom the tmpfile. As the fields mot appers in the same column in evry line you must write one awk line for evry

example (I think you understand it):

$5 == "Faxid:" {print $7 $8 $9}
$6 == "Faxid:" {print $8 $9 $10}

When you get all this working you can write a shell sript which do all this with one command.

Re: Help with a script!

MAD_2 — Mon, 26 Aug 2002 09:27:08 GMT

Harry, you basically produced a script I can use. There were a couple of lines where I encountered some problems in the original file I am trying to clean, but I still need to look more into it.

Although I know nothing about Perl (except that maybe I should learn it!), I will attempt to read the script you gave me and try to make modifications to the code (if I can understand) to see if the problem can be fixed. Thanks so much, the output looks much more like what I want. And Leif, I appreciate your help, I still need to look into your last solution.

Re: Help with a script!

MAD_2 — Mon, 26 Aug 2002 11:06:59 GMT

Harry, can I get a little more help? If you have the time. This is what is happening:

1. I made a few modifications to your original code (you can see the modified version attached), basically the output I would like to have is: date (DD/MO), time (HH:MM), FaxID, CaseID, UserID, and ComplaintType.

2. I do not know anything about Perl, so how to separate an output value like in the code below is a guessing trip for me:
# pattern match the time and break it down
if ($f_time =~ /(..):(..):(..)\.(.*)/) {
$f_time=$1;}

How do I make it look like $f_time = $1:$2? (to get HH:MM)
And for the date, how do I get rid of the "<" (to get DD/MO), I guess I would have to use something like:
if ($f_date =~ /(..)/(.*)/)
{$f_date= ????} -- to get $1/$2... I don't know.

Attached you will see the modified code, and in the next I will provide a newer and more complete version of the log (since there are still some lines that concatenate when running)...

Re: Help with a script!

MAD_2 — Mon, 26 Aug 2002 11:08:08 GMT

This is a longer version of the log... I am just starting to like Perl!

Re: Help with a script!

MAD_2 — Mon, 26 Aug 2002 11:09:31 GMT

And this is how the output looks like, using the current script you gave me, for the log I just attached (too bad I could not include all three files at once)

Re: Help with a script!

harry d brown jr — Mon, 26 Aug 2002 11:43:20 GMT

Here's the new prog attached

live free or die
harry

Re: Help with a script!

H.Merijn Brand (procura — Mon, 26 Aug 2002 11:45:08 GMT

If you are starting to like perl, please, please, please, upgrade to at least perl-5.6.1

Not only do you get a much more featured perl, but you also get many more people that are able to help you.

I know I am biased, but IMHO perl4 is dead. Dead as a doornail.

Recent perl binaries can be fetched on my ITRC page https:/www.beepz.com/personal/merijn (compressed tar archive) or (just a teeny bit older from the HP porting centers as software depots). If you have any HP-UX application CD's newer than September 2001, you can install HP's perl-5.6.1 from the last CD of the set. Be aware though that this perl is not able to talk to Oracle (but starting as a perl newbee, you are not to be worried about it :)

If you already know awk or sed, it's easy to learn how to solve the same problem in perl, since it ships with 'a2p' (awk2perl) and 's2p' (sed2perl) and loads of on-line documentation.

If you are starting with perl, please don't learn perl4.

perl4 is available as /usr/contrib/bin. Perl5 will be installed on /opt/perl

Enjoy, have FUN!