Header and Trailer Records..Continues

[intp]

Hi all..

sed -e '1'd -e '$'d file1 > file2 works fine for me. i.e it removes header and trailer records. however .. i want to make sure
i'm removing only header and trailer records and not actual data (if there is no header/trlr record in it)... i can distingush header / trlr record by following..

header records will be something like this
HDR*#20060327

and trailer will be
TRLR*#0002401372

so i need to check for this matching patterns ( HDR*# or TRLR*#) before i remove hdr/trlr records.if pattern doesnt match then it shudnt anything..


Sample Data
HDR*#20060327
CUST001 100 200 4500
CUST200 376 365 7889
CUST400 567 589 6869
TRLR*#0002401372

PLS HELP.

[James R. Ferguson]

Hi:

This is along the same lines as I suggested in your first post:

# perl -ne 'print unless m/^HDR\*#|^TRLR\*#/' filein

Regards!

...JRF...

[A. Clay Stephenson]

There are several approaches that will work and now that you have clarified that these are textual data then here is a simple awk solution that relies upon anchored RE's:

awk '{ if (!($0 ~ /^HDR/) && !($0 ~ /^TRLR/)) print $0 }' < infile > outfile

[A. Clay Stephenson]

One of the things that you should really make clear is if matching HDR/TRLR tuples should output to different files. You might have multiple data sets with very different formats in the same file. If so, how are you planning to split those out?

[intp]

Thanks..I'll try with this awk code..

Nope .. i dont need to store the HDR and TRLR records in different file..i just need to ignore them... and my input file going to be always in same format and thats

HDR*#20060327
CUST001 100 200 4500
CUST200 376 365 7889
CUST400 567 589 6869
TRLR*#0002401372

with first record (should) starts with HDR
and last records (should) start with TRLR..

[James R. Ferguson]

Hi:

Either the Perl or awk will support your needs.

Regards!

...JRF...

[intp]

This is the content of my shell script

****************8
#!/usr/bin/awk -f

date
echo "IN"
awk '{ if (!($0 ~ /^HDR/) && !($0 ~ /^TRLR/)) print $0 }' < mycust.dat > mycust_output.dat
echo "OUT"
date
******************

I get the following error

awk: syntax error near line 1
awk: illegal statement near line 1

I'm totally new to awk (and unix).. can you please let me know how to debug?

[Sandman!]

How about substituting your awk construct with the one below in your script...

# awk '!/^HDR\*\#/&&!/^TRLR\*\#/' inp

cheers!

[Sandman!]

Hi,

The error is with line below...
>>#!/usr/bin/awk -f<<
...since the shell expects awk commands instead of shell commands etc.

Change the above line to...
#!/usr/bin/sh
...and it should work for you.

cheers!

[A. Clay Stephenson]

The problem lies not with the awk syntax buth with your "shebang" line. Where in the Wide World of Sports did this come from?
#!/usr/bin/awk -f

Remove it completely or change it to:
#!/usr/bin/sh

[intp]

Hi initially had this in my shell script
#!/usr/bin/ksh

I got the following error

awk: syntax error near line 1
awk: illegal statement near line 1

and i saw #!/usr/bin/awk in someother program and thought in order for awk to work may be i need to include this ...but didnt work ..

Now i changed again to
#!/usr/bin/ksh and tried ..getting same error

#!/usr/bin/sh and tried ... getting same error.

[James R. Ferguson]

Hi (again):

I suggest you also evaluate:

# perl -ne 'print unless m/^HDR\*#|^TRLR\*#/' filein > fileout

...OR: if you prefer to update inplace (while creating an automatic backup file with an extension of ".old" corresponding to *your* input file name), do:

# perl -ni.old -e 'print unless m/^HDR\*#|^TRLR\*#/' filein

Regards!

...JRF...

[OldSchool]

how 'bout a simple grep:

grep -v -e '^HDR' -e '^TRLR' filename > outfile
?

[OldSchool]

Ignore the previous, I missed the "now needs to verify" bit......

[intp]

Hi,


awk '{ if (!($0 ~ /^HDR/) && !($0 ~ /^TRLR/)) print $0 }' < mycust.dat > mycust_output.dat

This works ..but i'm getting a basic doubt now..fine ...but what that && does here ??
does it checks

line should not start with
[HDR AND TRLR ] or
[HDR OR TRLR]

i assume its [HDR AND TRLR ]
if thats the case how it removes
my first records when it has(starts with) only HDR in it ?

i get a feeling..i'm missing something basic.. Can some clarify.

[James R. Ferguson]

Hi:

Yes, the "&&" is a logical AND and "!" is a negation, so the code reads "...if NOT a header AND NOT a trailer, then print".

The Perl code I suggested works analogously, saying "...print unless a header OR a trailer". Of course, Perl's syntax differs from 'awk'.

Regards!

...JRF...

Regards!

...JRF...