how to remove the specific feilds from a string

[Maaz]

Hello


# cat log1
Jun 9 08:36:21 192.168.6.3 ERP_PRO 203.81.203.5: XAuth login expired and was terminated for username operator1 at 172.20.1.30/255.255.255.255. (2009-06-08 10:30:23

I want to remove the 5th 7th 10th-13th, and 17th fields

# cat log1 |

Regards
Maaz

[James R. Ferguson]

Hi Maaz:

I presume that by "remove" you mean extract and print:

# awk '{print $5,$7,$10,$11,$12,$13,$17}' log1

ERP_PRO XAuth and was terminated for 172.20.1.30/255.255.255.255.

Regards!

...JRF...

[Dennis Handly]

>I want to remove the 5th 7th 10th-13th, and 17th fields

You are going to have problems if your messages have different formats. It seems that "XAuth login expired and was terminated for username operator1" is really one long logical field.

[Maaz]

Hi James R. Ferguson, Thanks for reply/help

>I presume that by "remove" you mean extract
>and print:

># awk '{print $5,$7,$10,$11,$12,$13,$17}' log1

no sir, I really want to remove these fields and print ;)

Hi Dennis Handly, thanks for reply/suggestion
>You are going to have problems if your
>messages have different formats.....
Yes, you are right, we have already managed this(I too have a brain :-) ), infect we are just monitoring some specific IP(es) because we have allowed those IP(es) only. And also only want to monitor just the "expired logins", so what we already doing is

# cat log1 | grep -e "Allowed.IP.Address" -e "login expired"

and now want something that removes some fields from the output
# cat log1 | grep -e "Allowed.IP.Address" -e "login expired" |

Regards
Maaz

[Steven E. Protter]

Shalom Maaz,

Doing an awk to a second string and omitting certain fields does do the job.

Example

var2=$(echo $var1 | awk '{print $1 $2 $4 $5}')

This will transfer fields 1 2 4 5 in string var1 to var2

That seems to be what you are asking.

SEP

[Maaz]

Hi Thanks SEP for help/reply
I am sorry for my bad English, I am sure I failed to ask what I need ;(

I am trying to elaborate

I need to print the file named 'log1', but I want to skip/remove some fields(5th, 7th, 10-13th, and 17th) from printing(i,e I do not want those fields) and every thing else should print


e.g

# cat log1 | awk '{DoNotprint $5 $7 $10-$13 $17}'

hope this help you guys understand my question

Regards
Maaz

[James R. Ferguson · Accepted solution]

Hi (again) Maaz:

> I need to print the file named 'log1', but I want to skip/remove some fields(5th, 7th, 10-13th, and 17th) from printing(i,e I do not want those fields) and every thing else should print

OK, then let's do this (easily) in Perl:

# perl -wnae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~$n};END{print "\n"}' file

Notice that the fields to skip are numbered from zero since Perl counts that way as opposed to 'awk' which counts one-relative.

Using your input yields:

192.168.6.3 203.81.203.5: login expired username operator1 at (2009-06-08 10:30:23

Regards!

...JRF...

[Maaz]

Hi Thanks James R. Ferguson for the help/reply/solution ;-)

if you can also translate this 'perl' into 'bash/awk/sed/cut/grep', highly appreciate.

Regards
Maaz

[Maaz]

I mean is it possible to do the same without using perl.. I mean using awk/sed/cut etc ?

[James R. Ferguson]

Hi (again) Maaz:

> I mean is it possible to do the same without using perl.. I mean using awk/sed/cut etc ?

OK, with 'awk' :

# awk 'BEGIN{skip="5 7 10 11 12 13 17"};{for (n=1;n<=NF;n++) {if (skip!~n) {printf "%s ",$n}}};END{print "\n"}' file

Regards!

...JRF...

[Jared Middleton]

James, on my system (RHEL 4.8), neither example (perl or awk) is printing the first 3 fields.

[Maaz]

Hi Thanks James R. Ferguson for help.

One problem

# perl -wnae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~$n};END{print "\n"}' log1
192.168.6.3 203.81.203.5: login expired username operator1 at (2009-06-08 10:30:23) 192.168.6.3 199.1.76.59: login expired username mrkt_emea at 192.168.6.3 199.1.76.55: login expired username mrkt_emea at 172.20.1.29/255.255.255.255.

both the perl and awk examples are not formating the newline("\n") properly.
I mean the perl and awk examples print the lines but without newlines, i.e from start till end no new lines.

Please help

Regards
Maaz

[Stephen P. Schaefer]

With light testing, this awk script does the job; the initialization of the array holding the fields to skip is more awkward than the perl; a better awk programmer might know something more idiomatic, but I'm pretty sure this works.

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1};{for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}}};END{print "\n"}' < log1

[Stephen P. Schaefer]

Sorry; need to fix the newline issue:

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1};{for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}}i;print};END{print "\n"}' < log1

[Stephen P. Schaefer]

Sigh. Another newline fix:

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1} {for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}};printf "\n"}'

[Jared Middleton]

How about:
awk '{$5=$7=$10=$11=$12=$13=$17="";gsub(/[ ]+/," ")}1' < log1

[Maaz]

Dear Stephen P. Schaefer thanks for such a nice help

Dear Jared Middleton its CCC(cool, crunchy, and crispy) ;), thanks a lot.

>awk '{$5=$7=$10=$11=$12=$13=$17=""
I understand the above part of the code

if you can please explain this ' gsub(/[ ]+/," ") ' part of the code/syntax, I mean whats its doing.. highly appreciated

Regards

[James R. Ferguson]

Hi (again) Maaz:

My apologies for not writing and testing my own code better. I failed to rigorously match the exclusion list and hence my output eliminated the first three fields.

In the modified Perl script, the '\b' represents word "boundry" character. This serves to delineate each value in the '$skip' list and thus allows exact matching whereas before values of 0, 1, or 2 were matched to segments of 10, 11, and 12.

Thus, use:

# perl -nae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~/\b$n\b/};END{print "
\n"}' file

...

@ Stephen: very good!

@ Jared: very nicely done!

Regards!

...JRF...

[James R. Ferguson]

Hi:

Sorry, that last Perl should have been (without the 'END' block):

# perl -nae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~/\b$n\b/};print "\n"' file

That aside, Jared's 'awk' solution could be emulated in Perl (thought not nearly as concisely) with:

# perl -ane '@F[4,6,9..12,16]=undef;for (0..$#F) {printf "%s ",$F[$_] if defined $F[$_]};print "\n"' file

Lastly, Jared's 'gsub' globally substitutes muliple spaces to one single space (left from nulling the fields you wanted to skip). In my second Perl script above, I skip undefined array elements.

Regards!

...JRF...

[ghostdog74]

don't have to use cat/grep/sed/perl....you can just use awk

awk '/Allowed.IP.Address|login expired/{ $5=$7=$10=$13=$17=""}1' file