topic Re: how to remove the specific feilds from a string in Operating System - Linux

how to remove the specific feilds from a string

Maaz — Tue, 09 Jun 2009 10:40:42 GMT

Hello

# cat log1
Jun 9 08:36:21 192.168.6.3 ERP_PRO 203.81.203.5: XAuth login expired and was terminated for username operator1 at 172.20.1.30/255.255.255.255. (2009-06-08 10:30:23

I want to remove the 5th 7th 10th-13th, and 17th fields

# cat log1 |

Regards
Maaz

Re: how to remove the specific feilds from a string

James R. Ferguson — Tue, 09 Jun 2009 10:47:37 GMT

Hi Maaz:

I presume that by "remove" you mean extract and print:

# awk '{print $5,$7,$10,$11,$12,$13,$17}' log1

ERP_PRO XAuth and was terminated for 172.20.1.30/255.255.255.255.

Regards!

...JRF...

Re: how to remove the specific feilds from a string

Dennis Handly — Tue, 09 Jun 2009 11:20:37 GMT

>I want to remove the 5th 7th 10th-13th, and 17th fields

You are going to have problems if your messages have different formats. It seems that "XAuth login expired and was terminated for username operator1" is really one long logical field.

Re: how to remove the specific feilds from a string

Maaz — Tue, 09 Jun 2009 14:51:20 GMT

Hi James R. Ferguson, Thanks for reply/help

>I presume that by "remove" you mean extract
>and print:

># awk '{print $5,$7,$10,$11,$12,$13,$17}' log1

no sir, I really want to remove these fields and print ;)

Hi Dennis Handly, thanks for reply/suggestion
>You are going to have problems if your
>messages have different formats.....
Yes, you are right, we have already managed this(I too have a brain :-) ), infect we are just monitoring some specific IP(es) because we have allowed those IP(es) only. And also only want to monitor just the "expired logins", so what we already doing is

# cat log1 | grep -e "Allowed.IP.Address" -e "login expired"

and now want something that removes some fields from the output
# cat log1 | grep -e "Allowed.IP.Address" -e "login expired" |

Regards
Maaz

Re: how to remove the specific feilds from a string

Steven E. Protter — Tue, 09 Jun 2009 16:05:40 GMT

Shalom Maaz,

Doing an awk to a second string and omitting certain fields does do the job.

Example

var2=$(echo $var1 | awk '{print $1 $2 $4 $5}')

This will transfer fields 1 2 4 5 in string var1 to var2

That seems to be what you are asking.

SEP

Re: how to remove the specific feilds from a string

Maaz — Tue, 09 Jun 2009 17:05:04 GMT

Hi Thanks SEP for help/reply
I am sorry for my bad English, I am sure I failed to ask what I need ;(

I am trying to elaborate

I need to print the file named 'log1', but I want to skip/remove some fields(5th, 7th, 10-13th, and 17th) from printing(i,e I do not want those fields) and every thing else should print

e.g

# cat log1 | awk '{DoNotprint $5 $7 $10-$13 $17}'

hope this help you guys understand my question

Regards
Maaz

Re: how to remove the specific feilds from a string

James R. Ferguson — Tue, 09 Jun 2009 17:26:48 GMT

Hi (again) Maaz:

> I need to print the file named 'log1', but I want to skip/remove some fields(5th, 7th, 10-13th, and 17th) from printing(i,e I do not want those fields) and every thing else should print

OK, then let's do this (easily) in Perl:

# perl -wnae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~$n};END{print "\n"}' file

Notice that the fields to skip are numbered from zero since Perl counts that way as opposed to 'awk' which counts one-relative.

Using your input yields:

192.168.6.3 203.81.203.5: login expired username operator1 at (2009-06-08 10:30:23

Regards!

...JRF...

Re: how to remove the specific feilds from a string

Maaz — Tue, 09 Jun 2009 17:57:07 GMT

Hi Thanks James R. Ferguson for the help/reply/solution ;-)

if you can also translate this 'perl' into 'bash/awk/sed/cut/grep', highly appreciate.

Regards
Maaz

Re: how to remove the specific feilds from a string

Maaz — Tue, 09 Jun 2009 17:59:30 GMT

I mean is it possible to do the same without using perl.. I mean using awk/sed/cut etc ?

Re: how to remove the specific feilds from a string

James R. Ferguson — Tue, 09 Jun 2009 18:59:12 GMT

Hi (again) Maaz:

> I mean is it possible to do the same without using perl.. I mean using awk/sed/cut etc ?

OK, with 'awk' :

# awk 'BEGIN{skip="5 7 10 11 12 13 17"};{for (n=1;n<=NF;n++) {if (skip!~n) {printf "%s ",$n}}};END{print "\n"}' file

Regards!

...JRF...

Re: how to remove the specific feilds from a string

Jared Middleton — Tue, 09 Jun 2009 23:06:40 GMT

James, on my system (RHEL 4.8), neither example (perl or awk) is printing the first 3 fields.

Re: how to remove the specific feilds from a string

Maaz — Wed, 10 Jun 2009 03:14:56 GMT

Hi Thanks James R. Ferguson for help.

One problem

# perl -wnae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~$n};END{print "\n"}' log1
192.168.6.3 203.81.203.5: login expired username operator1 at (2009-06-08 10:30:23) 192.168.6.3 199.1.76.59: login expired username mrkt_emea at 192.168.6.3 199.1.76.55: login expired username mrkt_emea at 172.20.1.29/255.255.255.255.

both the perl and awk examples are not formating the newline("\n") properly.
I mean the perl and awk examples print the lines but without newlines, i.e from start till end no new lines.

Please help

Regards
Maaz

Re: how to remove the specific feilds from a string

Stephen P. Schaefer — Wed, 10 Jun 2009 04:57:33 GMT

With light testing, this awk script does the job; the initialization of the array holding the fields to skip is more awkward than the perl; a better awk programmer might know something more idiomatic, but I'm pretty sure this works.

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1};{for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}}};END{print "\n"}' < log1

Re: how to remove the specific feilds from a string

Stephen P. Schaefer — Wed, 10 Jun 2009 05:03:08 GMT

Sorry; need to fix the newline issue:

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1};{for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}}i;print};END{print "\n"}' < log1

Re: how to remove the specific feilds from a string

Stephen P. Schaefer — Wed, 10 Jun 2009 05:10:07 GMT

Sigh. Another newline fix:

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1} {for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}};printf "\n"}'

Re: how to remove the specific feilds from a string

Jared Middleton — Wed, 10 Jun 2009 08:11:59 GMT

How about:
awk '{$5=$7=$10=$11=$12=$13=$17="";gsub(/[ ]+/," ")}1' < log1

Re: how to remove the specific feilds from a string

Maaz — Wed, 10 Jun 2009 08:59:14 GMT

Dear Stephen P. Schaefer thanks for such a nice help

Dear Jared Middleton its CCC(cool, crunchy, and crispy) ;), thanks a lot.

>awk '{$5=$7=$10=$11=$12=$13=$17=""
I understand the above part of the code

if you can please explain this ' gsub(/[ ]+/," ") ' part of the code/syntax, I mean whats its doing.. highly appreciated

Regards

Re: how to remove the specific feilds from a string

James R. Ferguson — Wed, 10 Jun 2009 12:31:28 GMT

Hi (again) Maaz:

My apologies for not writing and testing my own code better. I failed to rigorously match the exclusion list and hence my output eliminated the first three fields.

In the modified Perl script, the '\b' represents word "boundry" character. This serves to delineate each value in the '$skip' list and thus allows exact matching whereas before values of 0, 1, or 2 were matched to segments of 10, 11, and 12.

Thus, use:

# perl -nae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~/\b$n\b/};END{print "
\n"}' file

...

@ Stephen: very good!

@ Jared: very nicely done!

Regards!

...JRF...

Re: how to remove the specific feilds from a string

James R. Ferguson — Wed, 10 Jun 2009 14:50:06 GMT

Hi:

Sorry, that last Perl should have been (without the 'END' block):

# perl -nae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~/\b$n\b/};print "\n"' file

That aside, Jared's 'awk' solution could be emulated in Perl (thought not nearly as concisely) with:

# perl -ane '@F[4,6,9..12,16]=undef;for (0..$#F) {printf "%s ",$F[$_] if defined $F[$_]};print "\n"' file

Lastly, Jared's 'gsub' globally substitutes muliple spaces to one single space (left from nulling the fields you wanted to skip). In my second Perl script above, I skip undefined array elements.

Regards!

...JRF...

Re: how to remove the specific feilds from a string

ghostdog74 — Thu, 11 Jun 2009 05:18:02 GMT

don't have to use cat/grep/sed/perl....you can just use awk

awk '/Allowed.IP.Address|login expired/{ $5=$7=$10=$13=$17=""}1' file