Operating System - Linux
1828335 Members
3562 Online
109976 Solutions
New Discussion

how to remove the specific feilds from a string

 
SOLVED
Go to solution
Maaz
Valued Contributor

how to remove the specific feilds from a string

Hello


# cat log1
Jun 9 08:36:21 192.168.6.3 ERP_PRO 203.81.203.5: XAuth login expired and was terminated for username operator1 at 172.20.1.30/255.255.255.255. (2009-06-08 10:30:23

I want to remove the 5th 7th 10th-13th, and 17th fields

# cat log1 |

Regards
Maaz
19 REPLIES 19
James R. Ferguson
Acclaimed Contributor

Re: how to remove the specific feilds from a string

Hi Maaz:

I presume that by "remove" you mean extract and print:

# awk '{print $5,$7,$10,$11,$12,$13,$17}' log1

ERP_PRO XAuth and was terminated for 172.20.1.30/255.255.255.255.

Regards!

...JRF...
Dennis Handly
Acclaimed Contributor

Re: how to remove the specific feilds from a string

>I want to remove the 5th 7th 10th-13th, and 17th fields

You are going to have problems if your messages have different formats. It seems that "XAuth login expired and was terminated for username operator1" is really one long logical field.
Maaz
Valued Contributor

Re: how to remove the specific feilds from a string

Hi James R. Ferguson, Thanks for reply/help

>I presume that by "remove" you mean extract
>and print:

># awk '{print $5,$7,$10,$11,$12,$13,$17}' log1

no sir, I really want to remove these fields and print ;)

Hi Dennis Handly, thanks for reply/suggestion
>You are going to have problems if your
>messages have different formats.....
Yes, you are right, we have already managed this(I too have a brain :-) ), infect we are just monitoring some specific IP(es) because we have allowed those IP(es) only. And also only want to monitor just the "expired logins", so what we already doing is

# cat log1 | grep -e "Allowed.IP.Address" -e "login expired"

and now want something that removes some fields from the output
# cat log1 | grep -e "Allowed.IP.Address" -e "login expired" |

Regards
Maaz
Steven E. Protter
Exalted Contributor

Re: how to remove the specific feilds from a string

Shalom Maaz,

Doing an awk to a second string and omitting certain fields does do the job.

Example

var2=$(echo $var1 | awk '{print $1 $2 $4 $5}')

This will transfer fields 1 2 4 5 in string var1 to var2

That seems to be what you are asking.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Maaz
Valued Contributor

Re: how to remove the specific feilds from a string

Hi Thanks SEP for help/reply
I am sorry for my bad English, I am sure I failed to ask what I need ;(

I am trying to elaborate

I need to print the file named 'log1', but I want to skip/remove some fields(5th, 7th, 10-13th, and 17th) from printing(i,e I do not want those fields) and every thing else should print


e.g

# cat log1 | awk '{DoNotprint $5 $7 $10-$13 $17}'

hope this help you guys understand my question

Regards
Maaz
James R. Ferguson
Acclaimed Contributor
Solution

Re: how to remove the specific feilds from a string

Hi (again) Maaz:

> I need to print the file named 'log1', but I want to skip/remove some fields(5th, 7th, 10-13th, and 17th) from printing(i,e I do not want those fields) and every thing else should print

OK, then let's do this (easily) in Perl:

# perl -wnae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~$n};END{print "\n"}' file

Notice that the fields to skip are numbered from zero since Perl counts that way as opposed to 'awk' which counts one-relative.

Using your input yields:

192.168.6.3 203.81.203.5: login expired username operator1 at (2009-06-08 10:30:23

Regards!

...JRF...
Maaz
Valued Contributor

Re: how to remove the specific feilds from a string

Hi Thanks James R. Ferguson for the help/reply/solution ;-)

if you can also translate this 'perl' into 'bash/awk/sed/cut/grep', highly appreciate.

Regards
Maaz
Maaz
Valued Contributor

Re: how to remove the specific feilds from a string

I mean is it possible to do the same without using perl.. I mean using awk/sed/cut etc ?

James R. Ferguson
Acclaimed Contributor

Re: how to remove the specific feilds from a string

Hi (again) Maaz:

> I mean is it possible to do the same without using perl.. I mean using awk/sed/cut etc ?

OK, with 'awk' :

# awk 'BEGIN{skip="5 7 10 11 12 13 17"};{for (n=1;n<=NF;n++) {if (skip!~n) {printf "%s ",$n}}};END{print "\n"}' file

Regards!

...JRF...
Jared Middleton
Frequent Advisor

Re: how to remove the specific feilds from a string

James, on my system (RHEL 4.8), neither example (perl or awk) is printing the first 3 fields.
Maaz
Valued Contributor

Re: how to remove the specific feilds from a string

Hi Thanks James R. Ferguson for help.

One problem

# perl -wnae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~$n};END{print "\n"}' log1
192.168.6.3 203.81.203.5: login expired username operator1 at (2009-06-08 10:30:23) 192.168.6.3 199.1.76.59: login expired username mrkt_emea at 192.168.6.3 199.1.76.55: login expired username mrkt_emea at 172.20.1.29/255.255.255.255.

both the perl and awk examples are not formating the newline("\n") properly.
I mean the perl and awk examples print the lines but without newlines, i.e from start till end no new lines.

Please help

Regards
Maaz
Stephen P. Schaefer
Frequent Advisor

Re: how to remove the specific feilds from a string

With light testing, this awk script does the job; the initialization of the array holding the fields to skip is more awkward than the perl; a better awk programmer might know something more idiomatic, but I'm pretty sure this works.

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1};{for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}}};END{print "\n"}' < log1
Stephen P. Schaefer
Frequent Advisor

Re: how to remove the specific feilds from a string

Sorry; need to fix the newline issue:

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1};{for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}}i;print};END{print "\n"}' < log1
Stephen P. Schaefer
Frequent Advisor

Re: how to remove the specific feilds from a string

Sigh. Another newline fix:

awk 'BEGIN{split("5,7,10,11,12,13,17",a,/,/);for (j in a) skip[a[j]]=1} {for (n=1;n<=NF;n++) {if (!skip[n]) {printf "%s ",$n}};printf "\n"}'
Jared Middleton
Frequent Advisor

Re: how to remove the specific feilds from a string

How about:
awk '{$5=$7=$10=$11=$12=$13=$17="";gsub(/[ ]+/," ")}1' < log1
Maaz
Valued Contributor

Re: how to remove the specific feilds from a string

Dear Stephen P. Schaefer thanks for such a nice help

Dear Jared Middleton its CCC(cool, crunchy, and crispy) ;), thanks a lot.

>awk '{$5=$7=$10=$11=$12=$13=$17=""
I understand the above part of the code

if you can please explain this ' gsub(/[ ]+/," ") ' part of the code/syntax, I mean whats its doing.. highly appreciated

Regards

James R. Ferguson
Acclaimed Contributor

Re: how to remove the specific feilds from a string

Hi (again) Maaz:

My apologies for not writing and testing my own code better. I failed to rigorously match the exclusion list and hence my output eliminated the first three fields.

In the modified Perl script, the '\b' represents word "boundry" character. This serves to delineate each value in the '$skip' list and thus allows exact matching whereas before values of 0, 1, or 2 were matched to segments of 10, 11, and 12.

Thus, use:

# perl -nae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~/\b$n\b/};END{print "
\n"}' file

...

@ Stephen: very good!

@ Jared: very nicely done!

Regards!

...JRF...

James R. Ferguson
Acclaimed Contributor

Re: how to remove the specific feilds from a string

Hi:

Sorry, that last Perl should have been (without the 'END' block):

# perl -nae '$skip="4 6 9 10 11 12 16";for ($n=0;$n<@F;$n++) {printf "%s ",$F[$n] unless $skip=~/\b$n\b/};print "\n"' file

That aside, Jared's 'awk' solution could be emulated in Perl (thought not nearly as concisely) with:

# perl -ane '@F[4,6,9..12,16]=undef;for (0..$#F) {printf "%s ",$F[$_] if defined $F[$_]};print "\n"' file

Lastly, Jared's 'gsub' globally substitutes muliple spaces to one single space (left from nulling the fields you wanted to skip). In my second Perl script above, I skip undefined array elements.

Regards!

...JRF...
ghostdog74
Occasional Advisor

Re: how to remove the specific feilds from a string

don't have to use cat/grep/sed/perl....you can just use awk

awk '/Allowed.IP.Address|login expired/{ $5=$7=$10=$13=$17=""}1' file