counting the number of times a word appears in a file

[Belinda Dermody]

I have a extracted log file and with the word smtp or SMTP appearing either once or multiple times per line, is there a way that we can count how many times smtp appears in the log, can't just count the lines because it appears more than once on certain lines...

[Pete Randall]

grep -i smtp |wc -w

ought to do it.


Pete

[Belinda Dermody]

Sorry Pete, I tried that earlier, it grabs the lines with smtp and counts all the words in the line...

[baiju_3]

If the word smtp|SMTP is having any delimiter , if so you can write a script which search for each word compare the word and then incriment a counter if the word is matched .


Thanks,
BL.

[Pete Randall]

James,

ARGHH! You're right of course. I remember a similar question a month or so ago but I can't find it at the moment and don't remember what the answer was.


Pete

[Jeff Schussele]

HI James,

You can still use grep & wc -w
Save the grepped lines in a tmp file
Then just read through that file in a loop and increment a counter from the wc -w output.
Should work.

Rgds,
Jeff

[Belinda Dermody]

There is no special delimiter and the fields are not fixed, there could be a % ; or even a white space before the smtp or SMTP.

Jeff I do not understand your response, even if I throw all the lines with either smtp | SMTP into an output file I will still have all the other words (mail stuff that would come up in the wc command.

[curt larson_1]

a quick awk script

cat file | awk '{
for ( i=1; i<=NF; i++)
num[$i]++;
}
END {
for ( word in num )
print word, num[word];
}' | grep -i smtp

[curt larson_1]

another method

cat file | tr "[:upper:]" "[:lower:]" |
tr -cs "[a-z0-9']" "\012" | sort |
uniq -c | sort +0nr +1d

convert all uppercase to lowercase
replace all characters not a-z0-9' with a new line. that means one word per line
sort because uniq expects sorted input
uniq counts the number of times each word appears then sort first from most to least frequent then alphabetically

[Stephen Keane]

One way (though you'll have to change it to cope with uppercase SMTP to

exactly as typed, including the '\' !!


# sed -e 's/smtp/smtp\
/g' your_file | grep -i "smtp" | wc -l

maybe use tr in there to convert SMTP to smtp?

changes "smtp" into "smtp\n" so each smtp is on a separate line, then uses grep/wc to count them. Just a thought.

[curt larson_1]

here is a program that might do a beter job

http://hpux.cs.utah.edu/hppd/hpux/Misc/conc-0.5/

[Belinda Dermody]

Curt thanks for the two replies but no luck on the awk, I am not sure the correct writing format in a script.

Test file with 3 lines
this is a test of the smtp; but the SMTP would %smtp smtp SMTP& work
helo smtp

cat /tmp/test | tr "[:upper:]" "[:lower:]" | tr -cs "[a-z0-9]" "\012" |sort | uniq -c | sort +0nr +1d
+ cat /tmp/test
+ tr [:upper:] [:lower:]
+ tr -cs [a-z0-9] \012
+ sort
+ sort +0nr +1d
+ uniq -c
1 helo smtp
1 this is a test of the smtp; but the smtp
1 would %smtp smtp smtp& work

With your awk scrip i get the following error good ole bail out

Tahoe: /tmp ./t2.sh
#!/bin/ksh -xv
cat /tmp/test |awk '{
for (i=1, i<=NF;i++)
num[$i]++;
}

END{
for ( word in num )
print word, num[word];
}'|grep -i smtp
+ cat /tmp/test
+ awk {
for (i=1, i<=NF;i++)
num[$i]++;
}

END{
for ( word in num )
print word, num[word];
}
+ grep -i smtp
syntax error The source line is 2.
The error context is
for >>> (i=1, <<<
awk: The statement cannot be correctly parsed.
The source line is 2.
syntax error The source line is 2.

[James R. Ferguson · Accepted solution]

Hi James:

Try this:

# perl -lne '$count++ while (m/smtp/ig);END{print $count}' logfile

Regards!

...JRF...

[Belinda Dermody]

James, you have come through for me more times than I can count, thank you very much and thanks to the rest of you guys for and for all the help you provided.

[curt larson_1]

| tr -cs "[a-z0-9]" "\012"

should be
| tr -cs "[a-z0-9]" "\012*"
star after \012

and

The error context is
for >>> (i=1, <<<
awk: The statement cannot be correctly parsed.
i=1, < no comma, it is a semi colon ";"

[Belinda Dermody]

Curt, I got the awk statement to work, thanks, it gives me a count for each line, so I had to add a counter for that, still wouldn't give me a correct count for the tr usage. But thanks again guys...

[James R. Ferguson]

Hi (again) James:

To improve thins a bit, I'd amend my expression thusly:

# perl -lne '$count++ while (m/\bsmtp\b/ig);END{print $count}' logfile

This eliminates counting the string "smtp" (in any case or mixture) *within* the bounds of another string. That is, this ould *not* count "this_was_from_smtp".

Regards!

...JRF...

[Sandman!]

James,

Within awk set your field separator to the word smtp or SMTP and then count its occurence within the input file as:

# awk -F"smtp|SMTP" 'BEGIN{cnt=0} {cnt+=(NF-1)} END{print cnt}' input_file

cheers!

[Sandman!]

In fact to make your "find && count SMTP" case in-sensitive do the following:

# awk -F"[sS][mM][tT][pP]" 'BEGIN{cnt=0} {cnt+=(NF-1)} END{print cnt}' input_file

cheers!

[Belinda Dermody]

Thanks Sandman, your awk version also works, I had to change the cnt from cnt=0 to cnt=1 to get the accurate count.

[Sandman!]

James,

If "cnt" is declared in the BEGIN section of the awk construct then I don't know why it needs to be set equal to 1 and if it's defined in the ACTION section, then every new line read would clobber its contents and reset it back to 1. Could you share the awk construct you used to get the correct result?

Maybe I'm not able to understand your requirements.

thx