topic counting the number of times a word appears in a file in Operating System - Linux

counting the number of times a word appears in a file

Belinda Dermody — Mon, 26 Sep 2005 10:51:13 GMT

I have a extracted log file and with the word smtp or SMTP appearing either once or multiple times per line, is there a way that we can count how many times smtp appears in the log, can't just count the lines because it appears more than once on certain lines...

Re: counting the number of times a word appears in a file

Pete Randall — Mon, 26 Sep 2005 10:54:05 GMT

grep -i smtp |wc -w

ought to do it.

Pete

Re: counting the number of times a word appears in a file

Belinda Dermody — Mon, 26 Sep 2005 10:57:30 GMT

Sorry Pete, I tried that earlier, it grabs the lines with smtp and counts all the words in the line...

Re: counting the number of times a word appears in a file

baiju_3 — Mon, 26 Sep 2005 11:04:39 GMT

If the word smtp|SMTP is having any delimiter , if so you can write a script which search for each word compare the word and then incriment a counter if the word is matched .

Thanks,
BL.

Re: counting the number of times a word appears in a file

Pete Randall — Mon, 26 Sep 2005 11:05:56 GMT

James,

ARGHH! You're right of course. I remember a similar question a month or so ago but I can't find it at the moment and don't remember what the answer was.

Pete

Re: counting the number of times a word appears in a file

Jeff Schussele — Mon, 26 Sep 2005 11:11:11 GMT

HI James,

You can still use grep & wc -w
Save the grepped lines in a tmp file
Then just read through that file in a loop and increment a counter from the wc -w output.
Should work.

Rgds,
Jeff

Re: counting the number of times a word appears in a file

Belinda Dermody — Mon, 26 Sep 2005 11:19:17 GMT

There is no special delimiter and the fields are not fixed, there could be a % ; or even a white space before the smtp or SMTP.

Jeff I do not understand your response, even if I throw all the lines with either smtp | SMTP into an output file I will still have all the other words (mail stuff that would come up in the wc command.

Re: counting the number of times a word appears in a file

curt larson_1 — Mon, 26 Sep 2005 11:19:18 GMT

a quick awk script

cat file | awk '{
for ( i=1; i<=NF; i++)
num[$i]++;
}
END {
for ( word in num )
print word, num[word];
}' | grep -i smtp

Re: counting the number of times a word appears in a file

curt larson_1 — Mon, 26 Sep 2005 11:24:59 GMT

another method

cat file | tr "[:upper:]" "[:lower:]" |
tr -cs "[a-z0-9']" "\012" | sort |
uniq -c | sort +0nr +1d

convert all uppercase to lowercase
replace all characters not a-z0-9' with a new line. that means one word per line
sort because uniq expects sorted input
uniq counts the number of times each word appears then sort first from most to least frequent then alphabetically

Re: counting the number of times a word appears in a file

Stephen Keane — Mon, 26 Sep 2005 11:27:29 GMT

One way (though you'll have to change it to cope with uppercase SMTP to

exactly as typed, including the '\' !!

# sed -e 's/smtp/smtp\
/g' your_file | grep -i "smtp" | wc -l

maybe use tr in there to convert SMTP to smtp?

changes "smtp" into "smtp\n" so each smtp is on a separate line, then uses grep/wc to count them. Just a thought.

Re: counting the number of times a word appears in a file

curt larson_1 — Mon, 26 Sep 2005 11:42:35 GMT

here is a program that might do a beter job

http://hpux.cs.utah.edu/hppd/hpux/Misc/conc-0.5/

Re: counting the number of times a word appears in a file

Belinda Dermody — Mon, 26 Sep 2005 11:43:59 GMT

Curt thanks for the two replies but no luck on the awk, I am not sure the correct writing format in a script.

Test file with 3 lines
this is a test of the smtp; but the SMTP would %smtp smtp SMTP& work
helo smtp

cat /tmp/test | tr "[:upper:]" "[:lower:]" | tr -cs "[a-z0-9]" "\012" |sort | uniq -c | sort +0nr +1d
+ cat /tmp/test
+ tr [:upper:] [:lower:]
+ tr -cs [a-z0-9] \012
+ sort
+ sort +0nr +1d
+ uniq -c
1 helo smtp
1 this is a test of the smtp; but the smtp
1 would %smtp smtp smtp& work

With your awk scrip i get the following error good ole bail out

Tahoe: /tmp ./t2.sh
#!/bin/ksh -xv
cat /tmp/test |awk '{
for (i=1, i<=NF;i++)
num[$i]++;
}

END{
for ( word in num )
print word, num[word];
}'|grep -i smtp
+ cat /tmp/test
+ awk {
for (i=1, i<=NF;i++)
num[$i]++;
}

END{
for ( word in num )
print word, num[word];
}
+ grep -i smtp
syntax error The source line is 2.
The error context is
for >>> (i=1, <<<
awk: The statement cannot be correctly parsed.
The source line is 2.
syntax error The source line is 2.

Re: counting the number of times a word appears in a file

James R. Ferguson — Mon, 26 Sep 2005 11:46:26 GMT

Hi James:

Try this:

# perl -lne '$count++ while (m/smtp/ig);END{print $count}' logfile

Regards!

...JRF...

Re: counting the number of times a word appears in a file

Belinda Dermody — Mon, 26 Sep 2005 11:53:56 GMT

James, you have come through for me more times than I can count, thank you very much and thanks to the rest of you guys for and for all the help you provided.

Re: counting the number of times a word appears in a file

curt larson_1 — Mon, 26 Sep 2005 11:54:08 GMT

| tr -cs "[a-z0-9]" "\012"

should be
| tr -cs "[a-z0-9]" "\012*"
star after \012

and

The error context is
for >>> (i=1, <<<
awk: The statement cannot be correctly parsed.
i=1, < no comma, it is a semi colon ";"

Re: counting the number of times a word appears in a file

Belinda Dermody — Mon, 26 Sep 2005 11:58:02 GMT

Curt, I got the awk statement to work, thanks, it gives me a count for each line, so I had to add a counter for that, still wouldn't give me a correct count for the tr usage. But thanks again guys...

Re: counting the number of times a word appears in a file

James R. Ferguson — Mon, 26 Sep 2005 12:11:20 GMT

Hi (again) James:

To improve thins a bit, I'd amend my expression thusly:

# perl -lne '$count++ while (m/\bsmtp\b/ig);END{print $count}' logfile

This eliminates counting the string "smtp" (in any case or mixture) *within* the bounds of another string. That is, this ould *not* count "this_was_from_smtp".

Regards!

...JRF...

Re: counting the number of times a word appears in a file

Sandman! — Tue, 27 Sep 2005 02:43:13 GMT

James,

Within awk set your field separator to the word smtp or SMTP and then count its occurence within the input file as:

# awk -F"smtp|SMTP" 'BEGIN{cnt=0} {cnt+=(NF-1)} END{print cnt}' input_file

cheers!

Re: counting the number of times a word appears in a file

Sandman! — Tue, 27 Sep 2005 05:21:20 GMT

In fact to make your "find && count SMTP" case in-sensitive do the following:

# awk -F"[sS][mM][tT][pP]" 'BEGIN{cnt=0} {cnt+=(NF-1)} END{print cnt}' input_file

cheers!

Re: counting the number of times a word appears in a file

Belinda Dermody — Tue, 27 Sep 2005 08:54:54 GMT

Thanks Sandman, your awk version also works, I had to change the cnt from cnt=0 to cnt=1 to get the accurate count.