A really fun programming project. Operation spamstopper!

[Steven E. Protter]

I have a rather extensive database of spammers in my /etc/mail/access database.

I've compiled it all by hand.

I got the bright idea to create am account on my server called spam.

I forward all unsolicited emails to that account.

I'm attaching a nice sample file.

I have processed it as follows and need help to accomplish my goals.

cat filename | grep -i recieved

gets this data.

Received: (from invest@localhost)
Received: from met.police.uk (modemcable090.154-70-69.mc.videotron.ca [69.70.154.90])
Received: from 102.183.181.197 by smtp.jur.kun.nl;
Received: (from invest@localhost)
Received: from wozenilek.de ([211.114.193.243])
Received: from 110.186.71.219 by smtp.cc.shibaura-it.ac.jp;
Received: (from invest@localhost)
Received: from dublinia.ie (OL186-15.fibertel.com.ar [24.232.15.186])
Received: from 34.83.110.48 by smtp.optifit.de;
Received: (from invest@localhost)
Received: from augddzpd.regenerousity.com ([65.208.147.35])
Received: from investmenttool.com (66.92.143.194)
Received: from yjuhr.regenerousity.com (HELO yjuhr) (169.254.98.41)

What I actually need is those numeric IP addresses.

My current format has me input into the access file this data:

169.254.98.41
169.254.98

This blocks the originating IP and its 255 class C address block. Its like using a nuclear weapon but thus far its been 95% effective.

Intended output is thus

169.254.98.41 We charge $500 storage fee per unsolicited message.

Want a rabbit? Take this file and give me a shell script to give me my output. I'll insert the checks so none of yahoos mail servers and my mailservers end up getting blocked.

I know. I'm lazy, but I'm under the weather and tired of cut and paste.

I will test scripts prior to awarding points. All efforts get something.

If perl is better, go for it, I need those numeric IP addresses. I don't care about the domain name, most of those are forged anyway.

Regards,

SEP

[Elmar P. Kolkman]

It would become something like this:
cat filename | grep -i 'received' | perl -n -e 'print /.*[^0-9]([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)[^0-9].*/ ; print "\t\tWe charge $500 storage fee per unsolicited message.\n"' | grep '^[0-9]'

The latter will filter out errors due to lines not containing a IP addres...

Only lines it will fail are ones ending with an IP address, but since there are none in your example...

[Mark Grant]

I think this does what you want.

perl -n -e 'if(/.*\D(\d+\.\d+\.\d+\.\d+).*/){print "$1\t\tWe charge $500 storage fee per unsolicited message\n"}' datafile

[Mark Grant]

Duh!

Please change the "$500" to "\$500" otherwise you'll be storing for free!

[Massimo Bianchi]

Hi SEP,
i wanted to do it using shell-scripting. Not so nice and compact like perl, but an interesting exercise!

sed 's/[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*/X&X/g' BEASE_TEXT_FILE.txt | tr "X" "\012" | egrep [0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*


the above simple :) set of command extract the IP ADDRESS, the remainig is trivial....

MEGACOMMAND | while read IPADDR
do
echo "$IPADDR We charge \$500 storage fee per unsolicited message. "
done > YOUR_SUBSEQUENT_PROCESSING_CMD_FILE


Massimo

[Mark Grant]

Ok,

The shell route looks fun so

cat filename | grep -i received | while read a; do IP=`expr "$a" : ".*[^0-9]\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\)"`; [ $IP ] && /usr/bin/echo "$IP\t\tWe charge \$500 storage fee per unsolicited message."; done

Thi sdoe sthe whole thing in a one line shell script. The /usr/bin/echo is important because the shell builtin one doesn't interpret the \t as a tab.

[Massimo Bianchi]

Mark, I say just one thing: Beaten :)

Your expr is very nice, i was looking for something similar but i didn't know, so i wordarounded...

Thanks for the lesson :)

Massimo

[Mark Grant]

Massimo :)

Actually, I don't think it's fool-proof but I think that given that SEP is grepping "received" lines from an e-mail we'll probably get away with it.

I thought about trying to get his exceptions list in there as well. I can feel it's a simple thing to do if we have a file of IP's that we don't charge for storage :) but I just can't see it !

[Geoff Wild]

Not an answer - but just curious how you are trapping "unsolicited" email to spam@yourdomian.com

You have to be careful with automation....

What if someone has a case of fat finger syndrome - and mis-types your email address - I assume that if the user does not exist - then it goes to your spam account - and then you automatically block them!

I too block a lot of spam and have some pretty cool rules in place...

Rgds...Geoff

[Kent Ostby]

Caveat 1:

When you cut and paste the script, keep in mind that its a 2 line script so you may have to join lines after the cut and paste.

Caveats 2 & 3:
In creating your original file you want to make sure you're only grabing the originator and not the poor guy grabbed on a hop.

Also, you may encourage them to send you more spam if you make them mad, though I salute your valor.

Here is the script:

grep [0123456789] $1 | sed -e"s/\[//g" -e"s/\]//g" -e "s/(//g" -e"s/)//g" > .um2
awk ' { for (idx1=1;idx1<=NF;idx1++) {if (match($idx1,"[0-9][0-9]*\.[0-9][0-9]*\
.[0-9][0-9]*\.[0-9][0-9]*")>0) {printf ("%s\t\t We charge $500 storage fee per u
nsolicited message\n",$idx1)};} }' <.um2

[Geoff Wild]

Here's one that has stumped me - I get about 5000 of these as day:

Dec 17 07:13:46 myserver sendmail[2555]: hBHFDj9J002555: ... User unknown
Dec 17 07:13:46 myserver sendmail[2555]: hBHFDj9J002555: from=<>, size=3587, class=0, nrcpts=0, proto=ESMTP, daemon=MTA, relay=[213.86.143.191]

Sort of brute force guessing for users....

According to the RFC - I am not allowed to block from=<>

Anyone else have this issue?

I've thought of adding ip's to my iptables - and blocking there - cause with sendmail I am just wasting cpu...

SEP - sorry if I'm out of line posting in your thread - but I thought this was a good topic...

Rgds...Geoff

[Steven E. Protter]

I have no problem with any of the posts, even the last.

I will test these today if possible. I don't have the file at work, but can obtain it via ftp.

I think there are very good applications for this utility once its done. That perl answer looks wonderful.

I think there are going to be a lot of rabbits hopping around this script.

Once I get something working, I'll award 10 points. Anything that tests out correctly but isn't used will get 9. If it fails on testing ,I'll be creative.

I may need additional help to have the script check a list of ips that can not be added to the bounce list(my dad's ip, my own ip block for example.).

I shall get back to you.

SEP

[Jordan Bean]

Attached is a perl script that I've been using for a couple years to pull IPs directly from a unix mailbox containing only offensive material. I hope it is self-explanatory.

perl direct.pl < mailbox > iplist

[Steven E. Protter]

Jordon,

I love it.

I may need to name a child after you.

SEP

[harry d brown jr]

I use this to validate IP's and subnet masks:

###############################################################################
# validate IP address
###############################################################################
sub ValidateIPaddr {
############################################################################
# save the "passed" String IP address
############################################################################
my ($IPaddrSTR) = @_;
############################################################################
# validate the IP address - notes: ZERO (0) leading IP addresses are octal
# and we are NOT going to handle that - though it
# can easily be done - I consider that to be an
# error because system routines do not return IP's
# in octal string format!!! ie: 192.168.026.0 is
# NOT 192.168.26.0, it is 192.168.22.0 (decimal)
############################################################################
if( $IPaddrSTR !~ m/^ ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
\. ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
\. ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
\. ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
$
/xo) {
return(1); # wrong anwser
} else {
return(0);
}
}

live free or die
harry

[Steven E. Protter]

I'm going to have to buy some more rabbits.

SEP

[Karthik S S]

Hi SEP,

I know that this is a STUPID script. But unfortunately it works :-(

for i in `cat ssk`
do
echo $i
done | grep [0-9].[0-9].[0-9].[0-9]|grep -v - | sed s/\(//g | sed s/\)//g | sed s/'\['//g | sed s/'\]'//g | sed s/\$/"`echo "\t\t"`We charge \$500 storage fee per unsolicited message."/g


And the output comes like this,
69.70.154.90 We charge $500 storage fee per unsolicited message.
102.183.181.197 We charge $500 storage fee per unsolicited message.
211.114.193.243 We charge $500 storage fee per unsolicited message.
110.186.71.219 We charge $500 storage fee per unsolicited message.
24.232.15.186 We charge $500 storage fee per unsolicited message.
34.83.110.48 We charge $500 storage fee per unsolicited message.
65.208.147.35 We charge $500 storage fee per unsolicited message.
66.92.143.194 We charge $500 storage fee per unsolicited message.
169.254.98.41 We charge $500 storage fee per unsolicited message.

-Karthik S S

[Karthik S S]

Of course it has 2 tabs between the IP Addr and the message :-)

69.70.154.90 We charge $500 storage fee per unsolicited message.
102.183.181.197 We charge $500 storage fee per unsolicited message.

-Karthik S S

[Steven E. Protter]

I can do it with shell.

I can do it with perl.

I can do it whiled standing on my head or sitting down.

This is truly a wonderful forum and I'm going to be a better shell programmer and perl programmer.

I'm going to combine the concepts and create a perl version I can run via the web and a shell version.

I'm going to add a check for a list of IP adresses to not ban, and build in the part that writes a second line knocking out the entire class C block the spam came from.

I know its a nuclear weapon, but it is powerful.

I'm under the weather and have maintenance until late this evening. I'll try and turn this around and hand out those rabbits before close of business Thursday.

SEP

[Steven E. Protter]

Oh:

Karthik S S

I missed your congrats thread.

Congrats.

Nice hat.

Enjoy Hogwarts.

SEP

[Karthik S S]

Hi SEP,

Thanks a lot ... I feel great that I am getting noticed by one of the forum greats :-)

-Karthik S S

No points please ...

[Michael Schulte zur Sur · Accepted solution]

Hi SEP,

here my contribution. It cost me quite a lot of sweat.

Michael

[Steven E. Protter]

Gonna be fun handing out rabbits. I'm working on this right now.

Don't tell anyone, I'm testing on Linux. Its a language project.

Geoff Wild: Share your nice spam rules. Get an additional bunny rabbit. Will be awarding soon.

Patience.

SEP

[Steven E. Protter]

I have taken the last shell script and made changes. I'm still a little stuck.

I messed $IPB line for a while.
What I want it to do is this:

If the IP line outputs 192.168.0.15
I want the IPB line to ouput 192.168.0
B stands for block.

Believe it or not I can't figure it out.

In Linux shell scripting the tabs come out as \t. This is a Linux issue and I'll go to that forum for help there.

The \t comes out as a proper tab with the Korn shell and HP-UX


Linux version. HP-UX working version uses /usr/bin/ksh

#!/bin/sh

INPUTFILE=/var/mail/spam
TFILE1=/tmp/spam1.$$.dat
TFILE2=/tmp/spam2.$$.dat

cat $INPUTFILE | grep -i received > $TFILE1

while read LINE
do
IP=`expr "${LINE}" : ".*[^0-9]\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\).*"`
IPB=`expr "${LINE}" : ".*[^0-9]\([0-9]*\.[0-9]*\.[0-9]*\.[0-9]*\).*"`
if [ ${?} -eq 0 ]
then
echo ${IP} "\t\tWe charge \$900 storage fee per unsolicited message." >> $TFILE2
# echo ${IPB} "\t\tWe charge \$900 storage fee per unsolicited message." >> $TFILE2
fi
done < $TFILE1


I'm handing out points to those whose scripts I've tested. If you don't get a rabbit testing did not produce proper output.

Karthik SS's sed version produced a little more than IP addresses.

This is the output

Received:
from
met.police.uk
(modemcable090.154-70-69.mc.videotron.ca
[69.70.154.90])
Received:
from

It was hard work. This is so much fun I'm going to keep trying until I've tested everyone's work.

That will take time so be patient. I may be acting anal, but I think its important to test the scripts and give rabbbits to those that produce near perfect output.

I'm having fun. How about you?

SEP

[Steven E. Protter]

To answer another of Goeff's questions.

The spam mailbox is created by manual forwarding. I'm doing it now and I manually review messages sent by my various customers.

There is no totally automated method. Everyone gets one shot at my customers. The real problem is someone pulled my personnel address, maybe from forums and sold it widely. I think the two mouse clicks it takes to forward it to spam@mydomain.com is worth the effort.

I've been getting lazy about doing via cut and paste. After the manual operation sometimes I don't get spam at all for 7 days. Thats why nailing the entire Class C address block is important.

I'm working bottom to top(don't know why) Harry's perl code is worth a bunny and I may integrate it into one of the other perl scripts.

SEP