topic Re: A really fun programming project. Operation spamstopper! in Operating System - HP-UX

A really fun programming project. Operation spamstopper!

Steven E. Protter — Tue, 16 Dec 2003 21:27:08 GMT

I have a rather extensive database of spammers in my /etc/mail/access database.

I've compiled it all by hand.

I got the bright idea to create am account on my server called spam.

I forward all unsolicited emails to that account.

I'm attaching a nice sample file.

I have processed it as follows and need help to accomplish my goals.

cat filename | grep -i recieved

gets this data.

Received: (from invest@localhost)
Received: from met.police.uk (modemcable090.154-70-69.mc.videotron.ca [69.70.154.90])
Received: from 102.183.181.197 by smtp.jur.kun.nl;
Received: (from invest@localhost)
Received: from wozenilek.de ([211.114.193.243])
Received: from 110.186.71.219 by smtp.cc.shibaura-it.ac.jp;
Received: (from invest@localhost)
Received: from dublinia.ie (OL186-15.fibertel.com.ar [24.232.15.186])
Received: from 34.83.110.48 by smtp.optifit.de;
Received: (from invest@localhost)
Received: from augddzpd.regenerousity.com ([65.208.147.35])
Received: from investmenttool.com (66.92.143.194)
Received: from yjuhr.regenerousity.com (HELO yjuhr) (169.254.98.41)

What I actually need is those numeric IP addresses.

My current format has me input into the access file this data:

169.254.98.41
169.254.98

This blocks the originating IP and its 255 class C address block. Its like using a nuclear weapon but thus far its been 95% effective.

Intended output is thus

169.254.98.41 We charge $500 storage fee per unsolicited message.

Want a rabbit? Take this file and give me a shell script to give me my output. I'll insert the checks so none of yahoos mail servers and my mailservers end up getting blocked.

I know. I'm lazy, but I'm under the weather and tired of cut and paste.

I will test scripts prior to awarding points. All efforts get something.

If perl is better, go for it, I need those numeric IP addresses. I don't care about the domain name, most of those are forged anyway.

Regards,

SEP

Re: A really fun programming project. Operation spamstopper!

Elmar P. Kolkman — Wed, 17 Dec 2003 03:07:57 GMT

It would become something like this:
cat filename | grep -i 'received' | perl -n -e 'print /.*[^0-9]([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)[^0-9].*/ ; print "\t\tWe charge $500 storage fee per unsolicited message.\n"' | grep '^[0-9]'

The latter will filter out errors due to lines not containing a IP addres...

Only lines it will fail are ones ending with an IP address, but since there are none in your example...

Re: A really fun programming project. Operation spamstopper!

Mark Grant — Wed, 17 Dec 2003 03:25:20 GMT

I think this does what you want.

perl -n -e 'if(/.*\D(\d+\.\d+\.\d+\.\d+).*/){print "$1\t\tWe charge $500 storage fee per unsolicited message\n"}' datafile

Re: A really fun programming project. Operation spamstopper!

Mark Grant — Wed, 17 Dec 2003 03:30:11 GMT

Duh!

Please change the "$500" to "\$500" otherwise you'll be storing for free!

Re: A really fun programming project. Operation spamstopper!

Massimo Bianchi — Wed, 17 Dec 2003 08:06:36 GMT

Hi SEP,
i wanted to do it using shell-scripting. Not so nice and compact like perl, but an interesting exercise!

sed 's/[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*/X&X/g' BEASE_TEXT_FILE.txt | tr "X" "\012" | egrep [0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*

the above simple :) set of command extract the IP ADDRESS, the remainig is trivial....

MEGACOMMAND | while read IPADDR
do
echo "$IPADDR We charge \$500 storage fee per unsolicited message. "
done > YOUR_SUBSEQUENT_PROCESSING_CMD_FILE

Massimo

Re: A really fun programming project. Operation spamstopper!

Mark Grant — Wed, 17 Dec 2003 08:51:12 GMT

Ok,

The shell route looks fun so

cat filename | grep -i received | while read a; do IP=`expr "$a" : ".*[^0-9]$[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*$"`; [ $IP ] && /usr/bin/echo "$IP\t\tWe charge \$500 storage fee per unsolicited message."; done

Thi sdoe sthe whole thing in a one line shell script. The /usr/bin/echo is important because the shell builtin one doesn't interpret the \t as a tab.

Re: A really fun programming project. Operation spamstopper!

Massimo Bianchi — Wed, 17 Dec 2003 09:07:44 GMT

Mark, I say just one thing: Beaten :)

Your expr is very nice, i was looking for something similar but i didn't know, so i wordarounded...

Thanks for the lesson :)

Massimo

Re: A really fun programming project. Operation spamstopper!

Mark Grant — Wed, 17 Dec 2003 09:13:08 GMT

Massimo :)

Actually, I don't think it's fool-proof but I think that given that SEP is grepping "received" lines from an e-mail we'll probably get away with it.

I thought about trying to get his exceptions list in there as well. I can feel it's a simple thing to do if we have a file of IP's that we don't charge for storage :) but I just can't see it !

Re: A really fun programming project. Operation spamstopper!

Geoff Wild — Wed, 17 Dec 2003 09:45:22 GMT

Not an answer - but just curious how you are trapping "unsolicited" email to spam@yourdomian.com

You have to be careful with automation....

What if someone has a case of fat finger syndrome - and mis-types your email address - I assume that if the user does not exist - then it goes to your spam account - and then you automatically block them!

I too block a lot of spam and have some pretty cool rules in place...

Rgds...Geoff

Re: A really fun programming project. Operation spamstopper!

Kent Ostby — Wed, 17 Dec 2003 09:53:54 GMT

Caveat 1:

When you cut and paste the script, keep in mind that its a 2 line script so you may have to join lines after the cut and paste.

Caveats 2 & 3:
In creating your original file you want to make sure you're only grabing the originator and not the poor guy grabbed on a hop.

Also, you may encourage them to send you more spam if you make them mad, though I salute your valor.

Here is the script:

grep [0123456789] $1 | sed -e"s/\[//g" -e"s/\]//g" -e "s/(//g" -e"s/)//g" > .um2
awk ' { for (idx1=1;idx1<=NF;idx1++) {if (match($idx1,"[0-9][0-9]*\.[0-9][0-9]*\
.[0-9][0-9]*\.[0-9][0-9]*")>0) {printf ("%s\t\t We charge $500 storage fee per u
nsolicited message\n",$idx1)};} }' <.um2

Re: A really fun programming project. Operation spamstopper!

Geoff Wild — Wed, 17 Dec 2003 10:09:20 GMT

Here's one that has stumped me - I get about 5000 of these as day:

Dec 17 07:13:46 myserver sendmail[2555]: hBHFDj9J002555: ... User unknown
Dec 17 07:13:46 myserver sendmail[2555]: hBHFDj9J002555: from=<>, size=3587, class=0, nrcpts=0, proto=ESMTP, daemon=MTA, relay=[213.86.143.191]

Sort of brute force guessing for users....

According to the RFC - I am not allowed to block from=<>

Anyone else have this issue?

I've thought of adding ip's to my iptables - and blocking there - cause with sendmail I am just wasting cpu...

SEP - sorry if I'm out of line posting in your thread - but I thought this was a good topic...

Rgds...Geoff

Re: A really fun programming project. Operation spamstopper!

Steven E. Protter — Wed, 17 Dec 2003 11:45:49 GMT

I have no problem with any of the posts, even the last.

I will test these today if possible. I don't have the file at work, but can obtain it via ftp.

I think there are very good applications for this utility once its done. That perl answer looks wonderful.

I think there are going to be a lot of rabbits hopping around this script.

Once I get something working, I'll award 10 points. Anything that tests out correctly but isn't used will get 9. If it fails on testing ,I'll be creative.

I may need additional help to have the script check a list of ips that can not be added to the bounce list(my dad's ip, my own ip block for example.).

I shall get back to you.

SEP

Re: A really fun programming project. Operation spamstopper!

Jordan Bean — Wed, 17 Dec 2003 13:33:51 GMT

Attached is a perl script that I've been using for a couple years to pull IPs directly from a unix mailbox containing only offensive material. I hope it is self-explanatory.

perl direct.pl < mailbox > iplist

Re: A really fun programming project. Operation spamstopper!

Steven E. Protter — Wed, 17 Dec 2003 13:42:48 GMT

Jordon,

I love it.

I may need to name a child after you.

SEP

Re: A really fun programming project. Operation spamstopper!

harry d brown jr — Wed, 17 Dec 2003 13:56:20 GMT

I use this to validate IP's and subnet masks:

###############################################################################
# validate IP address
###############################################################################
sub ValidateIPaddr {
############################################################################
# save the "passed" String IP address
############################################################################
my ($IPaddrSTR) = @_;
############################################################################
# validate the IP address - notes: ZERO (0) leading IP addresses are octal
# and we are NOT going to handle that - though it
# can easily be done - I consider that to be an
# error because system routines do not return IP's
# in octal string format!!! ie: 192.168.026.0 is
# NOT 192.168.26.0, it is 192.168.22.0 (decimal)
############################################################################
if( $IPaddrSTR !~ m/^ ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
\. ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
\. ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
\. ( \d | 1?\d\d | 2[0-4]\d | 25[0-5] )
$
/xo) {
return(1); # wrong anwser
} else {
return(0);
}
}

live free or die
harry

Re: A really fun programming project. Operation spamstopper!

Steven E. Protter — Wed, 17 Dec 2003 14:30:25 GMT

I'm going to have to buy some more rabbits.

SEP

Re: A really fun programming project. Operation spamstopper!

Karthik S S — Wed, 17 Dec 2003 16:55:24 GMT

Hi SEP,

I know that this is a STUPID script. But unfortunately it works :-(

for i in `cat ssk`
do
echo $i
done | grep [0-9].[0-9].[0-9].[0-9]|grep -v - | sed s/$//g | sed s/$//g | sed s/'\['//g | sed s/'\]'//g | sed s/\$/"`echo "\t\t"`We charge \$500 storage fee per unsolicited message."/g

And the output comes like this,
69.70.154.90 We charge $500 storage fee per unsolicited message.
102.183.181.197 We charge $500 storage fee per unsolicited message.
211.114.193.243 We charge $500 storage fee per unsolicited message.
110.186.71.219 We charge $500 storage fee per unsolicited message.
24.232.15.186 We charge $500 storage fee per unsolicited message.
34.83.110.48 We charge $500 storage fee per unsolicited message.
65.208.147.35 We charge $500 storage fee per unsolicited message.
66.92.143.194 We charge $500 storage fee per unsolicited message.
169.254.98.41 We charge $500 storage fee per unsolicited message.

-Karthik S S

Re: A really fun programming project. Operation spamstopper!

Karthik S S — Wed, 17 Dec 2003 16:59:39 GMT

Of course it has 2 tabs between the IP Addr and the message :-)

69.70.154.90 We charge $500 storage fee per unsolicited message.
102.183.181.197 We charge $500 storage fee per unsolicited message.

-Karthik S S

Re: A really fun programming project. Operation spamstopper!

Steven E. Protter — Wed, 17 Dec 2003 17:14:58 GMT

I can do it with shell.

I can do it with perl.

I can do it whiled standing on my head or sitting down.

This is truly a wonderful forum and I'm going to be a better shell programmer and perl programmer.

I'm going to combine the concepts and create a perl version I can run via the web and a shell version.

I'm going to add a check for a list of IP adresses to not ban, and build in the part that writes a second line knocking out the entire class C block the spam came from.

I know its a nuclear weapon, but it is powerful.

I'm under the weather and have maintenance until late this evening. I'll try and turn this around and hand out those rabbits before close of business Thursday.

SEP

Re: A really fun programming project. Operation spamstopper!

Steven E. Protter — Wed, 17 Dec 2003 17:16:42 GMT

Oh:

Karthik S S

I missed your congrats thread.

Congrats.

Nice hat.

Enjoy Hogwarts.

SEP