sendmail spam alsert! Update

[Steven E. Protter]

I would like to thank all the members that assisted me in the past with spam relay problems. I though you would like to know that an attack was deflected by my servers today.

This is a no point thread, unless someone comes up with something really awesome.

A common abuse tactic is to discover a cgi scrip that does sendmail and then attempt to send mail to the webmaster with thousands of cc or bcc recipients.

Note that all forms on my servers that send mail have been modified to limit recipients and relay.

In general the spammer doesn't bother to access the website to discover the form. They just take guesses for common names. The form that was abused was called form.cgi. Thats not its name any more. It showed up in the access log for the website.

I changed to form name to something very long and unguessable now. Other mail settings were of great assistance.

1) In sendmail.mc I do not allow more than 5 recipeients on an email. Since the spammer did not know that the first attempt was about 500 messages. None got through. What eventually happens is the spammer figures it out and adjusts their scripts to messages 5 at time.
2) A script I have cron running that scours the /var/log/maillog for aol recipients, the most frequent spam victims popped me an email on my screen. Every piece of mail in or out of aol gets logged and if the number has changed in the last hour, the script sends me a mail to my personal account and at work. It includes a count of aol.com in the log file. Normally when logs are reset this number runs around 50 a week. When the mail popped up an hour ago, it was 3003.

I managed to catch this attack in progress.

I shut down cron because it checks that sendmail is down and restarts it. Then I shutdown sendmail and cleared the /var/spool/mqueue/* to stop any outbound queued mail.

Stopping sendmail does not good, but I was not aware of the problem. sendmail will still send mail if a process on the local server asks. Thats why formscript abuse is very important to watch.

Rules for formscripts:
1) Limit recipients and upgrade to the latest version of formmail from Matt's Script archive. 2)Don't choose easy go guess names.
3)Note that all someone has to do to discover the name is visit the site. This is apparently too much work for the average spammer.

Easily guessable names:
form.cgi
hostform.cgi
formscript.cgi
formmail.cgi

Don't use these names.

The scoreboard please:
Spam relay attempts in the attack: 3136
Successfully sent mail: 0

Not bad.

I'll be happy to share anything, I do advise vigillance.

SEP

[Steven E. Protter]

Note: restart the httpd server after something like this. It forces cached forms to do a nice memory dump.

SEP

[Stuart Browne]

Most formmail CGI routines should have the ability to limit the receivers to a given destination, including CC's and BCC's. If not, find a newer version.

In any case, I find capitalization a decent method of confusing people trying to abuse services.

.. ;) unf .. good work ;)

[Steven E. Protter]

True enough. Caps help.

I noticed a form I used to have on one of the websites but since renamed formscript.cgi was being used to try and send mail. Busy little beavers, 20,000 mail attempts, 100,000 recipients.

Also noticed the orginating ip's were all over the globe. Almost as if some virus were bouncing around.

I did a few new things:

I wrote a script called gkillnh that lets me shut down all sendmail daemons hard instantaneoiusly. its a ps -ef grep scrpit. I'm working on what conditions need to be present to invoke it via cron and not effect legitimate mail.

Also: I wrote a script called update.dns.zone
This script automatically updates all DNS zone records to todays date and then restarts the named daemon. This has the effect if done properly of forcing all cache servers like aol.com to reload all web data from my site. I was thinking that would make old cached copies of formscript.cgi go away. The cached copies did not have protection on source ip or destinations. All the current scripts did.

Lastly I set the following parameters in my sendmail.mc file.
dnl # note these settings are only for when an attack is detected
dnl # raise this number when normal operations are needed
dnl # Accept certain number of sendmail connections
define(`confCONNECT_RATE_THROTTLE', `1')dnl
dnl # Accept certain number of sendmail children
define(`confMAX_DAEMON_CHILDREN', `2')dnl


These number are ridiculously low, but they say 1 connection per second maximum children 2 for sendmail.

This does allow the spammers to do denial of service. If they are filling up my mail queue, legitimate email can get in or out. But it does two things: It dramatically lowers the capacity of my servers to send mail, making it less attractive to spammers during the attack. It prevents the sendmail daemon from pumping up the load factor on the machine and denying cpu cycles to other services. httpd is much more important than sendmail. The attack will eventually stop and the legitimate inbound mail will come in and the legitimate outbound mail will go out.

The last thing I did was what stopped the last major attach. In the abused website, whose domain name I can't reveal due to a confidentiality agreement I have put two blank, empty files with read only rights to the world. form.cgi formscript.cgi

The culmulative effect of the dns cache force and the dummy slugs being out there is that eventually those that are running old forms out of cache will find their cache dumps and they get the new, useless zero byte versions. This last change was done within the last 15 minutes and I'm not sure what if any effect its had.

I am curious, someone probed my server, found a vunlerability, form.cgi even though the form itself was secure and all spam relay attempts failed. Somehow, people all over the planet began to try the same thing, then the switch last night to the form that was dropped from the server 120 days ago.

Somehow this vulnerability was transmitted all over the world. I'm interested in how the information was propogated and how I can monitor it. I'll hand out a bunny for the script that was used to find the vulnerability. robots.txt was fully deployed and that should have made this attack impossible.

The bottom line thus far is that no mail has gotten through, though my logfiles are a little stuffed from all the fail messages.

SEP

[Roberto Polli]

One quest for such a guru like you:
a lot of people say that sendmail has an old architecture and postfix or qmail should be used. I don't currently work with mail servers but I studied a bit postfix and it seems good enough.
What do you think about and why are you using sendmail?

Thx and peace,
R.

[Steven E. Protter]

I'm using sendmail with postfix because its the standard, its extremely efficient, scalable and secureable.

25,000 spam attempts about 125,000 emails and so far NOT one has gone through.

My other steps are meant to shut down the attacks.

Sendmail itsself is not being exploited. The attack is coming from perl/cgi forms abuse. It doesn't matter what the backend mail MTA is, because all the mail is coming from the web server,which is a local, authorized user.

Think About that.

SEP

[Steven E. Protter]

I have noticed that the attacks are continuing, though I've got them pretty well throttled.

I've built a configurable daemon that does the following:

1) checks the number of sendmail processes with ps -ef. If the number exceeds a figure in a /etc/rc.config.d parameter file(an innovation i brought over from hpux), item two is invoked.
2) All sendmail processes including the daemon are killed. The sendmail daemon is restarted.

One thing I've noticed is there isn't very much activity in the access logs for the various websites. This means that people are attempting to run their own forms to abuse the server via their browser or scripting.

As yet, I've been unable to figure out how to detect that. Since robots.txt are fully employed its not happening in the cgi-bin directory.

Two things I could use assistance with:
1) How to look at the httpd daemon's with ps to figure out the source ip or hostname of the various httpd processes. This may not be possible, but if it is, I can look at the websites themselves and block httpd access to those that are abusing websites instead of reading and enjoying them.
2)Methodology for detecting who is running what and how.

Note that this may be much adue about nothing since not a single email has gotten to aol. A mere change in tactics would however let the spammers sucessfully transmit and I MUST stop this.

Currently my throttle on the sendmail daemon has snuffed out attacks that were lasting up to an hour in around 10 seconds. The problem is that the attacker is unaware that he's failed and keeps trying.

I'm going to write a program next that sweeps the htdocs directories and subs and eliminates all execute permissions. This is my current theory on how the attacks are continuing.

Spam attempts 26,000, 130,000 messages, successful transmits ZERO!

SEP

[Roberto Polli]

SEP,
on a private webserver I experience "robot attacks". What do you think about daily/hourly ipfiltering abusing addresses taken from apache access.log?

Peace, R.

[Geoff Wild]

I've been reading "LNUX and UNIX Security" Portable Reference HACK NOTES.

There's a cool utility called NASL - Nessus Attack Scripting Language.

http://www.nessus.org

It's great for testing (security scan) your server(s).

As far as sendmail goes, is there a way to limit how much memory is allocated to it?

I just upgraded my Linux server to 640 MB - and in one day, all my ram is almost used - 631 out of 640...What do I run - DNS, Mail, and Web. I'm assuming that DNS is caching a lot, and possibly sendmail?


Rgds...Geoff

[Steven E. Protter]

Roberto,

Can you elaborate. robots.txt excludes cgi-bin and is present in every htdoc and subdirectory tree on the server.

I'm having trouble figuring out who is doing it. I think its being done with a long http:// string in a browser, but I need to identify the client, the command line so I can shut it down.

Still, no successful relay, but I'm going to shut this down.

what about some form of ps -ef | grep httpd

SEP

[Uwe Zessin]

Steven,
I thought that 'robots.txt' is just a hint for friendly user agents - if somebody ignores it - though.

Can't help with your problems, but in a local computer magazine I have read that there are groups who 'capture' PCs and place a backdoor on it. Then they 'rent' those systems to other spammers.

[Steven E. Protter]

I have heard that as well.

I'm trying painfully to install the testing tool and the deps it requires. The screenshots seem to allow me to test and hopefully fix vunlerabilities that obviously exist.

Have yet to run Bastille on this system, but the attack seems to be a little advanced for Bastille hardening anyway. If I can identify a port thats being used besides say 80 then I can shut that down with iptales.

I will report.

SEP

[Roberto Polli]

Geoff,
if I unerstood you are looking for a method for allocating limited resources to a given process/user. Linux implements this feature through pam+/etc/security/limits.conf where you can limit a lot of resources to given users.

You should limit further features from config files of postfix/sendmail but I don't know exactly. You can run it in a 'ulimit' environment adding ulimits in /etc/init.d/YOUR_SERVICE

Hope it helps.
Peace, R.

SEP,
I recognized attacks 'cause they try to explit apache stack requesting a huge string. And so a cron script blocking that addresses is a script-kiddie affair. I don't know if it is a good idea or not. Anyway I'm learning something about snort.

Peace, R.

[Steven E. Protter]

I have gotten some more strangeness.

While working on bringing up the attack simulation tool I saw this:

I got this off the log when I restarted my httpd server.

Starting httpd: (98)Address already in use: make_sock: could not bind to address 0.0.0.0:80


I have been undergoing a number of unsuccessful attempts to relay spam through my server to aol. There have been 30,000 send attempts consisting of around 180,000 messages.

All have failed, but my logs are filling up and frankly if the abuser lowers the recipient count from 5 to 2 some messages will begin to go through.

I have invoked sendmail connect throttle, but the spammers respond by creating more child processes. It started out Saturday night as a form abuse attack. I found one form that may have been slightly insecure and corrected it.

I am also running special programs that stop all sendmail processes if more than 8 sendmail processes are detected. This is however slamming some legitimate outbound and inbound mail.

Here is something curious off my firewall logs.

Apr 27 22:38:50 jerusalem named[1068]: lame server resolving '4318933.fallriver.ns.ca' (in 'fallriver.ns.ca'?): 69.90.20.6#53
Apr 27 22:38:50 jerusalem kernel: IN=eth0 OUT= MAC=00:c0:9f:08:2a:8c:00:20:6f:13:a0:7c:08:00 src=69.90.21.1 DST=66.91.173.134 LEN=291 TOS=0x00 PREC=0x00 TTL=51 ID=2904 DF PROTO=UDP SPT=53 DPT=32769 LEN=271
Apr 27 22:38:50 jerusalem named[1068]: lame server resolving '4318933.fallriver.ns.ca' (in 'fallriver.ns.ca'?): 69.90.21.1#53


I have blocked the ip addresses involved with the iptables firewall.

It appears that someone is attempting to use the above dns servers to simulate my IP address and then use http:// scripting to send mail through my server.

Am I nuts or is someone trying to shadow my IP address and send mail?

I have changed my ip info but not the spammers.

SEP

[Steven E. Protter]

I found hundreds of entries for non-existant cgi forms on my httpd logs.

I already had a script that counted cgi runs based on access and error. I modified it to produce a list of bogus attempts, form names I don't use and add the source ip addresses to the iptables block list.

Though I manually review the logs, this step seems to have quieted but not stopped the activity.

SEP

[Steven E. Protter]

I too am looking at snort. I just found it while figuring out how to to iptables logging in a seperate file. Which i did by the way.

http://www.derkeiler.com/Mailing-Lists/linuxsecurity/2002-06/0013.html

All of these attacks are directed at apache and they are showing up in the http error log. It would seem that there is a bug in apache that allows for mail transfer even if the script does not exist.

On a practical level, once I processed all of the entries looking for common but non-existant forms and integrated those ip addresses into the iptables file, quiet once again reigned on my server.

The load factor dropped, and I now have a semi automated way of getting ip addresses from the error_log to the iptables. It would seem that all I have to do is arrange for auotmated reverse lookup and these script kiddies only get one shot.

Thats fine, since security is hard enough that they took 30,000 shots and didn't get an email through. It appeared to be a denial of service attack. They tried to deny my cpu and/or fill up my log files.

I believe I "provoked" this attack. I have blocked several major isp's such as comcast from doing any transmissions via my smtp server. The bounce message indicates they should call support for their isp. Someone got angry and decided to get even.

It cost me some sleep but they failed. I now have yet another tool in my toolbox for stopping spam relay.

I'm going to register my boxes with Red Hat and run up2date in the hopes that apache has dealt with this issue. I'm going to in any event get a new version of apache into production as soon as possible.

My next fun thing is setting up a Linux Cluster.

SEP

[Steven E. Protter]

The script kiddies are totally shut down.

The latest release of the firewall reader updates the iptables firewall. It does a dig on hostnames and adds the resolution ip address to the iptables firewall.

There is currently a lot of extraneous junk on the screen, but it is effective.

Seems the script kiddies have figured out how to use aol and yahoo's mail servers to attempt to relay mail as apache@mydomain.com

It happens every hour on the hour from aol. All the mail fails, but i fear if the spammer gets a little smarter, some mail can get through. All mail is directed at aol subscribers.

I've reported the activity and submitted logs to their postmaster.

I am wondering if there is anything I can do with the user sendmail configuration that can deal with this.

Obviously apache@mydomain.com is a valid user on my servers. I seem to have a choice of thousands of error messages with user reject versus thousands of messages where the reject is because I have low recipient limits.

Eventually, I'm going to want to raise my recipient limits. So, if you get to it before I figure it out, I'm wondering how to disable the apache users mail priviledges while still lettting my formmail scripts work. I doubt this is possible.

I'm checking redhat for sendmail and apache updates.

SEP

[Steven E. Protter]

AOL shut down their subscriber or modified their mail servers. Quiet has once again reigned.

Since the user apache@hostname.myfullyqualifieddomain.com

is not actually a valid user, I'm looking for a way that lets the apache web server send mail locally and doesn't let anyone else use it. A bunny will be awarded for a scheme in /etc/mail/virtusertable ... that makes that combination come up as an invalid user. My local traffic goes out as apache or apache@localhost.

The yahoo problem last night was a port 25 attack and they're saying i submitted incomplete logs. NOT!

I did catch some attempted forms abuse and tried upgrading the form in question to the latest version of matt's formmail script. Everthing works okay, except the security strips the from address out of the fields and blows up the MTA transaction.

Actually, it wasn't the script. It was my attempt to stop the apache traffic worked a little to well.

I added this to /etc/mail/access
apache@hostname.myfullyqualifieddomain.com REJECT 550 Message
apache@myfullyqualifieddomain.com REJECT 550 Message
apache@localhost REJECT 550 Message

No from address. All web based outbound mail fails. So do the attacks from aol.

I commeted out

# apache@myfullyqualifieddomain.com REJECT 550 Message

Rebuild the mail databases and outbound mail once again works. More than likely I've openned myself up to trouble again.

I'll report back. Anybody learning anything, finding this entertaining or should I stop posting this diary of fun?

Also: Children, we've been a little slow on point assignment lately. I've done a lot of work and actually saw my Linux total DROP last night.

Whats up with transferring old threads out weeks after they were closed with solution?

SEP

[Steven E. Protter]

I had an idea.

The attacks are using the apache user

apache@hostname.myfullyqualifieddomain.com REJECT 550 Message
apache@myfullyqualifieddomain.com REJECT 550 Message
apache@localhost REJECT 550 Message

The second change the /etc/mail/access totally stopped outbound mail from the web server.

So lets say I did this:

In /etc/mail/virtusertable

apache@myfullyqualifieddomain.com newaccount

In /etc/mail/genericstable

newaccount apache@myfullyqualifieddomain.com

rebuild the databases.

I again make

apache@myfullyqualifieddomain.com an invalid user in /etc/mail/access but the mail alias will still allow the user newaccount to send the mail.

Alternatively I migtht be able to fix it by herely changing the name of the apache user in /etc/passwd to something else. None of my scripts actually use apache, they just use the default owner of the apache binaries. That could be J Fred Muggs for all I care.

I'd like your thoughts on these ideas.

SEP

[Roberto Polli]

Hi SEP,
your diary is my boot page now ;-) so thank you very much.

What about this forum I'm sad to say that a lot of postings aren't RTFM compliant, and there are only rare interesting postings (like this one)

I think newsgroup became a 'flaming pie' and these 'commercial' forum are welcome. But people wants only solutions, not geek culture...

Peace, R.

[Steven E. Protter]

Thanks Roberto,

This thread is now closed.

ITRC exists and thrives because people don't like reading manuals or can't find them.

This puppy morphed into a blog.

I will make most of the scripts available to someone who starts their own thread and hands out a bunny. The adaptive firewall has commerical potential and I'm pondering upgrading it, creating a web management interface and selling it. It totally quieted down form abuse attempts.

SEP

[Vernon Brown_4]

Hi SEP; keep up the saga !!

I'm having many similar problems and fixing a lot of them with your insight. Right now I can't run sendmail during the night without losing my /var/log/messages, /var/log/maillog files, and syslogd function. So I shut down sendmail at night.

I happened to catch the hack a couple days ago. Error messages popped up on the server console; like, "messages dropped repeated over max limit 8192 times" (Not verbatem)

The errors were scrolling continously. Upon reboot, syslogd [ FAILED ] startup. Happened three times last week. Required reformat, reinstall, and recover from backups.

All that was with RedHat 7.1; I'm testing Fedora Core 1 and up2date.

Please keep on posting; and keep fresh threads going. This spam thing has to be fixed or email will become useless. So, keep on chuggin !!

[Steven E. Protter]

All quiet on the spam front. If there have been relay attempts they have been blocked at the firewall. The firewall now does adapt itself. Anytime the script kiddies try and run forms that are on my badform list their IP address is added to the firewall. It does it by itself, so I have to watch it since poetentially someone could use it to block access to my websites.

Its been a long road, with lots of lost sleep but things are still holding fast. I'm concerned about an adaptation in the attack using a valid id. Since I use squirrelmail, switching ownership of apache to another user id, but if the attack resumes, that is an option.

SEP

[Roberto Polli]

THX!
I'm always here!
Peace, R.