topic Re: Linux admin's What are you doing about the latest sendmail security problem in Operating System - Linux

Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Wed, 29 Mar 2006 02:59:27 GMT

Sendmail race condition issue

CERT has reported a race condition issue in sendmail which may lead to
arbitrary remote code execution.

CERT has assinged this issue the name VU#834865

This issue also affects RHEL3
This issue also affects RHEL2.1

To quote CERT regarding this patch:

A patch to correct this issue in sendmail versions 8.13 is provided
below. The patch also eliminates potential integer overflows in how
sendmail handles message headers. This patch was prepared manually by
Sendmail and in our experience will generate warnings about
offsets. We've discussed this with Sendmail and believe it to be
harmless. Aside from that, CERT/CC has not verified this patch, what
issues are corrected, and how those issues are corrected.

I have a mail gateway server RH AS 2.1 at risk.

RH seems to say upgrade to their sendmail 8.12 and then apply a patch at sendmail.org.

I'm having trouble finding the patch and would like to know what upgrade procedure people are using.

I'd really rather just install a 8.13.x rpm but RH does not seem to provide such a thing.

SEP

I find RH's notice confusing.

Re: Linux admin's What are you doing about the latest sendmail security problem

Stuart Browne — Wed, 29 Mar 2006 04:30:20 GMT

sendmail-8.12.11-4.21AS.8.src.rpm

This SRC file already has this patch applied to it.

You may just need to download it and compile it yourself.

ftp://ftp.redhat.com/pub/redhat/linux/enterprise/2.1AS/en/os/SRPMS/sendmail-8.12.11-4.21AS.8.src.rpm

Re: Linux admin's What are you doing about the latest sendmail security problem

Stuart Browne — Wed, 29 Mar 2006 04:41:32 GMT

'n heh.. no fair.. I was most of the way through writing nice instructions on how to modify the spec file to do it all fo ryou.. I go to do the build, and the patch doesn't apply!... already in there.. *sigh* ah well :)

Re: Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Wed, 29 Mar 2006 05:54:46 GMT

G'day Stuart,

It would appear we can just install the binary rpm file. I've downloaded it and have initiated our internal change management process in order to come up with a schedule.

SEP

Re: Linux admin's What are you doing about the latest sendmail security problem

Vitaly Karasik_1 — Wed, 29 Mar 2006 05:56:31 GMT

Steven, because you have RHN subscription, you can download *binary* sendmail RPM from RH.
According to RHSA-2006:0265-01 (https://www.redhat.com/archives/enterprise-watch-list/2006-March/msg00017.html), sendmail-8.12.11-4.21AS.8.i386.rpm contains the latest Sendmail path.

Re: Linux admin's What are you doing about the latest sendmail security problem

Stuart Browne — Wed, 29 Mar 2006 06:05:33 GMT

Ahh, good news. (I don't have a RHE subscription handy here, so couldn't check).

Certainly makes life easier.

Re: Linux admin's What are you doing about the latest sendmail security problem

Ivan Ferreira — Wed, 29 Mar 2006 08:10:54 GMT

Remember that CentOS provides the same packages that Enterprise:

http://rpm.pbone.net/index.php3?stat=26&dist=43&size=528888&name=sendmail-8.12.11-4.21AS.8.i386.rpm

Re: Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Wed, 29 Mar 2006 08:49:28 GMT

Thanks to all. Change Management request is in. Any symptons to worry about? Post em.

SEP

Re: Linux admin's What are you doing about the latest sendmail security problem

Vitaly Karasik_1 — Wed, 29 Mar 2006 11:50:10 GMT

I don't expect any problems, but reading Changelog will be a good idea. IIRC, "rpm --changelog packagename" will provide it.

Re: Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Wed, 29 Mar 2006 13:54:59 GMT

Thank You Vitaly.

Readers will probably benefit from knowing that Vitaly built the servers in question.

:-)

SEP

Re: Linux admin's What are you doing about the latest sendmail security problem

Stuart Browne — Wed, 29 Mar 2006 15:13:36 GMT

That'd be 'rpm -q --changelog '..

Re: Linux admin's What are you doing about the latest sendmail security problem

dirk dierickx — Thu, 30 Mar 2006 02:11:03 GMT

from the security alert from RH:

In order to correct this issue for Red Hat Enterprise Linux 2.1 users, it
was necessary to upgrade the version of Sendmail from 8.11 as originally
shipped to Sendmail 8.12 with the addition of the security patch supplied
by Sendmail Inc. This erratum provides updated packages based on Sendmail
8.12 with a compatibility mode enabled. After updating to these packages,
users should pay close attention to their sendmail logs to ensure that the
upgrade completed sucessfully.

Just install the RPM, it is a version increase which includes the fix. the only thing left for you to do is check if it still _runs as it should_ afterwards.

Re: Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Thu, 30 Mar 2006 02:15:04 GMT

Interesting,

There is nothing in the changelog (Thanks Vitaly 10 points to you) mentioning the recent security issue.

I must conclude that there is mroe to do. Where is the security patch from sendmail to add on?

SEP

Re: Linux admin's What are you doing about the latest sendmail security problem

Vitaly Karasik_1 — Thu, 30 Mar 2006 03:55:52 GMT

Steven,
security patch is already in:

"This erratum provides updated packages based on Sendmail
8.12"
https://rhn.redhat.com/errata/RHSA-2006-0265.html

Re: Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Thu, 30 Mar 2006 08:12:08 GMT

Update,

On the first server, the update went very well. Mail was being processed nicely before and after. Had to throw out the /etc/init.d/sendmail file because we had customization to permit our virus checker to listen on port 25 and then pass cleaned messages along to sendmail.

Second server, which has been periodically overloaded with sendmail processes began to function very poorly after the upgrade and restart of mail services.

The system became so overloaded during sendmail spikes it could scarecly do anything else.

Had to add the following macros:

define(`confCONNECT_RATE_THROTTLE', `100')dnl
dnl # Accept certain number of sendmail children
define(`confMAX_DAEMON_CHILDREN', `24')dnl

The system isn't processing much mail, but other critical services it provides are at least working.

The obvious conclusion is that this update fixes security issues, but it may not be as efficient in resource use, leading to a lower tolerance for simultaneous sendmail processes.

I'm going to study the sendmail macros and look for a parameter that limits the number of connections from a single ip address, because it appears a DOS type attack is underway.

Any clues on this could lead to more bountiful bunnies for those that provide the answer.

SEP

Re: Linux admin's What are you doing about the latest sendmail security problem

Vitaly Karasik_1 — Thu, 30 Mar 2006 11:22:04 GMT

How many sendmail processes did you see when second server was overloaded?
Did you really see tons of SMTP connections from the same domain/address?

Re: Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Thu, 30 Mar 2006 13:06:20 GMT

Shalom Vitaly,

After I throttled the connections, most of the connections were from other servers on the global network. As the primaries became unable to handle the load, the cost 200 servers began to pick up and process mail. You can see the MX record to see what I mean.

Connection throttle and some subtle changes to the sendmail.mc configuration have the situaion under control. I lifted the connection throttle a few hours ago and am monitoring.

Sendmail is a subtle creature, especially when you start using macros and can easily impact a global mail system.

Kol Beseder, Baruch Hashem. Kol Yomim, ani lomed dvarim chadashim.

We're looking into limit the number of simultaneous connections for non-nds sites to these servers. Maybe some firewall traffic shaping will help.

SEP

Re: Linux admin's What are you doing about the latest sendmail security problem

Stuart Browne — Thu, 30 Mar 2006 18:19:54 GMT

We use 3 servers with the same-priority MX level, with :

define(`confCONNECTION_RATE_THROTTLE', `10')dnl
define(`confMAX_DAEMON_CHILDREN', `1000')dnl

They handle without issue up to about 40,000+/hour without batting an eyelid.

These servers do virus scanning via clamav_milter, as well as two other custom milters (written in C).

What sort of volume are your's seeing?

Re: Linux admin's What are you doing about the latest sendmail security problem

Steven E. Protter — Thu, 30 Mar 2006 19:22:30 GMT

I'd have to run stats to answer your question Stuart,

Volume is pretty high though.

SEP

Re: Linux admin's What are you doing about the latest sendmail security problem

Stuart Browne — Thu, 30 Mar 2006 21:15:54 GMT

mailstats is your friend :)

But not that friendly.

mailstats + magic + mrtg :P