Linux admin's What are you doing about the latest sendmail security problem

[Steven E. Protter]

Sendmail race condition issue

CERT has reported a race condition issue in sendmail which may lead to
arbitrary remote code execution.

CERT has assinged this issue the name VU#834865


This issue also affects RHEL3
This issue also affects RHEL2.1

To quote CERT regarding this patch:

A patch to correct this issue in sendmail versions 8.13 is provided
below. The patch also eliminates potential integer overflows in how
sendmail handles message headers. This patch was prepared manually by
Sendmail and in our experience will generate warnings about
offsets. We've discussed this with Sendmail and believe it to be
harmless. Aside from that, CERT/CC has not verified this patch, what
issues are corrected, and how those issues are corrected.

I have a mail gateway server RH AS 2.1 at risk.

RH seems to say upgrade to their sendmail 8.12 and then apply a patch at sendmail.org.

I'm having trouble finding the patch and would like to know what upgrade procedure people are using.

I'd really rather just install a 8.13.x rpm but RH does not seem to provide such a thing.

SEP

I find RH's notice confusing.

[Stuart Browne · Accepted solution]

sendmail-8.12.11-4.21AS.8.src.rpm

This SRC file already has this patch applied to it.

You may just need to download it and compile it yourself.

ftp://ftp.redhat.com/pub/redhat/linux/enterprise/2.1AS/en/os/SRPMS/sendmail-8.12.11-4.21AS.8.src.rpm

[Stuart Browne]

'n heh.. no fair.. I was most of the way through writing nice instructions on how to modify the spec file to do it all fo ryou.. I go to do the build, and the patch doesn't apply!... already in there.. *sigh* ah well :)

[Steven E. Protter]

G'day Stuart,

It would appear we can just install the binary rpm file. I've downloaded it and have initiated our internal change management process in order to come up with a schedule.

SEP

[Vitaly Karasik_1]

Steven, because you have RHN subscription, you can download *binary* sendmail RPM from RH.
According to RHSA-2006:0265-01 (https://www.redhat.com/archives/enterprise-watch-list/2006-March/msg00017.html), sendmail-8.12.11-4.21AS.8.i386.rpm contains the latest Sendmail path.

[Stuart Browne]

Ahh, good news. (I don't have a RHE subscription handy here, so couldn't check).

Certainly makes life easier.

[Ivan Ferreira]

Remember that CentOS provides the same packages that Enterprise:

http://rpm.pbone.net/index.php3?stat=26&dist=43&size=528888&name=sendmail-8.12.11-4.21AS.8.i386.rpm

[Steven E. Protter]

Thanks to all. Change Management request is in. Any symptons to worry about? Post em.

SEP

[Vitaly Karasik_1]

I don't expect any problems, but reading Changelog will be a good idea. IIRC, "rpm --changelog packagename" will provide it.

[Steven E. Protter]

Thank You Vitaly.

Readers will probably benefit from knowing that Vitaly built the servers in question.

:-)

SEP

[Stuart Browne]

That'd be 'rpm -q --changelog '..

[dirk dierickx]

from the security alert from RH:

In order to correct this issue for Red Hat Enterprise Linux 2.1 users, it
was necessary to upgrade the version of Sendmail from 8.11 as originally
shipped to Sendmail 8.12 with the addition of the security patch supplied
by Sendmail Inc. This erratum provides updated packages based on Sendmail
8.12 with a compatibility mode enabled. After updating to these packages,
users should pay close attention to their sendmail logs to ensure that the
upgrade completed sucessfully.

Just install the RPM, it is a version increase which includes the fix. the only thing left for you to do is check if it still _runs as it should_ afterwards.

[Steven E. Protter]

Interesting,

There is nothing in the changelog (Thanks Vitaly 10 points to you) mentioning the recent security issue.

I must conclude that there is mroe to do. Where is the security patch from sendmail to add on?

SEP

[Vitaly Karasik_1]

Steven,
security patch is already in:

"This erratum provides updated packages based on Sendmail
8.12"
https://rhn.redhat.com/errata/RHSA-2006-0265.html

[Steven E. Protter]

Update,

On the first server, the update went very well. Mail was being processed nicely before and after. Had to throw out the /etc/init.d/sendmail file because we had customization to permit our virus checker to listen on port 25 and then pass cleaned messages along to sendmail.

Second server, which has been periodically overloaded with sendmail processes began to function very poorly after the upgrade and restart of mail services.

The system became so overloaded during sendmail spikes it could scarecly do anything else.

Had to add the following macros:

define(`confCONNECT_RATE_THROTTLE', `100')dnl
dnl # Accept certain number of sendmail children
define(`confMAX_DAEMON_CHILDREN', `24')dnl

The system isn't processing much mail, but other critical services it provides are at least working.

The obvious conclusion is that this update fixes security issues, but it may not be as efficient in resource use, leading to a lower tolerance for simultaneous sendmail processes.

I'm going to study the sendmail macros and look for a parameter that limits the number of connections from a single ip address, because it appears a DOS type attack is underway.

Any clues on this could lead to more bountiful bunnies for those that provide the answer.

SEP

[Vitaly Karasik_1]

How many sendmail processes did you see when second server was overloaded?
Did you really see tons of SMTP connections from the same domain/address?

[Steven E. Protter]

Shalom Vitaly,

After I throttled the connections, most of the connections were from other servers on the global network. As the primaries became unable to handle the load, the cost 200 servers began to pick up and process mail. You can see the MX record to see what I mean.

Connection throttle and some subtle changes to the sendmail.mc configuration have the situaion under control. I lifted the connection throttle a few hours ago and am monitoring.

Sendmail is a subtle creature, especially when you start using macros and can easily impact a global mail system.

Kol Beseder, Baruch Hashem. Kol Yomim, ani lomed dvarim chadashim.

We're looking into limit the number of simultaneous connections for non-nds sites to these servers. Maybe some firewall traffic shaping will help.

SEP

[Stuart Browne]

We use 3 servers with the same-priority MX level, with :

define(`confCONNECTION_RATE_THROTTLE', `10')dnl
define(`confMAX_DAEMON_CHILDREN', `1000')dnl

They handle without issue up to about 40,000+/hour without batting an eyelid.

These servers do virus scanning via clamav_milter, as well as two other custom milters (written in C).

What sort of volume are your's seeing?

[Steven E. Protter]

I'd have to run stats to answer your question Stuart,

Volume is pretty high though.

SEP

[Stuart Browne]

mailstats is your friend :)

But not that friendly.

mailstats + magic + mrtg :P