sendmail - increase number of sendmail processes

[Martha Mueller]

I would like to increase the number of sendmail processes that run at one time. Even if I have a large amount of mail stacked up, I only seem to get about 18 sendmail processes actually doing work. I have tons of resources available, my load average is under 1. How can I convince sendmail to use say, 150 processes instead of 18?

[Christopher Caldwell]

sendmail usually runs as many processes as it needs (supported by resources).

RefuseLA, QueueLA, and ConnectionRateThrottle act as governors.


MaxDaemonChildren serves as a cap.

Default is twelve
#MaxDaemonChildren=12

Recommendations are to use MaxDaemonChildren to cap processes when running the queue from the command line.

Most folks recommend that you _not_ set MaxDaemonChildren in sendmail.cf, becuase the setting can result in a simple DOS attack. When 0, less than 0, or not defined, there is no limit define.

When you look for sendmail in the process list (ps -ef | grep sendmail) if sendmail doesn't have a notation that indicates that it's reject connections, you don't need to tune it at all (it's got all of the processes it needs).

[Martha Mueller]

This is all good information. However, how can I increase the throughput of sendmail? It took half an hour to send out 8000 messages this morning. That was using about 18 to 20 sendmail processes at once. If I could increase this number of processes, perhaps it would only take 5 minutes to send out these messages. There again, I have tons of resources available. Why won't sendmail take advantage of all this idle cpu time?

[Christopher Caldwell]

Did you have 8000 messages because of a queue backup or because of mass mailings?

[Martha Mueller]

A little of both. I don't know why we had such a backlog from last night...that isn't normal. I simply restarted sendmail and it sent out the messages this morning.

We run a subscription service for people who are interested in keeping up with racehorses that are aiming for the big races, like the Kentucky Derby, Preakness, Belmont, and others. So we have many people that get the same information all at once. We would like to increase the throughput so that they all get mail about the same time.

[Martha Mueller]

O QueueLA=30
O RefuseLA=30
#O MaxDaemonChildren=12
O ConnectionRateThrottle=125
uptime
10:19am up 19 days, 9:48, 6 users, load average: 0.15, 0.28, 0.32

[Christopher Caldwell]

Managing the queue because of down time versus mass mailings are very different. Let's assume the down queue was off-normal, for now.

There's a fair bit of information available on tuning sendmail for lists; I'll give you a few pointers:

Order the domain portion of the To: addresses by domain name. When you do this, sendmail opens one connection to send to many recipients at the same domain (making delivery very efficient - since sendmail doesn't have to do connection setup for each recipient).

Consider breaking the list into queues by domain (for major domains) or at least groups of domains. Let different sendmail queue processes run the different queues.

Get the sendmail book by Costales and Allman - see the reference on alternate queues.

Look on Usenix (www.usenix.org) for documents related to tuning sendmail for lists.

[Martha Mueller]

Thanks. I'll start researching those points now.

[Christopher Caldwell]

ConnectionRateThrottle is for inbound connections - I think most of your "problems" are outbound related.

[Martha Mueller]

Yes, that's true. We have no incoming mail, just that which would be generated by system processes for root, or things like a user's print job being canceled.

[W.C. Epperson · Accepted solution]

Chris gave some good advice on tuning and researching tuning issues.

One obscure issue on outbound mail: sendmail likes to do an "ident" check on inbound connections. If it gets a "connection refused" (TCP RST) from the server on the ident check, it proceeds with the SMTP transaction right away. If the ident request is silently dropped by a firewall, the receiving server will wait for "Timeout.ident" interval before proceeding. We've seen severe impacts from this in medium-large scale sendmail environments like yours.

You can avoid this problem outbound by having your firewall issue a RST as a proxy for the mail server, or by letting it pass the request through to the server, which can either return an ident or decline the query with a RST. If ident (auth) is not set up, inetd will automatically return the RST from the server.

[Martha Mueller]

Oh, that sounds very interesting. How do I accomplish this? Our network people handle the firewall, so they may already know how to do this. But, would you be so kind as to clue me in?

Another thing I did yesterday that has helped was to relocate the mail queue and syslogd destinations onto SAN rather than the internal root disk. We did see a significant performance boost just from that change.

I am really looking forward to implementing this change as well.

[W.C. Epperson]

Martha,

Sorry I didn't see your follow up question until now. I described the behaviors of sendmail regarding ident at a tcp protocol level. How to configure the pass-through or do the proxy RST will depend on what sort of firewall it is. Your firewall admins should be able to relate the description to your specific firewall.

[Martha Mueller]

All very helpful. One of the big improvements to throughput was taking the large mail list and using a for loop to send individual mails, rather than sending mail directly to a mail list, and ending up with just a single mail message with many recipients. When I tried sending directly to a mail list, I found that some domains thought I was spamming, so they dropped the mail. Also, when you end up with just a single mail, there is only one sendmail process that attempts to process, rather than spawning many processes. I finally got about 60 processes at a time using the individual mail method.

Berlene also sent me information about kernel parm changes that should help. They were very informative. Berlene, do you mind if I publish those tips here?

[Christopher Caldwell]

Remember that number of process != (necessarily) thruput.

Using your loop technique, [when applicable] you should break e-mails by domain, and deliver a single e-mail to many users at the same domain. This technique has a much higher thruput than a single process per e-mail (though it will only take a single process to deliver this mail to more than user). This technique also "shields" you from the domains that reject your mail (they'll be handled by their own process).