Sendmail stopped working

[Waqar Razi]

We have RHEL 4.8 server and sendmail on this server is configured to forward email to the exchange server (IP Address 10.16.8.88)

# "Smart" relay host (may be null)
DS10.16.8.88

It was working fine few days ago and then suddenly it stopped working. I have checked it out by manually sending email and it looks like that instead of contacting the relay host, it is trying to contact the local host.

# sendmail -v aiz@rocketmail.com <test.mail
aiz@rocketmail.com... Connecting to [127.0.0.1] via relay...
220 star.apple.org ESMTP Sendmail 8.13.1/8.13.1; Wed, 25 Jul 2012 15:25:27 -0400
>>> EHLO star.apple.org
250-star.apple.org Hello localhost.localdomain [127.0.0.1], pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH DIGEST-MD5 CRAM-MD5
250-DELIVERBY
250 HELP
>>> MAIL From:<root@star.apple.org> SIZE=20 AUTH=root@star.apple.org
250 2.1.0 <root@star.apple.org>... Sender ok
>>> RCPT To:<aiz@rocketmail.com>
>>> DATA
250 2.1.5 <aiz@rocketmail.com>... Recipient ok
354 Enter mail, end with "." on a line by itself
>>> .
250 2.0.0 q6PJPR6K004401 Message accepted for delivery
aiz@rocketmail.com... Sent (q6PJPR6K004401 Message accepted for delivery)
Closing connection to [127.0.0.1]
>>> QUIT
221 2.0.0 star.apple.org closing connection
#

Any help would be highly appreciated.

[Matti_Kurkela]

Since Sendmail version 8.12, sendmail is no longer suid root by default. Instead, the email processing is split into two components: when you start sendmail from the command line (by root or by anyone else), it will drop all unnecessary privileges and use the configuration in /etc/mail/submit.cf: as a result, it will accept the message and pass it on to another sendmail process that runs as a daemon. The sendmail daemon process runs as root (in order to be able to write to the queue directory and to anyone's inbox). It will also handle transmitting the outgoing mail to the configured relay host.

If the connection to the relay host fails, the sendmail daemon can write the message to the queue directory to wait for further delivery attempts. The sendmail process you started from the command line has switched itself permanently to minimum privileges, so it cannot do that.

You're seeing only the first part of this two-part operation. To find out what happens when the sendmail daemon contacts the relay host, look into the mail log file (typically /var/log/mail*).

The reason for splitting the sendmail processing into two parts is privilege separation: it is intended to remove the requirement for local users (who are in the best position to perform attacks against the sendmail daemon) to access the privileged sendmail daemon directly. This should make Sendmail much stronger against local root exploit attempts.

[Waqar Razi]

Could you please provide some more guidance like how to fix it. One thing that I would like to point out that we have done Red Hat update on this server from 4.7 to 4.8 recently and it appears to stop working after that. It was working fine before that update.

Thanks again for your response.

[Matti_Kurkela]

With the information you provided so far, I cannot help: it just indicates the mail was delivered from the command-line "sendmail" command to the sendmail daemon running as root, and the sendmail daemon successfully accepted it. This is perfectly normal for sendmail 8.12.0 and newer. I need more information from you.

The next step is to look in the mail log to see what the sendmail daemon did with the mail and whether or not it was successful. If the Exchange server rejected the mail, the rejection error code can be found in the mail log. If the sendmail daemon attempted to deliver it to some wrong destination, that can be seen in the mail log too.

As your test output includes the queue ID given to the message by the sendmail daemon, this command might be the easiest way to find out all the relevant log lines and nothing else:

grep q6PJPR6K004401 /var/log/maillog*

You might also run the "mailq" command to see if your test message is still in the mail queue (i.e. the sendmail daemon is still trying to deliver it) or not.

If it is not in the queue, check your local mail inbox (for the root user, the file is /var/mail/root). Either Sendmail or the Exchange server might have generated a bounce message that describes the reason of the mail delivery failure.

[Waqar Razi]

Here is the output:

# grep q6PJPR6K004401 /var/log/maillog*
/var/log/maillog:Jul 25 15:25:27 star sendmail[4401]: q6PJPR6K004401: from=<root@star.apple.org>, size=313, class="0", nrcpts=1, msgid=<201207251925.q6PJPR4W004400@star.apple.org>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]
/var/log/maillog:Jul 25 15:25:27 star sendmail[4400]: q6PJPR4W004400: to=aiz@rocketmail.com, ctladdr=root (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=30020, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (q6PJPR6K004401 Message accepted for delivery)
/var/log/maillog:Jul 25 15:25:27 star sendmail[4403]: q6PJPR6K004401: to=<aiz@rocketmail.com>, ctladdr=<root@star.apple.org> (0/0), delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=120313, relay=10.16.8.88, dsn=5.1.2, stat=Host unknown (Name server: 10.16.8.88: host not found)
/var/log/maillog:Jul 25 15:25:27 star sendmail[4403]: q6PJPR6K004401: q6PJPR6K004403: DSN: Host unknown (Name server: 10.16.8.88: host not found)
#

I have also checked the nslookup command to verify the name resolution of the exchange mail server:

# nslookup
> 10.16.8.88
Server:         10.16.10.16
Address:        10.16.10.16#53

88.8.16.10.in-addr.arpa name = mx-tmc-06.apple.org.
> mx-tmc-06.apple.org
Server:         10.16.10.16
Address:        10.16.10.16#53

Name:   mx-tmc-06.apple.org
Address: 10.16.8.88

[Matti_Kurkela]

Apparently Sendmail does not like your smarthost setting, because it cannot find it in the DNS exactly as written.

Try specifying the smarthost using a fully-qualified hostname instead of IP address, i.e. like this:

# "Smart" relay host (may be null)
DSmx-tmc-06.apple.org

 A mail server should always have a valid hostname anyway; otherwise it will have trouble passin even the most basic anti-spam checks.

If you absolutely must specify the smarthost using an IP address, put [square brackets] around it, like this:

# "Smart" relay host (may be null)
DS[10.16.8.88]

Are you sure your smarthost configuration actually worked before the update? I think Sendmail has required square brackets around bare IP addresses for quite a while.

[Waqar Razi]

I have tried with both the configurations i-e, putting the fully qualified domain name as well as by enclosing the IP address in bracket. It is not working.

I dont know whats wrong. It was working before the update.

[Matti_Kurkela]

 Did you restart the sendmail daemon after each change to the sendmail.cf file?

And what does the mail log say about those attempts?

The mail log is your friend when troubleshooting Sendmail. If you don't read it, it's like trying to use a computer with the display switched off: while it may be possible, it is much more difficult than necessary.

To find all the mail log lines relevant to a particular mail message, you need to find the queue ID of the message.

Sometimes you can see it directly, as in your original post. In other situations, you may have to search the mail log for the correct source or destination address (note that the source address is not on the same log line as the destination address).

The format of Sendmail's log messages is:

<timestamp> <hostname> sendmail[<PID>]: <queue ID>: <information about the message Sendmail is processing>

Once you find one log line that refers to the message you want to know about, you can find the queue ID of the message, and make another search to find all the log lines relevant to that message. The dsn= and stat= fields are the most important when you want to see what went wrong.

After Sendmail version 8.12.0, the situation is slightly more complicated, as there may be two queue IDs involved when two separate sendmail processes are handling the message (as in the log search results you pasted to the thread). Also remember that log messages written by two separate processes on the same second may or may not be in the exact chronological order in the log file.