too many sendmail processes

[Nivesh Patnaik]

Hello,

I have a 9000/712 workstation receiving e-mails from several systems. The 712 box is completely bogged down. When I do:
ps -ef |grep sendmail
I get 140 odd processes that look like:

root 589 1 0 14:00:41 ? 0:00 /usr/sbin/sendmail -q30m -- only one of this kind actually
root 23041 543 0 10:12:01 ? 0:00 /usr/sbin/sendmail -bs -oL10
root 22899 543 0 10:04:09 ? 0:00 sendmail: server root@b888 [ppp.
qqq.rrr.sss] child wait
root 22805 22797 0 09:58:52 ? 0:00 sendmail: JAA22805 root@[rrr.ppp
.qqq.ttt]: DATA

I have no idea why so many process of the type above are being spawned. I rebooted the server but the processes built up again and bogged the system down at an average load of over 80,
which is terrible because the 712 box is only used as a mail repository.

I can provide more info as necessary. Please help.

Nives

[Craig Rants]

Is the system load high? If so follow this excerpt from the sendmail.cf file.

In order to limit load on a very busy system, the sendmail daemon #
# can be configured not to accept SMTP connections if the five- #
# minute load average is greater than some integer value, by #
# default 12. This value is defined on the line beginning #
# O RefuseLA=.

GL,
C

[Craig Rants]

You could also edit this value:

# maximum number of new connections per second
#O ConnectionRateThrottle=3

HH,
C

[Mark Greene_1]

Have you traced the Parent Pids of the various processes back to their source processes? Also, as root run mailq -v -d and see if your basic configuration is correct, and how much and what sort of mail may be stacking-up in the queue.

HTH
--
mark

[Craig Rants]

Ok, so I didn't read well, you load got to over 80, I am suprised that you could even login. So my first reply will probably help, but you will want to trace down the orgination of all you mail, are you getting a DoS?

MC,
C

[Sanjay_6]

Hi Nivesh,

Give this sample a try,

http://us-support2.external.hp.com/cki/bin/doc.pl/sid=8efa4c21160ec530fc/screen=ckiDisplayDocument?docId=200000024603555

Hope this helps.

Regds

[Nivesh Patnaik]

I get a bunch of e-mails which are a part of a system reporting process...not a denial of service attack here. I am able to login, but I can barely do anything on the system. I will try all your suggestions and see if that helps. Much thanks.
-Nivesh

[Nivesh Patnaik]

Most of the sendail processes had a PPID pointing back to /usr/sbin/inetd daemon.
Now the number of sendmail processes is 157.

I didn't find ConnectionRateThrottle in sendmail.cf.

What could be causing it?

[Nivesh Patnaik]

Found this line in /etc/inetd.conf:

smtp stream tcp nowait root /usr/lbin/tcpd /usr/sbin/sendmail -bs -oL10

...?

[Mark Greene_1]

do /sbin/init.d/sendmail stop to stop sendmail. This will also stop any new processes. They you can kill all the other sendmail processes. Given what you said the output was, check the crontabs to see if this mail is being generated by a scheduled job. If so, comment that job in the crontab, and then restart sendmail using /sbin/init.d/sendmail start. Then do (as root) mailq -v -d to verify your sendmail config. Maybe also check the frequency of the report process as setup in cron (if that is the case).

HTH
--
mark

[Craig Rants]

Why is stmp in your /etc/inetd.conf? Comment out that line and run inetd -c. See if that helps.

MC,
C

[Craig Rants]

Ok after looking at you message very closely I see someone is trying to use wrappers (tcpd) to filter smtp traffic. Good and Bad. If you want it wrapped, this is how it will act, if you want it to act normal, comment it out. I would say that unless you are a top secret facility with highly sensitive email... You probably don't need to have wrappers on smtp.

MC,
C

[Steven Gillard_2]

Its seems that your problem is simply that sendmail is trying to process too many messages. So either you have a huge backlog or there are just too many people trying to send mail through your server. Here is what I would do:

1. Stop sendmail. In your case this will involve commenting the smtp line out of inetd.conf and running inetd -c. Then you can either kill all the sendmail processes still running or let them finish. This will reduce the load and make troubleshooting easier.

2. Move everything in /var/spool/mqueue to another directory.

3. Restart sendmail by uncommenting the line in /etc/inetd.conf and running inetd -c. You're now only accepting new messages.

4. Monitor incoming mail for a bit with tail -f /var/adm/syslog/mail.log. Check for messages being deferred and also for mail from systems that you don't expect.

5. Examine the old queue with "sendmail -bp -O QueueDirectory=". Check for deferred messages and anything else out of the ordinary.

6. Once things are normal again, you can either slowly move the messages back into /var/spool/mqueue a few at a time (remember move the df and qf files together), or you can wait for a quiet time and run "sendmail -q -O QueueDirectory=" to process the old queue where it is.

Once you're at this point you should have a better idea about what is going wrong. It is possible that the system simply can't cope with the volume of mail that you're trying to pump through it. Its only a 712 workstation remember!

Good luck!
Steve

[Nivesh Patnaik]

Yup, we have tcp wrappers. I will try all your suggestions. Thanks for all your help. Happy holidays!

[Nivesh Patnaik]

I commented out the smtp line in /etc/inetd.conf and restarted the inetd daemon. I checked /var/mail and saw that none of the mail came in. Instead of killing all the waiting processes individually, I rebooted the server. Now, there is mail. There is only one sendmail process hanging, and the load has dropped to 1.08 from 99 at one point.
Super! Thanks to all for the assistance. I can at least log in and run commands on the system which seemed impossible earlier.

I'm curious, is there supposed to be a default smtp line in inetd.conf? Right now I have none since I commented out the smtp line that was spawning /usr/lbin/tcpd.

I figure, with the many e-mails coming in and all these tcpd daemons being spawned, the system must have been backlogged. I don't know if the old processes were being slowly executed and cleared as the new e-mails came in.

Any tips on where I can get more info on this tcpd daemon? Is it something that is suppied with HP-UX

[Craig Rants · Accepted solution]

smtp is not a default entry of inetd.conf. It usually is only entered when used by wrappers. Wrappers are not default on hp as well. Here is a description: http://www.sans.org/infosecFAQ/unix/TCP_wrappers2.htm

I would not run smtp through inetd, your configuration now should suffice.


HH,
C