ProLiant Servers (ML,DL,SL)
cancel
Showing results for 
Search instead for 
Did you mean: 

snmpd reboots every Sunday at 04:02 ?

kdecarlo
Occasional Advisor

snmpd reboots every Sunday at 04:02 ?

Redhat v4 running on Prolian 585 G2.   No errors reported, however reboots at 04:02 evry Sunday .

Sep 23 04:02:13 isdwlprdx10 snmpd[30954]: Received TERM or STOP signal...  shutting down...

Sep 23 04:02:13 isdwlprdx10 snmpd: snmpd shutdown succeeded

Sep 23 04:02:13 isdwlprdx10 snmpd: snmpd startup succeeded

as an example.

Ran diagnostics 24 hours no errors.   No errors reported from the OS ??

4 REPLIES
Matti_Kurkela
Honored Contributor

Re: snmpd reboots every Sunday at 04:02 ?

I don't have a RHEL 4.x system at hand at the moment, so I'm looking at a RHEL 5.x system here. But I know RHEL 4.x and 5.x are at least somewhat similar regarding snmpd, so I guess whatever I'll find might apply to RHEL 4.x too. But please don't rely on my guesses: take a look in your system and verify.

 

"Every Sunday at 04:02"? Seems like it might be a regular scheduled event. Those are handled by the cron daemon, which has per-user crontab files in /var/spool/cron and a "superuser" crontab file at /etc/crontab.

 

A quick peek to /etc/crontab indicates 04:02 is the time the system runs all the scripts located in the /etc/cron.daily directory. That might be worth a look.

 

And that directory includes a script called... "logrotate". Hmm, I seem to remember that the logrotate tool sometimes needs to stop & restart some processes to do its stuff. Let's investigate.

 

/etc/logrotate.conf indicates log files are rotated weekly by default. This file defines only /var/log/wtmp and /var/log/btmp as log files to be rotated. But it also says: (at least in RHEL 5)

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

 

That directory includes a file named /etc/logrotate.d/snmpd. In RHEL 5.x and later, it defines /var/log/snmpd.log as a log file to be rotated, and some commands to run after the log rotation (the "postrotate...endscript" part).

 

In RHEL 5.x, the postrotate command simply sends a "kill -HUP" to the snmpd process. A HUP signal causes snmpd to close all the files it's currently writing to (including the log file), reload its configuration and resume normal operation. Most (but not all) daemon processes react this way to the HUP signal: it is a kind of standard practice in Linux/Unix world.

 

This is exactly what you want to do when you're rotating a log file: if the log file is still open when logrotate renames it (e.g. from /var/log/snmpd.log to /var/log/snmpd.log.1), the daemon keeps writing to the renamed file (snmpd.log.1) until the file is closed. When snmpd opens the log file again, it follows its configuration and opens /var/log/snmpd.log again... which, thanks to logrotate, is now a pristine, empty file.

 

If the daemon did a fseek() to the end of file before every log write operation (some Java server applications do that), it would also be possible to copy the contents of the existing log file to another file and truncate the original file without informing the daemon at all. But in a multi-processor system, if a log entry was written to the file exactly between the "copy" and "truncate" operations, the log entry might be lost. So logrotate's strategy guarantees that no log entries are lost.

 

If the snmpd of RHEL 4.x does not support the HUP signal, the /etc/logrotate.d/snmpd might completely stop and restart snmpd instead. (I think it does exactly that, but please verify it yourself.)

 

In short: I don't think the snmpd restart weekly at 04:02 is an error. It is probably a consequence of the default configuration of logrotate on RHEL. Logrotate runs daily, but it is configured to act on most log files once a week only.

(If the system is down at the time the log should be rotated, the rotation will happen on the next day, instead of after a week.)

MK
kdecarlo
Occasional Advisor

Re: snmpd reboots every Sunday at 04:02 ?

Thank You for that information.  I will check that out tomorrow morning.

 

 

Matti_Kurkela
Honored Contributor

Re: snmpd reboots every Sunday at 04:02 ?

Now I got access to a RHEL 4.x box with snmpd installed. The configuration of scheduled jobs and logrotate is much like in RHEL 5.x, except /etc/logrotate.d/snmpd contains this:

/var/log/snmpd.log {
    notifempty
    missingok
    postrotate
        /sbin/service snmpd condrestart 2> /dev/null > /dev/null || true
    endscript
}

 

So, after rotating the /var/log/snmpd.log file, logrotate completely restarts snmpd if it is currently running (that's what the "condrestart" means).
That definitely explains why you're seeing the snmpd restart messages.

 

By looking at the snmpd.conf man page, I see RHEL 4.x snmpd recognizes the HUP signal... but apparently it does not close its log files when it receives the signal, only reloads its configuration. I guess this is one of the enhancements that happened going from RHEL 4 to RHEL 5.

MK
kdecarlo
Occasional Advisor

Re: snmpd reboots every Sunday at 04:02 ?

  Thanks,  I have submitted this information to the System Admin.   I am waiting for their input.