topic Re: Syslogd records same message repeatedly in Operating System - HP-UX

Syslogd records same message repeatedly

Steven Hargus_3 — Tue, 27 Jun 2006 07:47:54 GMT

I have a server set up to receive syslog messages from many other servers. After switching this server from an HP-UX 11iV1 server to a 11iV2 server, I am now seeing:

1. syslogd running at 100% cpu
2. The same messages repeated endlessly in syslog, with the same timestamp. New messages coming in are added to this loop.
3. All syslog messages from remote servers are marked as coming from the syslog server itself (the fourth field is always the hostname of the syslog server, not the remote server as it was on the V1 server).

Does anyone have any ideas?

Thanks in advance...

Re: Syslogd records same message repeatedly

DCE — Tue, 27 Jun 2006 08:16:44 GMT

Have you stopped and restarted the syslogd? It sounds like a runaway process at this point.

/sbin/init.d/syslog stop
/sbin/init.d/syslog start

After restarting does it still use 100% CPU?

Re: Syslogd records same message repeatedly

Steven Hargus_3 — Tue, 27 Jun 2006 08:35:02 GMT

After restarting the daemon I still get the same repeated messages. CPU utilization of syslogd starts out low, then ramps up linearly until it hits 100% again.

Re: Syslogd records same message repeatedly

Jeff Schussele — Tue, 27 Jun 2006 08:45:01 GMT

Hi Steven,

You might try the most recent syslogd cumulative - here:

https://www6.itrc.hp.com/service/patch/patchDetail.do?patchid=PHCO_34253&sel={hpux:11.23,}&BC=main|search|

HTH,
Jeff

Re: Syslogd records same message repeatedly

DCE — Tue, 27 Jun 2006 09:10:52 GMT

Is the system fully patched? There could be an issue with the syslogd that is resolved in the current ptaches.

Re: Syslogd records same message repeatedly

TwoProc — Tue, 27 Jun 2006 09:15:56 GMT

FWIW -

Steven,

Tell Lucky :-) that we got it working the following manner:

We started this syslog server project on HPUX and ended up running it on Linux, with no problems. It accepts messages from all the other flavors of Unix equally well.

Re: Syslogd records same message repeatedly

Steven Hargus_3 — Tue, 27 Jun 2006 11:40:31 GMT

Rebooted the server, and that seems to have fixed the problem. I have no idea what was wrong, but it's working now...

Re: Syslogd records same message repeatedly

Steven Hargus_3 — Wed, 28 Jun 2006 09:45:19 GMT

The same thing happened again this morning. Had to reboot the server again to get it to clear up.

I have the March 2006 11iV2 patch bundle loaded. I can look into the latest patch for syslog.

The problem starts at midnight according to the performance graphs (the CPU load suddenly gets very high).

I have a script that moves the syslog into a log archive directory that runs at that time. This is what the script is doing:

# Stop syslogd for a moment...
/sbin/init.d/syslogd stop > /dev/null
if [ $? -ne 0 ];
then
echo "Problem stopping syslog."
exit
fi

# Move system-wide syslog to log archive
if [ -r /var/adm/syslog/syslog.log ];
then
mv /var/adm/syslog/syslog.log ${SYSLOG}
fi

# Move system-wide mail log to log archive
if [ -r /var/adm/syslog/mail.log ];
then
mv /var/adm/syslog/mail.log ${MAILLOG}
fi

# Start syslog up again.
/sbin/init.d/syslogd start > /dev/null
if [ $? -ne 0 ];
then
echo "Problem starting syslog."
exit
fi

I could rewrite this to not shut down syslog, but instead copy the syslog.log to the log archive directory then zero out the existing syslog, but I am still curious why the current script is not working...

Re: Syslogd records same message repeatedly

DCE — Wed, 28 Jun 2006 09:52:49 GMT

In order for the script to work you may have to stop the syslog daemons on all of the client servers at midnight as well (one of them could be attempting to write to it at midnight). Try it with the copy and zeroing out of the file instead of the move.........