Syslogd records same message repeatedly

[Steven Hargus_3]

I have a server set up to receive syslog messages from many other servers. After switching this server from an HP-UX 11iV1 server to a 11iV2 server, I am now seeing:

1. syslogd running at 100% cpu
2. The same messages repeated endlessly in syslog, with the same timestamp. New messages coming in are added to this loop.
3. All syslog messages from remote servers are marked as coming from the syslog server itself (the fourth field is always the hostname of the syslog server, not the remote server as it was on the V1 server).

Does anyone have any ideas?

Thanks in advance...

[DCE]

Have you stopped and restarted the syslogd? It sounds like a runaway process at this point.

/sbin/init.d/syslog stop
/sbin/init.d/syslog start

After restarting does it still use 100% CPU?

[Steven Hargus_3]

After restarting the daemon I still get the same repeated messages. CPU utilization of syslogd starts out low, then ramps up linearly until it hits 100% again.

[Jeff Schussele]

Hi Steven,

You might try the most recent syslogd cumulative - here:

https://www6.itrc.hp.com/service/patch/patchDetail.do?patchid=PHCO_34253&sel={hpux:11.23,}&BC=main|search|

HTH,
Jeff

[DCE]

Is the system fully patched? There could be an issue with the syslogd that is resolved in the current ptaches.

[TwoProc]

FWIW -

Steven,

Tell Lucky :-) that we got it working the following manner:

We started this syslog server project on HPUX and ended up running it on Linux, with no problems. It accepts messages from all the other flavors of Unix equally well.

[Steven Hargus_3]

Rebooted the server, and that seems to have fixed the problem. I have no idea what was wrong, but it's working now...

[Steven Hargus_3]

The same thing happened again this morning. Had to reboot the server again to get it to clear up.

I have the March 2006 11iV2 patch bundle loaded. I can look into the latest patch for syslog.

The problem starts at midnight according to the performance graphs (the CPU load suddenly gets very high).

I have a script that moves the syslog into a log archive directory that runs at that time. This is what the script is doing:

# Stop syslogd for a moment...
/sbin/init.d/syslogd stop > /dev/null
if [ $? -ne 0 ];
then
echo "Problem stopping syslog."
exit
fi

# Move system-wide syslog to log archive
if [ -r /var/adm/syslog/syslog.log ];
then
mv /var/adm/syslog/syslog.log ${SYSLOG}
fi

# Move system-wide mail log to log archive
if [ -r /var/adm/syslog/mail.log ];
then
mv /var/adm/syslog/mail.log ${MAILLOG}
fi

# Start syslog up again.
/sbin/init.d/syslogd start > /dev/null
if [ $? -ne 0 ];
then
echo "Problem starting syslog."
exit
fi

I could rewrite this to not shut down syslog, but instead copy the syslog.log to the log archive directory then zero out the existing syslog, but I am still curious why the current script is not working...

[DCE]

In order for the script to work you may have to stop the syslog daemons on all of the client servers at midnight as well (one of them could be attempting to write to it at midnight). Try it with the copy and zeroing out of the file instead of the move.........