reduce the noise in syslog

[Les Ligetfalvy]

I have been trying to reduce the noise in the syslog of all my switches but am not getting the results I have been hoping for.

One of the sources of events is SNTP. My 5308xl switches are configured to poll my Active Directory DC for the time using SNTP but have found that the default of 720 seconds just wasn't cutting it. If/when the clock adjusts by more than 3 seconds, it logs an event. I have been increasing the frequency gradually, trying to find the right value that would sync the clock often enough to keep it under 4 seconds so as to not log but the timesync gods must not be on my side. I have the poll interval reduced to 180 seconds but I am still getting several syslog entries per day. The weird thing is that there appears to be no consistency between switches even though they are all configured identically.

Does anyone else have this problem or some words of advice? Is SNTP doomed to be flakey on these switches and should I be looking to setup a TimeP server?

Then there are all those PORTS: events in the syslog. Any advice on how to stop them from logging every time a link goes up or down?

[Regnar Bang Lyngsø_2]

How about taking a look at syslog-ng?
<>

[Les Ligetfalvy]

Thanks for the response. Your link has a trailing > but I did get to the site. It looks at first blush to be only for 'nix and not WIntel.

I was hoping to stop some of these at the switch. I already own two licensed apps for syslog. PCM+ and WhatsUp Gold (WUG) and was hoping not to have to buy a third.

WUG 2005 was recently released and I am waiting for my license key to install it, so have not yet seen what new features it has. I use WUG mainly for monitoring and alerting and don't actually browse through the logs with it, so it really matters not what noise it sees.

I use PCM+ to view my Procurve syslogs and that is where I would like to see improvement. I am supposed to be on the PCM+ 2.0 beta team whenever it goes to beta and I have been feeding suggestions to the beta stream already. If the noise cannot be controlled at the source, then maybe PCM+ can be enhanced to process them using rules and advanced filtering. Time will tell if any of my suggestions make it into the 2.0 product.

That said, I still seem to be having timesync issues and I have an incident open with HP. I have run a network trace (attached) and the switches do contact my time server every 180 seconds but for reasons unknown to me the switch does not update the time that frequently. Here is an excerpt from one of the logs:
I 11/26/04 01:53:20 SNTP: updated time by -4 seconds
I 11/26/04 04:17:16 SNTP: updated time by -4 seconds
I 11/26/04 06:44:12 SNTP: updated time by -4 seconds
I 11/26/04 09:08:08 SNTP: updated time by -4 seconds
I 11/26/04 11:32:04 SNTP: updated time by -4 seconds
I 11/26/04 13:56:00 SNTP: updated time by -4 seconds

[Les Ligetfalvy]

Hmmm... it might be me being a dufus again... :(

Initially, I was increasing the poll interval, not realizing that <4 second updates were not logged. I suspect this caused some of the switches to go as long as 4 days without an update, hence my comment "The weird thing is that there appears to be no consistency between switches even though they are all configured identically".

Some of the inconsistency between individual syslogs however may be due to the way PCM+ works. I have often found PCM+ to lag way behind my WUG syslog, and PCM+ puts the current time on the timestamp which then does not jive with what is in the switch's log. I have often seen PCM+ be hours behind which is very disappointing but that is another topic for another day.

There does seem to be consistency now that I am heading in the right direction with the poll interval but I still get time adjustments >3 seconds. My thinking now is that possibly my AD DC is adjusting its time by >3 seconds and that the switches are just reporting as expected.

I downloaded D4Time from http://www.thinkman.com/ and will use it to log the time adjustments. I am thinking that if the DC adjusts so infrequently that >3 second updates are likely, that I will just have to setup my own time server to point my switches to.

[Regnar Bang Lyngsø_2]

Hi again,

the syslog-ng is released under the GPL, meaning that is free (and in this instance, also free as in free beer). I have no clue, whether it will work under Wintel or not (but probably not).

As for the link, blame HP for not parsing RFC1738 compliant links correctly :-)

Happy hacking

[Les Ligetfalvy]

So much for my theory that the DC is changing time and the switches just following suit. After logging for over 24 hours, I see no evidence of that.

Back to the drawing board...

[Jeff Brownell]

Please review KB doc id KBAN00001158 in ITRC. Bottom line is that 5300's don't use an RTC but rather an ISR, hence the drift.

[Les Ligetfalvy]

Thanks for that, but it is not the drift per se that I have issues with. It is that no matter how frequently I sync the clock, it still reports a "SNTP: updated time by 4 seconds".

Could you please provide a link where I can search for that KB. I am not having any luck finding it.


Thanks

[Jeff Brownell]

unfortunately I've just discovered that the doc i referenced is not in the ITRC KB yet. My bad. I'm working with the knowledge base team to get it in there.

In the mean time, and what the referenced doc states is that there is an interrupt service routine (ISR) in the switch that wakes up every so often to increment the 'sysUptime' object (i'm not exactly sure what the interval is; lets say, 100ms). The switch will not make adjustments to the clock (or sysUptime object; and syslog) until there is a 4 second or greater drift from your time source (timeP or SNTP).

There are differences between the various switches regarding the frequency of drift (this is the inconsistency you note). Some switches are "driftier" than others. But all (or the 2500, 4100, 2600, 2800, 3400, 5300 rather) will not update your clock (and consequentially log to the syslog) until there is a 4 second or greater difference from your time source.

If this behavior of the switches is not acceptible to you in your environment, I suggest that you open a call with hp support and reference this forum post and KB doc (it should be in the KB soon). I will be glad to help in anyway I can if the call comes my way.

[Les Ligetfalvy]

I know that this issue is mostly cosmetic, in that it is not detrimental to the delivery of packets, but still... it does not behave as the manual says it should. Like I said in my first post, I've been trying to find the right value that would sync the clock often enough to keep it under 4 seconds so as to not log.

I have asked that the incident be reopened and have sent it to your attention at NetHelp.


Thanks