Switches, Hubs, and Modems
1748011 Members
4040 Online
108757 Solutions
New Discussion юеВ

Re: reduce the noise in syslog

 
Les Ligetfalvy
Esteemed Contributor

Re: reduce the noise in syslog

My hopes were raised only to be let down. :(

I have hearn no more on this even though I did resubmit this to nethelp.

One of the problems is that PCM has too small (max = 1500 events) a size for each device. I cannot afford to squander these on "SNTP: updated time by 4 seconds" events.

It is bad enough that every time a computer is rebooted, it throws somewhere between two and eight entries in the log. That limits my history to just a few days worth, but that is really a PCM issue, not a timesync one which is what this thread is about.
Jeff Brownell
Valued Contributor

Re: reduce the noise in syslog

I viewed a case you had on sntp updates and your syslog ( 3208598143 ). It was not reopened. I'm not sure but I think that "nethelp" is specific to the Loveland RC's so cannot vouch for what to expect with that avenue.

I can say that if you call hp support at 800 633-3600 and tell the call agent that you want to reopen a case, then they will reopen the case and route it to the appropiate resource. I recommend this avenue for you.

There is development work being done to increase switch time accurancy which would decrease the log update frequency for SNTP. The more cases we have open on this, the greater the priority will be. At present we have "0" customers requesting this development so therefore it is hopefully clear that priority is naturally low. With just one active customer the priority will jump quite a bit.

Regards,
Jeff
Les Ligetfalvy
Esteemed Contributor

Re: reduce the noise in syslog

Thanks Jeff. I called in and had Diane reopen the incident. While I will not repeat her words, if I was placing odds, my bet would be on that snowball in hell.
Les Ligetfalvy
Esteemed Contributor

Re: reduce the noise in syslog

No update on the SNTP thing but I heard from Division that they may increase the syslog size from 1500 to maybe 10,000. Now, if we can get some better filtering...
Jeff Brownell
Valued Contributor

Re: reduce the noise in syslog

I've received the escalation from Loveland and am working through them to determine your "precise" requirements. I have passed on a way to squelch all sntp log events (or any broad category of 'eventtype' messages for that fact) from being written to the systems and/or event logs. I am awaiting response from them to know if this is all you need.
Les Ligetfalvy
Esteemed Contributor

Re: reduce the noise in syslog

I realize that a lot has been said on this topic and that it may be hard to see the forest for all the trees, so I will restate the fundamental issue.

The "rules" of SNTP event log entries according to the manual, "If an SNTP time change of more than three seconds occurs, the switch├в s event log records the change. SNTP time changes of less than three seconds do not appear in the Event Log"

That's how it should work. In reality, the SNTP timesync in the switch will not make adjustments of less than four seconds, therefore assuring that every time adjustment gets logged.

There are one of two possible fixes. Either have the timesync make adjustments of less than three seconds so that they do not get logged, or else raise the threshold for logged timesyncs to five seconds.

While it may be possible to poke certain OIDs in the MIB to suppress all SNTP events, that should be a last resort since it may be of value to know if/when wild swings in timesync are happening.

As for PCM+, it may be of value to have rules/filters for the syslog that could auto-acknowledge or delete event that match certain criteria. Certainly, upping the number of events from 1500 to 10,000 would be ni
Les Ligetfalvy
Esteemed Contributor

Re: reduce the noise in syslog

I am told this will go to engineering and that the propsed solution is to improve the accuracy of the RTC. While it does nothing about the timesync not logging to the syslog, at least it should reduce the sheer volume of log events.
Jeff Brownell
Valued Contributor

Re: reduce the noise in syslog

The labs have made some headway on the "SNTP time update by 4 seconds" issue, and expect to ship enhancments on the various platforms over the next few months.

If you're looking for time adjustments to be configurable with respect to logging or not logging to the syslog, we have provided this option already with the attached doc.

I am aware that you were not interested in this option since (as I was led to understand) large time synch adjustments from a switch reboot (causing a large variance in time sync from the initial synch) were used as another means notifying you that a switch crashed/rebooted/etc. If this indeed being the case, we did explain that there are many other options available to you outside of sntp logging to be notifed of such occurances.

Lastly we recieved notification that you would like a very specific set of configurable options to be implemented for sntp. We devised a version of your request in psudocode:

If absolute value of currentTime - SNTPtime < 4 seconds, do not change currentTime.
If absolute value of currentTime - SNTPtime >= 4 seconds, update currentTime to SNTPtime and no event will be logged.
If currentTime is updated by < 10 seconds, no event will be logged.
If currentTime is updated by >= 10 seconds, log an event.

While this enhancemnt request is sound in theory, in practice it requires more resources to implement than would be considered for an idle fancy. We do have an enhancment "specials process" for issues that clearly affect a customers operational environment. If you are adamant about this functionality, please continue with the call centers and request this to be reviewed via our "specials process". Every consideration will be given to any specials request recieved.

FYI, We have submitted a request to upadte our manuals with the below verbiage to hopefully prevent inadvertent mis-understandings in the future:

SNTP Messages in the Event Log
If an SNTP time change of at least four
seconds occurs, the switch├в s Event Log
records the change. The switch's clock
will not change when an SNTP update is
received which differs from the current
time on the switch by less than four
seconds. Therefore, SNTP time changes of
less than four seconds do not appear in
the Event Log.
Les Ligetfalvy
Esteemed Contributor

Re: reduce the noise in syslog

Jeff,
Killing the messenger (suppressing all SNTP logging) is not my preference but thanks for the how-to. It appears at first glance though to be only for TrapLog messages and not Syslog, which is where the SNTP messages go. I have been able to stop some alerts from going to the TrapLog before with MIB pokes but not the Syslog.

While very large increments in time adjustment do indicate a crash/reboot, I do not rely on large SNTP adjustment alerts in SysLog solely. There should be (but there are not) SNMP traps of greater than "Informational" on a crash/reboot, but that is another topic for another day. Before I pressed the issue there was even less, so I guess I should be grateful for what advances I got.

Now I understand that adjustments less than 4 seconds do not happen, so it is superflous to state in the manual that only adjustments of more than 3 seconds get logged. You may as well just say that ALL adjustments get logged! Why cloud the issue with double-speak?

Since these switches do not actually have a RTC but rather manufacture synthetic time, I don't understand why it is so difficult to reference the SNTP time source and make minor adjustments in the synthetic time heartbeat so that it drifts back (or forward) toward the time reference. If it is fast, add a few ticks, if it is slow shave a few ticks. Very small but frequent adjustments to the clock speed IMHO, would be better that resync'd time.

In my last conversation with Division, I was told the accuracy of the clock will be worked on and not the logging of SNTP events. If the clock is kept accurate using the method I outlined above, there would seldom be time resyncs of >3 seconds and therefore nothing to log. If the time were not recalibrated on the fly, the adjustments would hopefully be less frequent and log far less often.

If there is in place logic to not log adjustments <=3 seconds, why would it be so difficult to change it to a bigger number like 10?

As I mentioned very early "I know that this issue is mostly cosmetic, in that it is not detrimental to the delivery of packets" and I don't want to steal away time from someone that might write more "useful" code. I was optimistic that in PCM2 we would get better filters for the SysLog like the TrapLog has but alas, it was not to be. Maybe PCM2.1 will deliver.
Les Ligetfalvy
Esteemed Contributor

Re: reduce the noise in syslog

I have not been informed of how this was resolved but 530x code version E_10_04 has done away with the SNTP spam in my SysLogs. No word on whether they will also do the same for other models like the 28xx.

Another interesting thing to note is that the volume of events spurred by devices coming online has been reduced significantly.