NTP Problem, receiving alerts...

[piyush mathiya]

Hi All,
I am getting the bellow alerts in "/var/log/messages.log" related to NTP. We are facing this issue from long back. Please find the required information below and help me to get the exact solution...

Thanks,

=============================
Nov 23 01:20:02 XXXXXXX crond[27349]: (root) CMD (/sbin/service ntpd restart > /dev/null)
Nov 23 01:20:02 XXXXXXX ntpd[20993]: ntpd exiting on signal 15
Nov 23 01:20:02 XXXXXXX ntpd: ntpd shutdown succeeded
Nov 23 01:20:06 XXXXXXX ntpdate[27379]: step time server 155.49.111.198 offset -0.119499 sec
Nov 23 01:20:06 XXXXXXX ntpd: succeeded
Nov 23 01:20:06 XXXXXXX ntpd[27387]: ntpd 4.2.0a@1.1190-r Thu Oct 5 04:11:32 EDT 2006 (1)
Nov 23 01:20:06 XXXXXXX ntpd: ntpd startup succeeded
Nov 23 01:20:06 XXXXXXX ntpd[27387]: precision = 6.000 usec
Nov 23 01:20:06 XXXXXXX ntpd[27387]: Listening on interface wildcard, 0.0.0.0#123
Nov 23 01:20:06 XXXXXXX ntpd[27387]: Listening on interface wildcard, ::#123
Nov 23 01:20:06 XXXXXXX ntpd[27387]: Listening on interface lo, 127.0.0.1#123
Nov 23 01:20:06 XXXXXXX ntpd[27387]: Listening on interface eth0, 155.49.227.10#123
Nov 23 01:20:06 XXXXXXX ntpd[27387]: kernel time sync status 0040
Nov 23 01:20:06 XXXXXXX ntpd[27387]: frequency initialized 235.354 PPM from /var/lib/ntp/drift
Nov 23 01:20:16 XXXXXXX ntpd[27387]: synchronized to 155.49.111.198, stratum 2
Nov 23 01:20:17 XXXXXXX ntpd[27387]: time reset +0.191092 s
Nov 23 01:20:17 XXXXXXX ntpd[27387]: kernel time sync disabled 0041
Nov 23 01:21:31 XXXXXXX ntpd[27387]: synchronized to 155.49.111.198, stratum 2
Nov 23 01:23:34 XXXXXXX ntpd[27387]: no servers reachable
Nov 23 01:24:34 XXXXXXX ntpd[27387]: synchronized to LOCAL(0), stratum 10
Nov 23 01:24:56 XXXXXXX ntpd[27387]: synchronized to 155.49.111.198, stratum 2
Nov 23 01:24:56 XXXXXXX ntpd[27387]: kernel time sync enabled 0001

=============================
# cat /proc/version
Linux version 2.6.9-67.ELsmp (brewbuilder@ls20-bc1-14.build.redhat.com) (gcc version 3.4.6 20060404 (Red Hat 3.4.6-8)) #1 SMP Wed Nov 7 13:58:04 EST 2007

=============================

# more /etc/ntp/ntpservers
clock.redhat.com
clock2.redhat.com

=============================

# more /etc/sysconfig/ntpd
# Drop root to id 'ntp:ntp' by default.
OPTIONS="-u ntp:ntp -p /var/run/ntpd.pid"

# Set to 'yes' to sync hw clock after successful ntpdate.
SYNC_HWCLOCK=no

=============================

# more /etc/ntp.conf
# Stripped down for readability -Stv
# Tue Nov 10 14:28:54 EST 2009

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.

restrict default nomodify notrap noquery
restrict 127.0.0.1

server 127.127.1.0
fudge 127.127.1.0 stratum 10

driftfile /var/lib/ntp/drift
broadcastdelay 0.008

keys /etc/ntp/keys
server XXXXXXXXX burst iburst
restrict XXXXXXXXX mask 255.255.255.255 nomodify notrap noquery


=============================

[Kapil Jha]

what exatly is the issue?

Well what I have seen is
----------
# more /etc/ntp/ntpservers
clock.redhat.com
clock2.redhat.com
----------

Is your server is connected to internet, if not please specify a local server and then start NTP daemon.

BR,
Kapil+

[piyush mathiya]

Kapil, the configuration is correct and working in all other servers.

I have given all required information to understand the issue. I suspect this issue is because of the some configuration mismatch.

[Michal Kapalka (mikap)]

hi,

check if you can connect to the *redha.com NTP server.

mikap

[Matti_Kurkela · Accepted solution]

These are mostly informative messages, not alerts as such.

> Nov 23 01:20:02 XXXXXXX crond[27349]: (root) CMD (/sbin/service ntpd restart > /dev/null)

You have a cron job running periodically at 01:20 that stops and restarts the ntpd service.

Please check both "crontab -l -u root" output and the /etc/cron.* directories.

> Nov 23 01:20:06 XXXXXXX ntpdate[27379]: step time server 155.49.111.198 offset -0.119499 sec

The ntpd startup script expects ntpd start to happen only at system start-up, so it uses the ntpdate command to step the time to correct value. The startup script uses time servers listed in /etc/ntp/ntpservers, i.e. clock.redhat.com and clock2.redhat.com.

At start-up, this is a good thing because it eliminates any gross errors in system clock and allows ntpd to achieve synchronization faster. But if ntpd is restarted while applications are running, this may cause the time to "jump" forwards or backwards, which may interfere with applications.

If your time server (IP address 155.49.111.198) is not in sync with clock.redhat.com and clock2.redhat.com, this causes the time to jump back and forth at each ntpd restart: first the ntpdate steps time one way, then ntpd syncs with 155.49.111.198 and steps it back.

Suggestion: if 155.49.111.198 is your local time server, remove the clock.redhat.com and clock2.redhat.com from /etc/ntp/ntpservers and put in 155.49.111.198 instead.

Also it is not obvious why there is a cron job that keeps restarting ntpd. Perhaps this is an old misguided attempt to work around some time problems?

> Nov 23 01:23:34 XXXXXXX ntpd[27387]: no servers reachable
Nov 23 01:24:34 XXXXXXX ntpd[27387]: synchronized to LOCAL(0), stratum 10

These messages seem to indicate that your time server 155.49.111.198 is not always reachable, and this server must fall back to its local clock for a while. This may indicate a problem in the network or in the time server. Further troubleshooting is needed.

MK

[BUPA IS]

Hello ,
I think that the problem is related to the ntp jumping from sychronising with the local clock which always has nearly zero delay (but is not always very accurate) and the remote time source . This happens because the fudge line is treated as a reliable time source and ntp has only one other source to compare it with.
Under NTP V4 orphan mode was designed to get around this problem .

Please see the output of ntpq -c rv for the version and current state .

To try an cure the problem:

- remove the cron restart
- In ntp.conf replace the line fudge with
tos oprhan 10
This enables orphan mode or remove the fudge line altogether

- By the way burst mode makes NTP a poor network citizen and in most cases there is no need to use it .
- It is good practice to use at least three time sources so that if one goes wrong the other two will be believed.
- stop ntp
- use ntpdate to set the time once .
- empty /var/lib/ntp/drift
- start ntp with the new configuration file

> Nov 23 01:23:34 XXXXXXX ntpd[27387]: no servers reachable
- Please check that your network connections are sound .

Another possibility is a hardware problem .
You do not say which hardware you are running on.

> frequency initialized 235.354 PPM

This value is moderately large but not impossible for NTP to manage if it remains stable.
If it keeps going up and reaches 512 parts per million then NTP will give up and stops. The cron restart as a work around leads me to believe that this is the most liklely cause of the trouble. (monitor it with ntpdc -c kerninfo or loopinfo or cat ntp.drift )
In this case, as well as making the changes listed above, please check for firmware updates related to clock errors or get the system hardware clock repaired.

I hope this helps

Mike

[piyush mathiya]

Hi All,

Thanks for the reply, I will be implementing the suggestions which I got.

Still I will keep this thread open until the accurate solution. Hope I will get few more suggestion...


Thanks,
Piyush Mathiya

[piyush mathiya]

Hi,

Now we are not getting network related alerts like "no servers reachable". Still we are receiving same kind of NTP alerts. I am sending you some other information, hope it will help us to reach to the accurate solution...

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
[root@XXXXXXX log]# ntpdc -c kerninfo
pll offset: 0.001078 s
pll frequency: 239.893 ppm
maximum error: 16.384 s
estimated error: 16.384 s
status: 0041 pll unsync
pll time constant: 2
precision: 1e-06 s
frequency tolerance: 512 ppm
pps frequency: 0.000 ppm
pps stability: 512.000 ppm
pps jitter: 0.0002 s
calibration interval: 4 s
calibration cycles: 0
jitter exceeded: 0
stability exceeded: 0
calibration errors: 0

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-
[root@XXXXXXX log]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) LOCAL(0) 10 l 54 64 7 0.000 0.000 0.008
*AAAAAAAAAAAAAA 155.49.165.201 2 u 39 64 7 0.971 -83.652 52.434
[root@XXXXXXX log]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) LOCAL(0) 10 l 56 64 7 0.000 0.000 0.008
*AAAAAAAAAAAAAA 155.49.165.201 2 u 41 64 7 0.971 -83.652 52.434

-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-.-

Also I found the below information in /var/log/messages.log. can somebody tell me what it mean

Nov 28 04:40:16 XXXXXXX ntpd[3716]: kernel time sync disabled 0041
Nov 28 04:40:24 XXXXXXX ntpd[3716]: kernel time sync enabled 0001

[BUPA IS]

hello,
I am fairly sure it is oscilating between the sources of local and the single remote source .
Remove the LOCAL definition it is making things worse. i.e delete these two lines:
>server 127.127.1.0
>fudge 127.127.1.0 stratum 10
and if you are on NTP 4.2 or above replace them with :
tos orphan 10
Also the drift is going up still please investigate firmware patches for your server .

I hope this helps
Mike

[BUPA IS]

Hello,

Nov 28 04:40:16 XXXXXXX ntpd[3716]: kernel time sync disabled 0041
Nov 28 04:40:24 XXXXXXX ntpd[3716]: kernel time sync enabled 0001

These are the log messages indicating that NTP changed its synchronisation status the codes are the bit flags . The kerninfo output lists which ones are currently on.
or use ntpq -c rv
For the full description you can see and enjoy the NTP documentation here :
http://www.eecis.udel.edu/~mills/ntp/html/index.html
All the event codes are all listed here :
http://www.eecis.udel.edu/~mills/ntp/html/decode.html

Mike