NTP Clients not sync with NTP Server

[KarloChacon]

I have several Red Hat 5.7(7) all of them sync time with two NTP Servers based on Windows but the issue is I configure the NTP service, I start the ntp service and use ntpdate -u to set the time the first time

This is what I do

1. Configure /etc/ntp.conf

Code:
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

restrict 127.0.0.1
restrict -6 ::1

server 101.122.3.111
server 101.122.6.112

broadcastdelay 0.008
authenticate no

server 127.127.1.0 
fudge 127.127.1.0 stratum 10

keys /etc/ntp/keys

2. set the date

Code:
ntpdate -u NTP1 NTP2

3. Start the NTP daemon

Code:
service ntpd start

4. Add ntpd to bootup

Code:
chkconfig ntpd on

but today servers are one hour ahead of the ahead of the NTP Servers...

any idea why?

I configured NTP Server and Clients with no problem but in Linux-Linux Configuration now this time NTP Server are Windows servers but I don't think thats the issue since we have other Windows Servers that sync with these 2 NTP Servers and time is fine.

Am I missing something here?

thanks a lot

[KarloChacon]

hi guys

I went onsite again

Some stuff I discovered today. First of all these Linux are Virtual Machines running on Hyper-V in case this is relevant I am mentioning.

Second these are 7 server.

1 Red Hat 5.7 x64 - Sync with external NTPs since it has internet access 0.rhel.pool.ntp.org - 1.rhel.pool.ntp.org This server is working just fine time is always OK. But at this time I don't why is that. I mean is it because RH 5.7 has better time sync than RH 5.5? or is it because is using this public NTPs?

6 Red Hat 5.5 x64 - These are in a DMZ so they synchronize with 2 Windows Servers which are synchronizing with another NTP Servers in USA and these servers which are in USA are synchronizing with public NTPs like the ones above.

now let me show you some output of the ntpq -pn command

RH 5.7
As you can see NTP 38.229.71.1 is being used as NTP Server

remote refid st t when poll reach delay offset jitter
==============================================================================
+184.22.51.42 128.118.25.5 2 u 57 64 37 36.844 -8.795 26.897
*38.229.71.1 172.16.65.22 2 u 52 64 37 91.484 -12.876 28.096
+71.252.219.43 69.25.96.13 2 u 47 64 37 84.564 -3.186 28.744
127.127.1.0 .LOCL. 10 l 25 64 77 0.000 0.000 0.001

Now output from the servers where time goes fast

As you can see I waited for about 30-45 minutes and 127.127.1.0 always was the source
you know the 10.x.x.x should be elected in order to be an external NTP server to sync with but these 2 servers never got sync with the NTP servers configured in its ntp.conf

remote refid st t when poll reach delay offset jitter
==============================================================================
10.122.2.198 10.124.50.52 4 u 197 1024 377 1.079 -55406. 30150.0
10.122.5.177 10.124.50.52 4 u 209 1024 377 0.849 -55135. 30095.1
*127.127.1.0 .LOCL. 10 l 1 64 377 0.000 0.000 0.001


remote refid st t when poll reach delay offset jitter
==============================================================================
10.122.2.198 10.124.50.52 4 u 20 64 37 1.011 -7434.2 3958.52
10.122.5.177 10.124.50.52 4 u 20 64 37 1.205 -3130.6 2773.80
*127.127.1.0 .LOCL. 10 l 15 64 37 0.000 0.000 0.001

now /etc/ntp.conf is identical for RH 5.7 and RH 5.5 obviously NTP Servers changes.

restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery

restrict 127.0.0.1
restrict -6 ::1

RH 5.7 server 0.rhel.pool.ntp.org server 1.rhel.pool.ntp.org

RH 5.5 server 10.122.2.198 server 10.122.5.177

server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10
driftfile /var/lib/ntp/drift
keys /etc/ntp/keys

Final no errors in messages...

and iptables and ip6tables are stopped

any input or ideas guys?

thanks a lot

[Matti_Kurkela]

remote refid st t when poll reach delay offset jitter
==============================================================================
10.122.2.198 10.124.50.52 4 u 197 1024 377 1.079 -55406. 30150.0

Whoa, that "jitter" value is huge. If I remember correctly what I once read from the NTP documentation, if this value is more than about 100, it will be very difficult for ntpd to sync with the time server.

The jitter value is a standard deviation of the difference between polls, in milliseconds. Here it seems to say that polling the NTP server seems to produce results that routinely have a difference of up to +/- 30 seconds(!) compared to what is expected by the system clock. Yet the "delay" value is small, suggesting that the time from sending the poll request to receiving a reply is minimal.

This does not make much sense... unless the local system clock that is used as a basis for these calculations is unstable. And in a virtual machine, that is definitely possible, as the VM host gives the VM guest a variable number of CPU cycles according to the workloads of each guest running on this VM host.

Apparently, this is a fact of life on a HyperV guest. This is a patch sent to the Linux driver development mailing list in April 2010:

http://kerneltrap.org/mailarchive/linux-driver-devel/2010/4/28/6884368

This patch is a clocksource implementation suitable for guests hosted on HyperV.
Time keeping in Linux guests hosted on HyperV is unstable. This clocksource 
driver fixes the problem. 

 Unfortunately it looks like this patch has not yet reached the RHEL 5 kernel: by your experiences, I assume it is not in RHEL 5.5 yet, and I could not find any mention of a Hyper-V timekeeping patch in the RHEL Technical Notes of the 5.6 nor 5.7 releases.

A description of the timekeeping challenges a Hyper-V guest must deal with:

http://blogs.msdn.com/b/virtual_pc_guy/archive/2010/11/19/time-synchronization-in-hyper-v.aspx

And here's a suggested solution for CentOS on Hyper-V. It is probably applicable to RHEL too:

http://blog.hompus.nl/2010/01/08/correcting-time-drift-with-centos-on-hyper-v/

In short:

Edit your grub.conf and add the following to the kernel line:

    divider=10 clocksource=acpi_pm

This makes the kernel take its time information from the ACPI timer, which in a VM is obviously virtual... but apparently it is a good enough simulation, so that the guest system can get good time information from it.

Based on my experiences on VMware virtual machines, I'd also suggest preventing ntpd from using the local clock as a reference time source on any VM guests. The local clock is obviously always in sync with itself, so if the clock itself is unstable (as it can be on VM guests) it may have an unfair advantage to real time sources. So I'd recommend commenting out these lines from your ntp.conf file:

server 127.127.1.0 # local clock
fudge 127.127.1.0 stratum 10

 Note: even with these lines commented out, ntpd will still fall back to the local system clock if the connection to external time servers is lost. It will also keep applying the average drift rate correction it may have calculated previously and stored to the drift file (/var/lib/ntp/drift). Commenting out these lines just stops ntpd from treating the system clock as a potential source of reliable time information.

VMware has a long history of similar timekeeping issues too, although by now the solutions are implemented in current kernels:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427

This article contains other NTP configuration tips too, which might also be applicable to your situation with Hyper-V.

The local clock lines are there to ensure ntpd can act as a time server even if there is no connection to external time servers: they allow ntpd to a) satisfy the synchronization algorithm and b) to serve the local clock time to any NTP clients this system may have. Since your VM guests are supposed to be purely NTP clients, b) is unnecessary to you, and a) is actually counter-productive if the local clock is not reliable.

[KarloChacon]

thanks a lot Matti

After a lot search I found similar stuff regarding these Hyper-V issue one of these is similar what you are suggesting including some trick on Hyper-V Virtual MAchine settings

1. From Hyper-V, turn off 'Time synchronization' within the VM's Integration Services section

2. Adjust the Date & Time within Red Hat

- Enable NTP

- Within Advanced options

+ Select 'Synchronize system clock before starting service'

+ uncheck 'Use Local Time Source' (if selected)

3. (using Grub boot loader) Modified:

/boot/grub/grub.conf

- Added the following kernel command line parameters at the end of the "kernel" line

notsc divider=10

thanks a lot