Any way to set up servers to sync with ntp servers at exactly the same time

[John Jimenez]

I recently had my Oracle RAC cluster which runs on HP-UX 11.23 servers reboot because of a time clock issues.

I currently have my UNIX servers getting time from a local Windows server, that all other windows servers sync their time to.  But this weekend it seems that this windows server got over 2 minutes ahead and had to reverse 2 minutes 2 seconds.   Within 45 minutes all my HP-UX servers also adjusted back in time.    When this happend both the servers below that are clustered through Oracle RAC rebooted.     I am guessing that maybe they rebooted because servers checked at different times and that maybe the clocks were not synced for a little while.    Has anyone ever heard of this happening?    this is the 2nd time in a few years that these servers have rebooted right after clocks were changed.   Any way to have these two servers query the NTP server at the same time?  maybe if they did, the servers would not reboot.

Oct 22 11:05:12 server1 xntpd[1988]: synchronisation lost
Oct 22 11:05:12 server1 xntpd[1988]: Unsynchronized, possible remote server failure  or defective hardware clock

Oct 22 11:10:16 server1 sshd[10627]: Accepted keyboard-interactive/pam for root from 10.0.8.215 port 3582 ssh2

Oct 22 11:10:20 server1 sshd[11547]: Accepted keyboard-interactive/pam for root from 10.0.8.215 port 3649 ssh2

Oct 22 11:08:22 server1 xntpd[1988]: Unsynchronized, possible remote server failure  or defective hardware clock
Oct 22 11:10:22 server1  above message repeats 3 times
Oct 22 11:10:30 server1 xntpd[1988]: synchronized to 10.0.8.1, stratum=1
Oct 22 11:18:59 server1 xntpd[1988]: time reset (step) -62.726189 s
Oct 22 11:18:59 server1 xntpd[1988]: synchronisation lost

Oct 22 11:05:05 server2 xntpd[2009]: synchronisation lost
Oct 22 11:05:05 server2 xntpd[2009]: Unsynchronized, possible remote server failure  or defective hardware clock
Oct 22 11:08:00 server2 xntpd[2009]: Unsynchronized, possible remote server failure  or defective hardware clock
Oct 22 11:08:58 server2  above message repeats 3 times
Oct 22 11:10:08 server2 xntpd[2009]: synchronized to 10.0.8.1, stratum=1
Oct 22 11:18:37 server2 xntpd[2009]: time reset (step) -62.734808 s
Oct 22 11:18:37 server2 xntpd[2009]: synchronisation lost

[Patrick Wallek]

You can't control when servers query NTP servers.  That is part of the NTP protocol itself.

However, what you should do is have multiple NTP time sources.  That way one rogue NTP server won't adversely affect you in this fashion.  The best bet is to use the NTP pool servers.  If you are in North America, there are 4 pools of servers available for use.

The ntp.conf file I use is:

server 0.north-america.pool.ntp.org
server 1.north-america.pool.ntp.org
server 2.north-america.pool.ntp.org
server 3.north-america.pool.ntp.org
fudge 127.127.1.1 stratum 10    # localhost if NTP servers fail
driftfile /etc/ntp.drift

More options for publicly available NTP servers are available from

http://www.pool.ntp.org/en/use.html

http://support.ntp.org/bin/view/Servers/WebHome

http://support.ntp.org/bin/view/Servers/NTPPoolServers

[John Jimenez]

Thank you very much for that information Patrick.   Still learning this new forum, I will see if anyone else responds, but will certainly assign you kudos for this.         

Currently managerment wants all the servers synced together which is why they are asking me sync with this windows server.   But your suggestions does seem like a better one then I have now.     I will see if I can get approval to change.  thanks again

[John Jimenez]

Patrick, have you ever heard clustered severs rebooting when time changes like this?

[donna hofmeister]

http://www.informit.com/articles/article.aspx?p=1635203&seqNum=4

i know the above link isn't for hp-ux...but the principle remains the same.

as far as not being able to control ntp --- i'll suggest that if the ntp daemon isn't running then you're not going to have the situation you encountered -- which solves one problems but (probably) creates another.  you may want to consider having a cron job run ntpdate.

[Patrick Wallek]

I think having a cron job to run ntpdate is a worse solution.  If the times on 2 servers get off by different amount (say 1 is 15 minutes ahead and one is 20 minutes ahead) then you will still have the servers having different times and, potentially, updating at different times.

Multiple NTP sources solve this problem.  Have a read through "5.3.3. Upstream Time Server Quantity" in the doc below:

http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers