Time change caused one of my clustered servers to reboot

[John Jimenez]

I have to HPUX 11.23 clustered with Oracle RAC. Yesterday one of the Servers just booted. The sys log on this and all my other servers show this. the last thing on the OLDsyslog.log is syncronisation lost. I assume something happened with the time change. I am using 2 windows Domain Controlers as the Network Time Protocol Server for my HPUX servers. Earlier this year the Windows department moved the time on these servers and this also happend then. the windows Has anyone ever had this issue? Any idea on how to fix it from happening again?

Nov 1 17:30:26 maxux2 xntpd[1911]: time reset (step) -2.004320 s
Nov 1 17:30:26 maxux2 xntpd[1911]: synchronisation lost
Nov 1 17:37:53 maxux2 xntpd[1911]: synchronized to 10.0.8.1, stratum=1
Nov 1 17:35:50 maxux2 xntpd[1911]: time reset (step) -123.254810 s
Nov 1 17:35:50 maxux2 xntpd[1911]: synchronisation lost

[Torsten.]

Use another (external) time server.

It's always a bad thing when windowz admins chrashes un*x servers.

[John Jimenez]

I just spoke with windows and asked them if they changed their clocks backwards by 123 seconds. These Domain Controllers before we not being synced to a outside clock. So yesterday they installed a program to sync it up with an outside atomic clock.

So either my problem will be fixed and go away, or if this cluster is really time sensitive it might now start rebooting all the time.

[John Jimenez]

Any body out there using clustering with or without RAC ever have any similar problems?

[Bill Hassell]

These step changes are not normal and should never be allowed in clustered systems. If it's OK for the Windows department to crash your servers, go ahead and use the PCs for your time source. If these servers are important, get off the Windows boxes immediately and use real time servers. You can get inexpensive WWV/WWVH radio receivers for your data center, or better yet, have the network administrators configure your firewall routers to obtain NTP from the NTP pool and broadcast NTP timesync to the rest of your data center. The PCs will also benefit by pointing to these NTP sources from the firewall.

[John Jimenez]

Thanks for your reply Bill.
I am no expert on Clusters. These two RP7420 servers were set up by the application vendors 3 years ago and are my first and only clusters I have administrated.

So the issues we experienced with clusters is to be expected when someone manually modifies the time backwards.

Last year an Admin manually changed the time on the D.C. Server (10.0.8.1) and one server rebooted. But last weekend The Windows group actually did set up the D.C. server (10.0.8.1) to connect to a automic clock on the outside. When it synced up it had to go back 123 seconds. But now that this D.C. server is synced with a real outside source I wonder if this is okay?

But change to oustide source? I probably better direct this question to application vendor.
Since these clustered Unix 11.23 servers work hand in hand with other applications that run on windows servers. I want to make sure that connecting the HP servers to outside time source while other servers running applications running on windows will not cause other issues.
If you have any more comments please let me know. But you two have been a great help. thanks.

[Bill Hassell · Accepted solution]

If you want the cluster to work reliably, then the Windows group CANNOT make any changes to their clock while the cluster is enabled. Windows boxes have a poor reputation as NTP sources for a variety of reasons, one of which is that changes are sometimes made without regard to other systems that depend on them.

As far as changing your server to use an outside source, this is already setup for the Windows boxes. Just change the server IP address in the file /etc/ntp.conf from the IP for the Windows box to the address that the Domain Controller is using. Now the Windows boxes and your server will all point to the same source. You are correct that the applications and cluster must be in sync. It is best to obtain the correct information directly, like the Windows boxes.

[John Jimenez]

Thanks again Bill.

[John Jimenez]

In back of my mind I was wondering about this too. Thank you for the suggestion. I will find out and use the same source as the Windows Domain Server as you suggested. This way if they ever do something funny to that Domain Server, I will not be effected. Thanks again.