NTP

[Nobody's Hero]

Hey all,
Users are complaining about the time on my servers, they do a lot of percise validation and need time correct always. I have 45 UNIX systems using ntp, pointing to a windoze firewall for time. Firewall gets global time of course. Soooo, the firewall time is 15 minutes fast, sooo all of my servers are also. Is it safe to set time backwards? If so how should I do this. I believe the drift or lets say the gap is 15 minutes off, I dont think ntp will adjust correctly if the gap is too large. Any advice?????

[Jeff Schussele · Accepted solution]

Hi Robert,

No, it's usually not safe to "jump" the time backwards - especially for DBs.

The safe way is to use the
date -a -XXX
where XXX=slew in seconds - max 999 seconds.
NOTE the negative value for XXX, w/o that it would slew forward.
Would take about 6-10 hours to slew the full 999 seconds.
NTP cannot slew any further than 999 seconds either. This is slightly more than 16 minutes.

But the real issue here is the NTP server itself. What good is it if it's off by that much? I'd be looking at another NTP config.

HTH,
Jeff

[Pete Randall]

Robert,

Maybe I missed something. Why is the firewall 15 minutes fast? Shouldn't that be corrected first?


Pete

[Nobody's Hero]

Yes I understand that the firewall is the issue, but how do I correct this without clobbering the oracle DB's.

[Pete Randall]

Robert,

The problem is that, even if you correct all your servers, the fact that they're running NTP will bring them (gradually) back to whatever time their NTP source says. If you can get the NTP server to correct itself (preferably gradually), the other boxes should follow its lead.


Pete

[Steven E. Protter]

What should be doen is this:

The firewall should be set back 15 minutes. It should then be pointed to a valid time source on the outside, which is apparently not happening.

You hp-ux servers will NOT jump back 15 minutes. They will run their clocks slightly slower, drifting, adjusting until then synch up with the firewall.

Setting time back is not a good idea unless the system is idle with no processing running other than the OS. You even want cron shut down to prevent problems.

SEP

[Mark Greene_1]

>>Yes I understand that the firewall is the issue, but how do I correct this without clobbering the oracle DB's

You need to pick a day and shutdown the DBs to fix the time. Oracle does not play well with records time&date stamped in the future.

mark

[Jeff Schussele]

Hi Robert,

Use that date command -> date -a -XXX

BUT you'll *have* to disable NTP on the clients or the two will just fight each other.

The date will still march the clock forward - just with minutely slower seconds until the slew value is complete. This will NOT affect DBs as they will always see time going forward.

But get that NTP server fixed or another accurate one in place.

Could it be that your external source is the problem here?

HTH,
Jeff

[Nobody's Hero]

Ok, I am going to point to an accurate source now. So I can shutdown all the Oracle DB's and apps, change my ntp server to point to a new source and then restart the DB's.

But if UNIX/ntp wont jump forward 15 minutes, than how will the ntp server change it forward. Will this happen gradually forward if I point to a new ntp source and have the DB's down? Plus this may take a while correct?

So using ntp to adjust itself forward, gradually, is safer than setting the time forward?

[Pete Randall]

Robert,

Actually, jumping forward is OK, it's jumping backwards that can wreak havoc with timestamps on DB logs, etc. I believe you said that the Firewall was 15 minutes fast? If that's the case, you'd want to jump backwards. I would look into whether the firewall can utilize the nptdate command to make an immediate update while your other boxes are down. Once you get the firewall squared away, the other boxes should be able to gradually correct themselves.


Pete

[Steven E. Protter]

So long as the adjustment is forward, there is no need to shut down oracle apps or anything else. Some time based performance data will be distorted, but Oracle will not be disrupted by an adjustment forward.

A backward adjustment not using the firewall time server will disrupt oracle and possibly cause a database crash.

In general, its better to adjust the firewall in your situation. ntp will handle the adjustment of the clocks on the 45 servers.

Its also less work. Do you really want to jump on 45 servers and run a time adjustment command? I wouln't/

Any unix ntp client will work the way I described in my first post.

SEP

[James A. Donovan]

FYI...as long as the slew is less than 1000 seconds (16 minutes+40 seconds)then the ntp daemon will, eventually, catch things up. If the slew is greater than this, you'll need to run ntpdate on each server, in order to catch up.

[Bill Hassell]

When the xntpd daemon is running, massive changes in time servers are correctly handled, that is, there will never be a loss or gain of a single second during each day (there will always be 86,400 seconds no matter what). So how is this possible? xntpd has the ability to shorten or lengthen each second by a small amount. If the current time is off by 10 minutes, then the client will slowly (several hours is needed) drift towards the time server. Every second will exist (which is very important) but the width of a second will change slightly.

There is NO NEED to change any of the clients nor is there a need to 'JUMP' the firewall unless it is way off (15 mins is getting close to 'way off'). First: the firewall MUST have several NTP sources, never use just one. NTP is very smart in handling network delays and server outages, but it is important to monitor the firewall. You can do this in HP-UX with ntpq -p and see how well the NTP servers are doing. For the definitive source on NTP standards and time sources, see http://www.ntp.org web page.

Now if the windoze firewall cannot slew it's clock, then you can still jump the time as long as it is less than the 16 minutes window mentioned previously. All the clients will see the jump but will NOT jump as the NTP standard defines how to slowly adjust. HP-UX will slowly adjust and the Oracle databases will never notice any difference--all seconds will be accounted for.

Now for HP-UX, if the server is jumped say 40 minutes, it will ignore the server and just run with the internal clock. However, upon reboot, long before Oracle is started, ntpdate -b will be run in the xntpd startup script and this *WILL* jump the clock, but then it's OK as none of the applications have been started yet. IMPORTANT: do NOT run the xntpd start/stop script on a production system--it is designed to jump sync on startup and whatever the NTPDATE_SERVER (/etc/rc.config.d/netdaemons) is set to will jump the time when /sbin/init.d/xntpd start is run.

And if the change is more than the maximum allowed, all running clients will ignore the time server and freewheel using their own internal clock. Again, no loss or gain in seconds even though the server has gone wacky. So if the windowze code can't slew it's internal clock, jump the time about 5 minutes each day in the right direction until windoze is within a minute of real time. It should then start syncing properly. All the clients will simply drift towards the new reference values provided by the firewall. (and you might consider replacing the windoze NTP server with something that works as the RFC 1305 standard says it should)

[Sorrel G. Jakins]

After dealing with imaginitive time servers we bought two appliances that suck time from GPS satellites. If you want further info (brand name, $$, etc.), just ask or mailto: sorrel@byu.edu.nospam

[Nobody's Hero]

OK, just an update. The windoze time server that feeds my Unix systems was 15 minutes fast yesterday. So my systems were also 15 minutes fast. I took no action and behold, my servers are only 1 minute fast this morning. I gues its doing its thing. The problem is, why did my firewall(windoze), who uses MIT as a time resource, get 15 minutes fast in the first place. I am going to configure an HP-UX server for NTP, and punch it through the firewall with multiple time resources. I want to thank everyone who helped me understand this, especially Bill Hassell, excellent description and suggestions. I bow to you...

10x
RPM

[Bill Hassell]

That's why you NEVER use a single time server for your firewall. I typically use 4 to 6 different servers and NTP protocol works out the math to find the most accurate time. When one server goes a wacky, it will just be ignored and other servers will be used. Sounds like the windoze server needs a many additional servers. And in a production environment, I would not exclusively use .edu time sources. They may or may not exist from day to day. Look at the Weather Channel, US Naval Observatory, Jensen Research, etc. The addresses are in the ntp.org web page.