Regarding the NTP Server Configuration

[Narendra Uttekar]

Hi,
I have configured the Windows Domain Controller NTP server as time sync for HP-UX servers as NTP client. And it is working good.
The configuration of /etc/ntp.conf file as below,
server 10.0.25.91 prefer
server 10.0.25.92
driftfile /etc/ntp.drift

And the /etc/rc.config.d/netdaemons
export NTPDATE_SERVER=10.0.25.91
export XNTPD=1
export XNTPD_ARGS=

But after 1 month there was some maintenance activity on both Windows NTP server and both the server was unavailable for 1 hour. During that time HP-UX servers lost the synchronization of time with Windows NTP server and later when Windows NTP server was available the HP-UX clock got in sync.
The question over here is can we eliminate this type of problem in future because of this problem we got an error in SAP application (ZDATE_ILLEGAL_LOCTIME). I am thinking can I configure one of HP-UX server in each location as HP-UX NTP Server (We have our HP-UX servers in California and Texas. And Windows NTP servers in New York). So that even if both the Windows NTP server are unavailable or network is down still the clock will be in sync and I can eliminate the clock sync error.
Please let me know if I modify my current NTP configuration as below including the HP-UX server from each location as HP-UX NTP server will be a good solution. So that I won’t get any more clock sync error in future.
The configuration of /etc/ntp.conf file as below,
server 10.0.25.91 prefer #Windows NTP Server New York location
server 10.0.25.92 #Windows NTP Server New York location
server 10.0.28.82 #HP-UX NTP Server California location
server 10.0.14.96 #HP-UX NTP Server Texas location
driftfile /etc/ntp.drift

And the /etc/rc.config.d/netdaemons
export NTPDATE_SERVER=10.0.25.91
export XNTPD=1
export XNTPD_ARGS=

Will this above configuration will work as I required means the HP-UX clock should always be in sync with Windows NTP server. In case if both Windows NTP servers are unavailable then only it should get the clock in sync from HP-UX NTP server. And also will this above configuration will have some problem later as I will be implementing it in production system.

Thanks,
Narendra

[Michal Kapalka (mikap)]

hi,

the configuration should be good.

one remark to your situation from SAP :

the SAP application server recognizes time inconsistencies
in all relevant cases. This then triggers the runtime error
ZDATE_ILLEGAL_LOCTIME. The work process that emitted the
runtime error repeats this runtime error
periodically, since under these conditions, the SAP system is not
executable. When the time space of the time inconsistency is over,
no periodic runtime error of this type is triggered anymore.
As a default, the SAP application server permits time inconsistencies
of up to 2 seconds. In such a case, the process in question is
halted for the relevant time.


mikap

[mvpel · Accepted solution]

Rather than using the "server" directive, you should use the "peer" directive, so that if both of the principal NTP servers on your Windows systems are unavailable, the two systems will negotiate with each other to determine which of them has the best clock and make that one the server:

server 10.0.25.91
server 10.0.25.92
peer 10.0.28.82
peer 10.0.14.96

In addition, in order to permit them to continue chiming even when they've lost both Windows servers, you'll need to specify the system local clock as a time server:

server 127.127.0.1
fudge 127.127.0.1 stratum 10

Without this, when a system loses its upstream server it will simply stop offering NTP service to anyone else. Only a synchronized system (with a * next to a hostname in "ntpq -p") can offer NTP services to other hosts.

With your config, both of the systems would wind up unsynchronized and looking for a valid server at the same time, and each of them will not consider themselves or the other HP to be a valid server and you'll be back to square one with each HP running its own clock.

With the 127.127.0.1 local-clock lines above, the systems will still be able to offer NTP data to each other in order to stay in sync with each other, even though it's at a low stratum number. One system will be the server peer with its local clock as the stratum 10 time source, and the other system will be the client peer at stratum 11.

And once either one of the Windows servers comes back online, the systems will revert to using them instead, since they'll be a lower stratum number.

You should also include both 91 and 92 in the NTPDATE_SERVER directive, so that you can get an initial sync even if 91 is offline at the moment you're booting:

NTPDATE_SERVER="10.0.25.91 10.0.25.92"

You could even include the two HP servers as well - the ntpdate command will pick the best one out of the four.

[Rita C Workman]

I suppose I find it most interesting that you are relying on your in-house servers to be your central time keeper.

Have you considered rethinking your overall time grid. You might set up a couple servers, just a couple, to sync to outside servers that are designated as time-sync providers. One such site:

http://tf.nist.gov/service/its.htm
Or you can google for internet time servers for more.

Then using those couple of your servers as your internal time-masters you balance the other servers and have them sync to your internal time-masters in respective order. In the event one goes down or off-line, the clients then sync to the other time-master.
Since both the internal time-masters are syncing to the outside source, your time remains constant.
And you would only have to address manually the time on a down time-master when it comes back up.

Just a thought,
Rita

[Narendra Uttekar]

Hi,
Thanks MVPEL...I have questions as you said to use a local clock
server 127.127.1.1
fudge 127.127.1.1 stratum 10
But then every HP-UX server acting as NTP client then they will use there local clock to sync the time and then all the HP-UX server will have different time correct?

And also if i configured both the HP-UX NTP server as peer then which one will be used for the time sync for other HP-UX servers acting as NTP client when both Windows NTP servers are un-available?

And also then how to configure the HP-UX server as NTP server with respect to above information so that i can include both of this HP-UX NTP server for HP-UX NTP client configuration along with two windows NTP server?

Thanks,
Narendra

[mvpel]

Rita, there may be many different reasons why a network might be standalone and unable to reach outside servers - security, poor internet service availability, etc. Given that the HP systems are running SAP, and considering the kind of data SAP usually handles, my bet is on "security."

Of course, the solution in that case is to install a GPS clock - perhaps one of the Windows servers has one that hasn't been mentioned? Or, perhaps accurate time is less important in the environment than consistent time.

Narendra,

When you set up the local clock and fudge it to the high stratum of 10, the NTP service on the HP system will only use that clock as a time source as a last resort, and only as long as lower-stratum servers are not available. Think of the NTP system as a "tree" - stratum 0 is the GPS satellite's atomic clock - the "root directory" as it were, and stratum 1 is an HP system with a GPS clock receiver attached - "/home".

When you fudge to stratum 10, you're basically putting that clock in "/home/mvpel/src/gnu/gcc-4.4.2/contrib/regression/GCC_Regression_Tester.wdgt/en.lproj/InfoPlist.strings," as it were.

A clock of last resort, in other words.

Now, if a server resorts to this local clock, it's true that it will be keeping its own time. But you want the downstream systems to stay in sync with each other, and that's where the "peer" directive comes in.

Any systems which are peers of one another will negotiate among themselves to choose a server and the rest of them will become clients, as you understood it in your second question.

This means that all of the peers will keep the same time, even if that downstream time drifts out of sync with the unavailable upstream time servers. It'll come back into sync once the upstream time servers are available again.

So you're looking for the following in the config:

server windows1
server windows2
peer hp1
peer hp2
driftfile /etc/ntp.drift

And I'm sure that you've already suggested to the the Windows and network admins that they not do maintenance on both servers at the same time.

[Narendra Uttekar]

Thanks a lot MVPEL for brief explaination.

Yes we have our Windows NTP server configured with GPS clock. But why i am planning to have one of HP-UX server as HP-UX NTP server from
each location becuase the SAP application server's require the clock in sync with SAP database server all time. If for any reason
the clock are not in Sync then SAP will through an error (ZDATE_ILLEGAL_LOCTIME). We are more concern about consistent time among SAP Application
server's and SAP database server. We are fine if we are less accurate with GPS clock.

Please verify the configuation which i have done on the test server is good enough to implement on Production Systems.

Configuration of the HP-UX as NTP Server.
/etc/ntp.conf
server 10.28.0.42 prefer #Windows NTP Server
server 10.28.0.45 #Windows NTP Server
peer 10.33.12.29 #HP-UX NTP Server
peer 10.33.12.15 #HP-UX NTP Server
server 127.127.1.1
fudge 127.127.1.1 stratum 10
driftfile /etc/ntp.drift

/etc/rc.config.d/netdaemons
export NTPDATE_SERVER="10.28.0.42 10.28.0.45"


When i have run the command ntpq -pn on HP-UX NTP server the output is as below,
hostname:[/]# ntpq -pn
remote refid st t when poll reach delay offset disp
==============================================================================
#10.28.0.42 56.77.184.80 2 u 19 64 377 82.72 11.613 8.48
10.28.0.45 45.14.71.132 4 u 802 1024 377 83.21 -2305.3 112.46
10.33.12.29 0.0.0.0 16 - - 1024 0 0.00 0.000 16000.0
+10.33.12.15 10.28.0.42 3 u 8 64 377 1.16 -1.520 1.04
127.127.1.1 127.127.1.1 10 l 15 64 377 0.00 0.000 10.01
+10.33.12.8 10.28.0.42 3 u 1 64 377 0.99 -3.606 1.69

Why it is showing the entry for 10.33.12.8 in the output above?--As this is the HP-UX server which is configured as HP-UX NTP client with NTP configuration
as below,
/etc/ntp.conf
server 10.28.0.42 prefer #Windows NTP Server
server 10.28.0.45 #Windows NTP Server
peer 10.33.12.15 #HP-UX NTP Server
peer 10.33.12.29 #HP-UX NTP Server
driftfile /etc/ntp.drift

/etc/rc.config.d/netdaemons
export NTPDATE_SERVER="10.28.0.42 10.28.0.45"


When i have run the command ntpq -pn on HP-UX NTP client the output is as below,
hostname02:[/]# ntpq -pn
remote refid st t when poll reach delay offset disp
==============================================================================
#10.28.0.42 56.77.184.80 2 u 34 64 377 85.75 -3.804 6.68
10.28.0.45 45.14.71.132 4 u 59 1024 377 84.11 -2385.5 106.95
+10.33.12.15 10.28.0.42 3 u 22 64 377 0.63 1.810 1.10
+10.33.12.29 10.28.0.42 3 u 21 64 377 -0.35 5.520 2.72


Please let me know any modifictaion required and why it is showing the HP-UX client entry in the output of ntpq -pn on HP-UX NTP server? And also whether if
i configured all the remaning HP-UX servers as HP-UX NTP client with above NTP configuration then all the servers HP-UX NTP Client entry will appear in the output on HP-UX NTP server?

[mvpel]

Hi Narenda,

"Why it is showing the entry for 10.33.12.8 in the output above?"

When you specify a "peer" directive instead of a "server" directive, NTP goes into a "symmetric active" mode with the remote system. "Symmetric" means that it's the same on both sides.

This means that technically only one of any two given peers needs to have a "peer" line in the local NTP config file for it to join the pool of peers.

For this third HP server, adding the "peer" lines means that it has become a third peer in the pool of HP-UX time servers, even though the other two don't specify it. It will participate with the other two systems in determining an average time, and will be a candidate as being a server for the other two systems when the peer group synchronizes the time among themselves.

You can block the two main HP servers from accepting peering from any other server by specifying:

restrict default nopeer
restrict 10.0.28.82
restrict 10.0.14.96

The last two lines remove the nopeer restriction only for those two IP addresses.

If you want the third HP system to only be an NTP client, rather than a peer of the two main HP systems, then you should change the "peer" lines to "server" in that third system's NTP config.

Now, if you get to feeling more adventurous, you should look into NTP multicast/manycast. Using a 224 multicast subnet and proper configuration of your routers, you can have your four servers transmit their chime packets across your entire WAN, to be received and processed by any server configured as a "multicastclient."

This means that you wouldn't have to maintain a whole mess of NTP configuration files to be updated any time an IP address changes. Generally you'd want to use authentication keys along with this, to prevent spurious servers from disrupting things.

[Narendra Uttekar]

Thanks again MVPEL...Now i have again make the changes to NTP configuration.Please can you confirm whether this configuration
is correct...
Configuration of the HP-UX as NTP Server.
/etc/ntp.conf
server 10.28.0.42 prefer #Windows NTP Server
server 10.28.0.45 #Windows NTP Server
restrict default nopeer
restrict 10.33.12.29 #HP-UX NTP Server
restrict 10.33.12.15 #HP-UX NTP Server
server 127.127.1.1
fudge 127.127.1.1 stratum 10
driftfile /etc/ntp.drift


/etc/rc.config.d/netdaemons
export NTPDATE_SERVER="10.28.0.42 10.28.0.45"

And when i run the ntpq -pn on this HP-UX NTP server it is showing the output as below, It is showing 2 Windows NTP server and 1 own local clock.
hostname:[/]# ntpq -pn
remote refid st t when poll reach delay offset disp
==============================================================================
10.28.0.42 10.28.0.46 4 u 315 1024 377 78.66 227.730 7958.19
#10.28.0.45 10.28.0.46 4 u 23 64 377 78.67 -4.808 12.18
127.127.1.1 127.127.1.1 10 l 9 64 377 0.00 0.000 10.01

And Configuration of the HP-UX as NTP Client.
/etc/ntp.conf
server 10.28.0.42 prefer #Windows NTP Server
server 10.28.0.45 #Windows NTP Server
server 10.33.12.15 #HP-UX NTP Server
server 10.33.12.8 #HP-UX NTP Server
driftfile /etc/ntp.drift


/etc/rc.config.d/netdaemons
export NTPDATE_SERVER="10.28.0.42 10.28.0.45"

And when i run the ntpq -pn on this HP-UX NTP Client it is showing the output as below, It is showing 2 Windows NTP server and 2
HP-UX NTP server.

hostname01:[/]# ntpq -pn
remote refid st t when poll reach delay offset disp
==============================================================================
10.28.0.42 10.28.0.46 4 u 483 1024 377 80.15 305.986 7968.87
#10.28.0.45 10.28.0.46 4 u 178 512 377 78.22 10.809 4.17
+10.33.12.15 10.28.0.45 5 u 218 512 377 0.75 16.907 11.96
+10.33.12.8 10.28.0.45 5 u 302 512 377 0.81 10.450 11.40


I have also check the syslog for the information and it is working good as when both Windows NTP server was unavailable it got sync with one of
HP-UX NTP server...And later when Windows NTP server became available it got sync with Windows NTP server.

I have one more question when i make some changes in the /etc/ntp.conf file on HP-UX NTP client the clock goes 9 min back and then it gets sync with Windows NTP server almost within 5-8min. Can i avoid this so that when i am making NTP client configuration changes on the Production System so that the clock won't go back?

Please let me know whether this is the good configuration or any changes required. So that i can implement it in Production.And also we have only 20 HP-UX servers on which i have to maintain the clock of all 20 HP-UX servers in sync.


Thanks,
Narendra

[mvpel]

In your latest example, you don't need the "nopeer" restrictions since you're not setting up a peer relationship between the two HP servers, only a client/server relationship.

However I think you'd be better off having the two HP systems as peers of one another, as that's a bit more resilient configuration - the machine with the best local clock becomes the server when the upstream servers are unavailable, rather than always and only the local clock you specify.

You asked: "When i make some changes in the /etc/ntp.conf file on HP-UX NTP client the clock goes 9 min back and then it gets sync with Windows NTP server almost within 5-8min."

There must be something very wrong if you're getting 9 minutes out of alignment so quickly. Even the worst-performing clock that NTP can handle with its loop (500ppm frequency offset) is only going to drift about 7 minutes per day, and that's worse than an antique wristwatch.

When NTP starts up, the "ntpdate" command is run to get immediate step synchronization with the best upstream time server - that's what the NTPDATE_SERVER variable is for. No matter how far off your local clock is, it will be stepped to match the upstream server. Once that's done, the local clock starts drifting from zero again, and shouldn't need to be stepped very far in future restarts.

But if you're just restarting an already-running NTP service to give it a new configuration, that initial step reset should never be more than a tiny fraction of a second, far less than the two-second limit of the SAP server.

So, I'm not sure I'm understanding you correctly. Could you post a cut and paste?

You said: "And also we have only 20 HP-UX servers on which i have to maintain the clock of all 20 HP-UX servers in sync. "

If these HP servers are all dispersed across the WAN as the other two are, you may not want to make them all peers of one another, since that would involve a a flurry of NTP packets flying around the WAN, as each host in the peer group has to do a client/server exchange with the other at least every 1024 seconds.

That would be a good use for the multicast setup, as all 20 servers would be able to get their time from a single NTP multicast packet, but short of that, you'd want to make a set of three HP servers a "peer" group, and have the other 17 HP servers use those three as "server" along with the two Windows systems.

However, considering that the Windows and network administrators out in New York have proven themselves inexperienced enough to have set up both the primary and backup domain controllers on the same IP subnet, and foolish enough to bring them both (or the subnet) down for maintenance at the same time, I don't think you should count on them to provide this mission-critical service for your SAP servers.

Consider spending the comparatively small amount of money (compared to the cost of an SAP outage) for a Spectracom NetClock 9389, or a Brandywine NTV-100RG, or something similar:

http://www.spectracomcorp.com/Products/NetworkTimeServers/9389NetClockNetworkTimeServer/tabid/1000/Default.aspx

http://www.brandywinecomm.com/products/network-time-servers/85-ntv-100rg-low-cost-gps-synchronized-ntp-server-with-front-panel-display

These are 1U rack-mounted GPS time server appliances. When you hook it up to the proper antenna system (they offer both outdoor and window-mounted GPS antennas - the NTV-100RG includes an antenna & 100' of cable), it becomes a Stratum-1 network time server which you can use as a time server along with the Windows servers in New York.

The NetClock and the NTV-100RG don't blue-screen, will never have a disk crash, never need reboots or maintenance outages, and don't require weekly backup tapes. They'll run for year after year after year without a peep.

If you get two of these - one in California and one in Texas - you can protect yourself from WAN outages as well.

Think of it as a cheap insurance policy.