@paulgear wrote:
I know this is probably not helpful to your situation, but if you have a data centre full of machines, you probably should have two other important things:
- A standard installation process which includes the necessary notes about using NTPv3 only.
- A configuration management system like Puppet, Chef, or cfengine, which allows you to make bulk changes to configurations like this one.
So just to set the record straight on this... We use a multi-phased approach to build, launch, deploy, etc machines. When you are dealing with numbers from 1000-20,000 machines, you have a process. Since deployments are automated, there are LOTS of things that are thought of, dealt with, etc that service the basic functions of these machines. Where do logs go, how are disks of varying sizes partitioned and deployed, what auditing and automated reporting systems are in place, etc.
When building a machine from "raw metal", you have many steps to overcome. One of the most basic are getting time correct in your overall infrastructure.
For us, we start from raw metal booting the iLO of our machines off of DHCP servers. This of course uses the SNTP function of iLO as distributed by DHCP. Some interesting facts however... The iLO system uses GMT as its base system and does not necessarly have the "brains" to deal with DHCP 112 to allow for the changing of timezones. Weither you knew this or not, GMT is *not* the same as UTC or (more currently) Etc/UTC timezones. Modern OSs have to deal with ever changing politics of time and whatnot, but somehow time has to be universal.
Fortunately SNTP is v3 based. So we get the time. iLO then puts the time in the RTC for you (nothing you can do about it) during a reboot of the machine (a bug that was reported and later fixed. They now put it in the RTC upon physical power cycle, not on reboots). Why is this distinction important?!? Because it turns out the default European/London time zone is 3600 seconds off for half the year.
When iLO populates the RTC, half the year it puts the wrong time in for Etc/UTC. When the machine boots, the first thing that happens (time wise) is that ntpd starts to run, however half the year it recognizes a time slew of 3600 seconds.
Unfortunately, before ntpd runs AND syncs with its stratum servers, puppet starts... and of course the time checks are 3600 seconds off, so puppet refuses to work with the puppet master.... and the entire process falls apart.
So time is a HUGE issue. Network devices in a data center are considered (on scale) your backbone of operations and relatively immutable. When you have 200+ access switches, multiple cores, etc, you want a framework that is synced across the board, etc.
So the real issue is that on scale, all the intricate parts effect each other in ways not always understood. In case you hadn't guessed, we have spent a HUGE amount of time (and consequently man hours * $150/hr) trying to understand the problem and how to deal with it. Easily enough money spent to buy a few core switches. Or understanding of the problem is much better (and better documented throughout the organization) but its the frustration of "going backwards" on an area we thought was done that prompted the above posting.
Don't get me wrong, we're using HP switches now throughout our backbone system (for reasons beyond the fact that they cost a quarter of cisco gear) but that doesn't change the $$$ spent in understanding what we lost in transition.
My point above was not to necessarly complain, but to help document what we found and our frustrations. Also to maybe (though a forum) get someone at HP/H3C to notice and think maybe someone out there does actually care about NTP4 as an industry standard, not as something to "add on later".
Marcos
