Pre-requisite for online time adjustment

[Peter tk]

Hello,

Customer’s environment involving clusters and standalone Alpha Boxes (totally 10 systems), they need to change the system time (backward for a month or more) for their application testing and the time synchronization’s accuracy up to millisecond between systems/clusters, so they use $ set time/cluster to accomplish this task.

To minimize the impact caused by the $ set time (online), they used to stop their applications and dismount the disk(s) before performing a system/cluster reboot.

They would also like to know if any service(s)/system core components to be stopped prior to the $ set time, for example Queue Manager, software like RTR, Disk File Optimizer, ABS… to ensure maximum security.

Any feedback would be appreciated.

[Jon Pinkley]

If I understand, they are setting their clocks to some time (1 month) in the past.

Note that there are going to be many file timestamps that will then be in the future, and things like analyze disk (verify) will complain, and if you let it, it will modify dates. That may or may not be what they want. If you don't change the timestamps, other thing like incremental backups may not work as expected.

I would recommend using something other than set time/cluster (for instance NTP or DTSS). Set time/cluster will set all the clocks to the same value, but some clocks may be set forward and some backward, i.e. it won't be a smooth transition. If you use NTP you won't be able to use an external source with the correct time, so you will need a local NTP source that you can control.

1 ms synchronization accuracy may be hard to achieve... some alpha's hardware clock interrupt frequency is less than 1000 per second (ES47 at 974 Hz for example).

See http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1214213 for more info.

If they are going to shutdown the cluster, then just boot the first system with a conversational boot, set the settime param to 1, and the writesysparam 0 (in that order) then continue. Then boot other systems and they will get their time from the first member. But without something to actively keep them synchronized, they will soon drift apart.

Jon

[Peter tk]

Hi Jon,

Thanks for your advise, they do setting their clock (at least one month) in the past, and they don't have the NTS or DTSS enabled. They did it because they need to test their application, and they changed it more than once per day if necessary.

We had already provide them a more stable and secure method to change their time by
SYSGEN> set settime 1, and enter the new desired time/date then let the system to boot. This approach ensure all the log/audit/..... files initialized/reset from the boot time and less risky. However, user want to automate those steps by just $ set time/cluster and then perform reboot. Due to the huge time difference by $ set time/cluster interactively, will there be any possible risk during shutdown and reboot?

Thanks.

[labadie_1]

You have a tool call datesim to "play" with the time
http://www.sp32.com/

and this thread on that subject
http://forums12.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1209110374457+28353475&threadId=434328

[The Brit · Accepted solution]

I believe that the SYSMAN "Config Set Time" command (when the environment is defined to be "/Clust") will give a more accurate "point-in-time" synchronization of your cluster nodes than the DCL "Set Time/Cluster". You could probably include this in your startup procedures. (simply create a file in a known location which contains the time you want to set on reboot, and let the "systartup.com" read it, and us it to set the time.)

Synchronization can be maintained within the cluster by enabling NTP on all nodes, and designating one of the nodes as the NTP time server (in SYS$SPECIFIC:[TCPIP$NTP]TCPIP$NTP.CONF on all cluster nodes.) Add a line similar to

Server ..com

or

Server

(Of course, now you only have to set the time on the "Server" node, and the others will synch with it.

Dave

[Hoff]

Why not use NTP?

Run your own NTP stratum and your own NTP servers for the purposes of this test, on your own subnet.

Have your own time provider and your own timebase provide the right? wrong? time for your clients.

I'd think that doing this any other way is too much like real work.

Do reboot the boxes as part of setting the time back, and expect to see some few errors and some warnings if the disks have future dates encoded on them -- details around the old Y2K-vintage testing will be of interest here.

[John Gillings]

tk,

To do a reliable test from a point of time in the (significant) past, I'd strongly recommend working from disk images taken at the target time.

The issue are the numerous "future" timestamps in log files, and in file headers which will almost certainly affect the behaviour of any software which cares about time.

For speed, use physical disks for the images. Maybe keep yourself 3 or 4 sets of disks, a month or two apart. Keep track of the most recent date executed against each set. Don't let them run against a clock in their future. Keep your application on a different drive. To run a test, shutdown, set the time on the console and boot from the appropriate system disk, and mount the appropriate data disks.

This type of scheme will take significant management effort (as you'll have to independently patch and update each environment, with the time set to the right place). However, you'll end up with test results that can be trusted, and as a bonus, you'll have excellent recovery from any disk failures!

[Peter tk]

Dear all,

Thanks for your great support.

Actually customer neither using NTP nor DTSS. What customer has experienced the queue manager problem after the system time has been changed ($ set time/cluster for online time adjustment which is not recommended) and eventually resolved by a Cluster reboot. Therefore, I would like to know (or pin point) how sensitve of the time shifted to the queue manager (or job entry, queue file); so that I can explain to them.

Much thanks,
tk

[John Gillings]

tk,

> What customer has experienced the queue
>manager problem after the system time has
>been changed

What was the problem, which direction was the time changed and by how much?

This is probably not how the queue manager works, but consider a potential, valid implementation for the queue manager.

At a particular time, it releases a job, then looks at the next event - say the next job to be released. It puts itself to sleep at that time, but may be woken if there are any other changes to queue event times.

Now you set the time backwards by a long way. What should the queue manager do? If you schedule a job in what the queue manager thinks is a past time, should it release the job immediately, or should it reset its time frame according to the clock? Should it sample the clock constantly just in case time goes backwards? Should the majority of customers suffer the performance consequences for the benefit of extremely rare events, such as a clock being set backwards by more than 1 hour?

Are your test results going to be valid when most software is "entitled" to assume that time doesn't ever (or at least very rarely) goes backwards?

>like to know (or pin point) how sensitve
>of the time shifted to the queue manager
>(or job entry, queue file); so that I can
>explain to them.

Without looking into the source code, you won't find a good answer, but I wouldn't expect the queue manager to be at all happy about a large backward timeshift. I'd want a reboot or, at the very least, a stop and restart of the queue manager.