JVM Clock Drift, date / time not in synch across multiple JVMs

[Ben1970]

I am a developer for a real time java application (J2SE 1.3.1_10, jre/lib/IA64/server/libjvm.so) running on HP-UX 11i. The application has several components which each run under separate JVMs on a single server. When all components/JVMs are started at the same time they are all in synch with one another in regard to the value returned by System.currentTimeMillis(). However if one of the components has to be stopped and restarted at some later date the value returned by currentTimeMillis is no longer in synch with the other components. The value returned by currentTimeMillis in a JVM started later is always less than the value returned by the same call in a JVM started earlier.

For example, say we have 3 components A, B, and C, and that a message is passed first to A, then A passes to B, and then B passes to C, and that each component logs the value returned by currentTimeMillis() when the message is received. When all 3 components are started at the same time the following is always true regarding the logged time stamps: A <= B <= C

Now say a day or two passes and B needs to be shut down and restarted the following becomes true regarding the logged time stamps:
A > B < C

This is very bad in the context of our application as these timestamps are integral and must be in synch across the application.

We recently ported the application to HP-UX from Windows 2000 where we did not have this problem. It looks almost like the JVM is maintaining its own clock on HP-UX which is drifting at a consistent rate across all JVMs.

So I have two questions for the forum:
1) Does this make any sense at all? Can anyone explain to me what is happening?
2) How can I fix or work around this? Are there any custom java date/time libraries for 11i which get the time from the system clock?

Any advice is greatly appreciated.
-Ben

[A. Clay Stephenson · Accepted solution]

While this reference is to patches for 11.0; I have no idea which version EXACTLY of 11i that you are running; it does describe your problem.

http://h18012.www1.hp.com/java/patches/g-11.00-1.4-1100.0000.html

Search within this page for "currentTimeMillis" and you will find a list of patch. Go to the patch database and see if there are equivalent patches for whatever release of 11i (11.11?) that you are running.

[Steven E. Protter]

Shalom Ben,

Even if you configure java and patch the system to use the system clock, that will probably not solve the problem.

I accidently left work today about 20 minutes early and went to a langauge class. I sat there for 25 minutes and did my homework and posted to ITRC.

Why? Because I left the office for class based on an HP-9000 server clock and didn't check my cell phone. My cell phone gets its time from the Cellular network at Orange in Israel. That network gets its time from an NTP server.

HP-9000 system clocks are well known for gaining time and losing time. If yours keeps accurate time, you are very lucky.

I recommend working the software as recommended and connecting your HP-UX system to an ntp time source. The software is built into the OS.

The configuration depends on whether or not you have access to an accurate network time source.

http://www.ntp.org

SEP

[A. Clay Stephenson]

Actually, since this is all happening on one server, I don't expect NTP to be of much benefit. My experience has been is that yes, the system clock drifts, but it drifts in a constant direction -- as long as the temperature is held reasonably constant. It always a good idea to have an NTP timesource -- but the accuracy of NTP w/o extremely accurate and reliable timesources is on the order of a few ms. Since you are trying to measure ms, you are already at the limits of NTP and in your case, NTP -- especially if using Internet timesources -- might perturb your value more than they would help. Problems with system.currentTimeMillis are fairly well known in Java and are not restricted to HP-UX.

My solution to problems of this kind is to use shared memory to store a value. One dedicated process is responsible for updating this value and the clients just read. You could add semaphores if you are super-picky but that would generally not be necessary. Having a single process update the value would insure that the value always increases atomically -- and you could always add a little logic that always increases the value upon every read so that each read is assurred of getting a unique timestamp. The timestamp might not always be "right" but it would always be consistant. If you are having to coordinate values across multiple hosts then you have to use sockets and the overhead of client-server requests and latency destroys ms accuracy --- but again, the values are always guaranteed to be increasing and that id often more important than being "exactly" right all the time.

[Ben1970]

OK, I have finally figured out the problem, Clay, your reply put me on the scent. When looking at the patches you linked me to I noticed a reference to a JVM option I was not familiar with: -XX:+UseHighResolutionTimer.

Searching for that I found the following SDK 1.3 release notes: http://www.hp.com/products1/unix/java/itanium/infolibrary/sdk_rnotes_itanium1-3-0-02.html#date_time.

That lead me to the JVM option -XX:+UseGetTimeOfDay and searching for that I found the answer to my problem in an HP Java programmers guide at http://www.hp.com/products1/unix/java/infolibrary/prog_guide/hotspot.html#-XX:+UseGetTimeOfDay

As I suspected the JVM does maintain its own time based on the number of cpu ticks since the VM was started. And in this case NTP was part of the problem, when the server updates it's clock from NTP the JVM is not notified so at that point the JVM clock is out of synch with the OS clock. That is why JVMs started at the same time are all in synch with one another and JVMs started at a later date are out of synch.

So the answer is to use the JVM option -XX:+UseGetTimeOfDay

I have yet to validate this solution but I will update the group if this does not solve the problem.