Incorrect file size returned in JRuby (Java in general?) for file open for write

[John Gillings]

Ben,

   I don't know anything about JRuby, but in general, I like to open log files as SHARED WRITE. This means you can write to the file without worrying about flushes. When you want to check the file from another process, open the file for APPEND SHARED WRITE access and let RMS worry about synchronising with the writer, then close the file immediately. From DCL it looks like this:

$ OPEN/APPEND/SHARE=WRITE synch OUTPUT.TMP

$ CLOSE synch

The contents and EOF are now up to date, and can be accessed by readers up to the point of the most recent close. Repeat as necessary.

There should be similar constructs in most languages. 

There's a small cost associated with a shared write (but remember, most of the time, there are no sharing processes, so not much ongoing overhead), but you avoid the cost of synchronous writes for every output.

As I said, I don't know anything about JRuby, so I'm not sure of the semantics of your code. Are you reopening the file for every line of output? If you keep the file open and just write sequentuially to the channel (or whatever the file object is called), does it still overwrite the first record?

[BenAArmstrong]

The point is not to find the Ruby construct that would accomplish synchronized writing, but rather to improve the JRuby port for OpenVMS so that when sync is turned on, it "just works". A correct solution will not change what the application programmer has to write, but should be transparent, have the same semantics as on other platforms, and have no undesirable side effects. Here is the documented behaviour of sync=true:

 --------------------------------------------------------------- IO#sync=
     ios.sync = boolean   => boolean
------------------------------------------------------------------------
     Sets the ``sync mode'' to true or false. When sync mode is true,
     all output is immediately flushed to the underlying operating
     system and is not buffered internally. Returns the new state. See
     also IO#fsync.

        f = File.new("testfile")
        f.sync = true

     (produces no output)

So from what I can gather, I need to understand the Java source code handling synchronized writes, which I have still not fully grokked, as it is split across multiple layers in different Java classes and then modify it it to flush so that the position is correct before writing the appended material. It is doubtful if we will get much further in this thread until I can at least provide a working Java reproducer, distilled from the JRuby source, which shows the problem, but thanks for the responses so far.

Ben

[Ph Vouters]

Ben,

As a real C based Ruby has not been ported to OpenVMS and the only thing available is JRuby which is a Java based Ruby implementation, I do suggest you stay at Java level to not add more complexity to the problem. Build up the as simple as possible Java source from this code extract at http://www.exampledepot.com/egs/java.io/Sync.html. This pure Java code in this Web document clearly makes two distincts calls : os.flush() and fd.sync(). You should note it never mentions any specific operating system.

Also, and I have been confrontated to, on Unix/Linux you must distinguish between buffered on non buffered writes. On Unix/Linux all the difference lays between each time calling the C write function versus calling fwrite. The C write on Unix/Linux is a system call whereas fwrite is a C library function. One main difference though with your working environment, OpenVMS is not a Unix like operating system. So on OpenVMS do $ help CRTL [f]write to read what the CRTL actually claims.

As I noticed it buffered versus non buffered file writes makes all the performance difference on Unix/Linux. In consequence, my guess is that Java only calls the underneath C fwrite (buffered) and this would compell Java code to os.flush() followed by fd.sync(). You may test the Java code in the URL I provide in various conditions.

In this hope this will help clarify the debate.

Philippe

[BenAArmstrong]

Thanks, Philippe. Clearly I need to do more study on this, and I will certainly try out the test code.

[Ph Vouters]

Ben,

The way I understand these CRTL functions, fflush flushes out the CRTL file buffers to the underneath RMS buffers meanwhile fsync causes a flush of  the RMS buffers to the physical medium. Because { stat | lstat | fstat } as said to return an updated st_size field when flushing all buffers to disk, this would mean you need in any Java code (even more using Java/VMS) to use Java's os.flush() and fd.sync()

This is also likely the reason why Ruby and thus Jruby implement the fflush and fsync methods as I pointed out in a Ruby's URL I previously mentionned.

In the hope this clarifies your technical knowledge regarding your JRuby concern.

[Hoff]

This environment is using Unix tools and expecting the Unix file system model here, and not VMS.

Were it feasible, the easiest would be to entirely disable locking and the file system blade guards.

Given that is generally not feasible, try configuring DECC$FILE_SHARING for full (sharing) access.

See if this sharing allows your file processing to continue.

Given VMS made different choices deep within its I/O stack, and given VMS lacks foreign file system interfaces, it's difficult to completely and correctly emulate the Unix file system model on VMS; various compromises do exist here.