Java output to VFC file has vacuous records - excessive fflushes?

[BenAArmstrong]

Is there any way to prevent Java from writing additional vacuous records to a VFC file (probably one per fflush) after every print() or println()?

Consider this code sample:

import java.lang.*;

class twochar
{
   public static void main(String[] args) {
      System.out.print("A");
      System.out.print("B");
      System.out.println("Hello");
      System.out.println("World");
   }
}

 What would you expect the output to look like? I would think something like this:

ABHello
World

And that is, more or less, what you'll see ... that is, until you try to use something other than "TYPE" to view the output, if it was output to a VFC file. Take EDIT/TPU, for example. Instead, you'll see:

A
B
Hello

World

 It would appear that a record is forced out with no newline flag each time we print() or println(). Here's some dump/rec output to see the structure of the resulting file:

Record number 133 (00000085), 1 (0001) byte, RFA(0007,0000,00E2)

                                           41 A............................... 000000

Record number 134 (00000086), 1 (0001) byte, RFA(0007,0000,00E8)

                                           42 B............................... 000000

Record number 135 (00000087), 5 (0005) bytes, RFA(0007,0000,00EE)

                                  6F 6C6C6548 Hello........................... 000000

Record number 136 (00000088), 0 (0000) bytes, RFA(0007,0000,00F8)


Record number 137 (00000089), 5 (0005) bytes, RFA(0007,0000,00FC)

                                  64 6C726F57 World........................... 000000

Record number 138 (0000008A), 0 (0000) bytes, RFA(0007,0000,0106)

Basically, any utility that doesn't understand VFC files will have this problem, including DIFF, and that's a big problem for us, as we have logs of output generated in batch by non-Java applications and logs of output that are supposed to look the same generated by Java ports of those applications. They're a big mess when you DIFF them due to all the spurious newlines.

Now, we've written a script to do the cleanup of the VFC files to drop the vacuous records so that they can be compared cleanly, but that's only covers certain use cases. Ideally, if we could tell the Java app not to behave that way, then we wouldn't have to clean up afterwards, and we'd be much happier. Any ideas?

Ben

[H.Becker · Accepted solution]

>>> What would you expect the output to look like?

ABHello

World

It seems Java is using the CRTL. So I would try some "CRTL Feature_Logical_Names". On my system with Java 1.3.1 and your example I can get around this with:

$ def DECC$STDIO_CTX_EOL 1
$ def sys$output x.log
$ java "twochar
$ deass sys$output
$ dump /rec x.log
Dump of file SYS$SYSDEVICE:[HARTMUT]X.LOG;1 on 15-SEP-2011 19:12:57.98
File ID (301,98,0)   End of file block 1 / Allocated 16

Record number 1 (00000001), 7 (0007) bytes, RFA(0001,0000,0000)

                     6F6C6C 65484241 ABHello......... 000000

Record number 2 (00000002), 5 (0005) bytes, RFA(0001,0000,000C)

                         64 6C726F57 World........... 000000
$ 

[John Gillings]

culture clash! I'd also expect

ABHello

World

It looks to me like the Java may be correct. I'd guess that the VFC headers for the first 3 records specify no carriage control or linefeed, and the (apparently) extra one is the real linefeed between Hello and World. Try dumping the file and look at the VFC information.

I'm wondering why you'd want to generate VFC from Java anyway? Wouldn't this be better as stream_lf? VFC is great for record oriented languages like COBOL and Fortran, but it's a very poor match for the stream oriented output of Java & C (as evidenced by this post). VFC is also very OpenVMS specific (indeed, it's arguably OpenVMS COBOL specific, as it's designed to implement the archaic COBOL "BEFORE/AFTER ADVANCING LINES/PAGE" type clauses). It will always need some kind of conversion to be intelligible on any other system, furthermore, said conversion usually gets some aspect of interpreting VFCheaders wrong, especially for the more esoteric features. No doubt the bone headed decision to make VFC the default output format for DCL, which we're now stuck with, is another Cutlerism :-(

Note that you can control the attributes of files directed to SYS$OUTPUT. Try this:

$ CREATE/FDL=SYS$INPUT SOMEOUTPUT.TXT
$ DECK
RECORD
  FORMAT STREAM_LF
$ EOD
$ OPEN/APPEND MYOUTPUT SOMEOUTPUT.TXT
$   PIPE somecommand > MYOUTPUT
$   SHOW SYSTEM/OUTPUT=MYOUTPUT
$   RUN myprogram
$ CLOSE MYOUTPUT

If you're V8 or higher, the CREATE can be done inline:

$ CREATE/FDL="RECORD ; FORMAT STREAM_LF" SOMEOUTPUT.TXT

(aside: For long running processes,you can also make the file externally readable by changing the OPEN to

$ OPEN/APPEND/SHARE=WRITE MYOUTPUT SOMEOUTPUT.TXT

To read the file from another process, you can reset the EOF to the current position by opening and closing the file:

$ OPEN/APPEND/SHARE=WRITE FIXEOF SOMEOUTPUT.TXT
$ CLOSE FIXEOF
$
$ TYPE/TAIL SOMEOUTPUT.TXT

[BenAArmstrong]

Oh, that is gold! So, using a bit of JNA and http://labs.hoffmanlabs.com/node/1513#comment-2387 I presume I could set this feature just for the Java app in question (JRuby).

[BenAArmstrong]

It looks to me like the Java may be correct. I'd guess that the VFC headers for the first 3 records specify no carriage control or linefeed, and the (apparently) extra one is the real linefeed between Hello and World. Try dumping the file and look at the VFC information.

---

Yeah, did that. That's how we wrote the filter.

I'm wondering why you'd want to generate VFC from Java anyway? Wouldn't this be better as stream_lf? VFC is great for record oriented languages like COBOL and Fortran, but it's a very poor match for the stream oriented output of Java & C (as evidenced by this post). VFC is also very OpenVMS specific (indeed, it's arguably OpenVMS COBOL specific, as it's designed to implement the archaic COBOL "BEFORE/AFTER ADVANCING LINES/PAGE" type clauses). It will always need some kind of conversion to be intelligible on any other system, furthermore, said conversion usually gets some aspect of interpreting VFCheaders wrong, especially for the more esoteric features. No doubt the bone headed decision to make VFC the default output format for DCL, which we're now stuck with, is another Cutlerism :-(

Note that you can control the attributes of files directed to SYS$OUTPUT. Try this:

$ CREATE/FDL=SYS$INPUT SOMEOUTPUT.TXT
$ DECK
RECORD
  FORMAT STREAM_LF
$ EOD
$ OPEN/APPEND MYOUTPUT SOMEOUTPUT.TXT
$   PIPE somecommand > MYOUTPUT
$   SHOW SYSTEM/OUTPUT=MYOUTPUT
$   RUN myprogram
$ CLOSE MYOUTPUT

If you're V8 or higher, the CREATE can be done inline:

$ CREATE/FDL="RECORD ; FORMAT STREAM_LF" SOMEOUTPUT.TXT

(aside: For long running processes,you can also make the file externally readable by changing the OPEN to

$ OPEN/APPEND/SHARE=WRITE MYOUTPUT SOMEOUTPUT.TXT

To read the file from another process, you can reset the EOF to the current position by opening and closing the file:

$ OPEN/APPEND/SHARE=WRITE FIXEOF SOMEOUTPUT.TXT
$ CLOSE FIXEOF
$
$ TYPE/TAIL SOMEOUTPUT.TXT

---

Although that's a nice trick for isolated cases, that adds a hell of a lot of complexity in our environment, where the JRuby apps are only one small part of larger procedures (too numerous to enumerate here) run in batch. How would I go about convincing the batch symbiont to output to a Stream-LF file instead? I don't think I want to go around patching every blessed procedure that happens to use JRuby in it to divert output as shown above.

In any case, enabling decc$stdio_ctx_eol does exactly what I want. Now to figure out if it's OK to just define this everywhere (as in the login.com of the accounts that run these procedures) or if it would be better to try to embed the setting in JRuby itself.

Thanks to both of you for your excellent answers.

Ben

[BenAArmstrong]

A quick bit of JNA to fix this:

import com.sun.jna.Library;
import com.sun.jna.Native;

public class LibC {
    public interface CLibrary extends Library {
        CLibrary INSTANCE = (CLibrary) Native.loadLibrary("DECC$SHR.EXE",CLibrary.class);
        int decc$feature_get_index(String name);
        int decc$feature_set_value(int index, int mode, int value);
    }
    
    public static int feature_set_value(String name, int value) {
        return CLibrary.INSTANCE.decc$feature_set_value(CLibrary.INSTANCE.decc$feature_get_index(name),0,value);
    }
}

 And the ruby that calls it:

require 'libc.jar'
LibC.feature_set_value('DECC$STDIO_CTX_EOL',1)

 Works beautifully.

[Hoff]

You're not going to convince VMS to move off of its VFC file creation defaults for most RMS-level operations, while convincing C and Java to play well with files that aren't StreamLF can be a hassle.

The C and Java feature logical names can be hassle to establish (though certainly not intractable), and more subtly because they can sometimes perturb something else within a "hybrid" application and usually with unexpected results.  This whole logical name implementation is a well-documented and non-modular hack, IMNSHO.

One of the simplest approaches for reprocessing a VMS batch log file would probably be to aim a CONVERT at the log file before any Java native access, or invoke a callable convert conv$conv() requiest via JNI, and then use Java and its native I/O to access your newly-StreamLF log file.  Yes, this approach is (also) somewhat of a hassle, but one that you have complete control over without adding non-modular dependencies on the C and Java feature logical names.

If this were a typical Unix stack, you would probably pipe the stuff through col or sed and be done with it.