Subversion Client for OpenVMS

[falk wiegand]

We tried to compile the Subversion client on OpenVMS but we failed because the Apache runtime library is not available on OpenVSM. Is'nt it? Or does anyone compiled it successfully? My customer wants to use Subversion because it's sources are so different types and the OpenVMS sources are about 25% of the overall projects that the major part is out of OpenVMS but an essential part of it.

[Jan van den Ende]

Falk,

Apache on VMS is a layered product.
Instal SWS (the VMS port of Apache) from the freeware CD or the HP VMS website, and you should be home.

PS What is your VMS version? For older versions there is no SWS, but you should be able to get OSU or WASD.

hth

Proost.

Have one on me.

jpe

[Jansen_8]

You probably also need the sources (at least for the .h files) You can download them from HP.

Jouk

P.S. I'm very interested in a SVN client for VMS myself. Porting it is on my wish-list.

[Richard Brodie_1]

Jan,

The Apache portable runtime is an OS
abstraction layer developed by the Apache web server project (covering things like memory allocation, threading etc.). It is only loosely related to Apache itself; and an alternative web server, not at all. Subversion is a CVS (or CMS) like source code management system.

I seem to recall a kit around for Apache 1.0 but that would be too low an APR version. Digging the current one out of the Apache sources and getting it to build sounds like a largeish project. Having a properly kitted APR library would be useful; in the meantime, I think the practical thing to do would be to do something like put the working copy on some flavour of Unix and NFS mount it on VMS.

[Craig A Berry]

FWIW, the Apache for VMS sources can be downloaded at:

http://h71000.www7.hp.com/openvms/products/ips/apache/csws_source.html

The APR must be in there somewhere. Whether Subversion would exercise all the same bits as Apache I don't know.

I've often thought a Subversion client would be a nice thing to have on VMS. A first cut would probably have to assume stream files. Even nicer would be a way to store RMS attributes as metadata so you could deposit any type of VMS file. All of this probably requires real work.

[Patrick Traill]

Here is another user very interested in a Subversion client.
It could be so much more reliable and efficient than what we now do with FTP.

On the other hand, an advantage of going via a PC is having Tortoise to check the Commit.

I recently ported a pair of CMS libraries (including history and classes) to Subversion,
and considered trying the apache port,
but eventually decided to write a client on VMS to push the stuff via a server on a PC into Subversion.

The RMS attributes should be easy enough to encode with Subversion attributes.
These should be symbolic (no numeric codes) as in Set File /Attributes,
and in a namespace such as "svn-vms:".

Who knows which version of the APR Subversion requires, and what has currently been ported?

[labadie_1]

Hello

Have you thought about using something else than subversion ?

Thierry Uso has ported Superversion to Vms
http://perso.orange.fr/thierry.uso/suv.html#suv-descriptif

Hope that helps.

Gerard

[Craig A Berry]

Patrick Traill said:

=======
The RMS attributes should be easy enough to encode with Subversion attributes.
These should be symbolic (no numeric codes) as in Set File /Attributes,
and in a namespace such as "svn-vms:".

Who knows which version of the APR Subversion requires, and what has currently been ported?
========

It would not just be a matter of encoding the attributes from the file header. For example, with a VFC file, you need another 2-byte "header" for each record in the file. Even if you successfully read in and store the control bytes when checking something in, you would need a way to recognize record boundaries when checking it back out. No doubt this type of problem has been solved many times before, such as in the STRUVMS support in the Process Software FTP utilities. But as I said before, it would be real work to implement this in Subversion, though well worth doing if someone has the time.

What's been ported would be whatever version of the APR is in Apache 2.0.

[Steven Schweda]

> It would not just be a matter of encoding
> the attributes from the file header.

Yes, it would.

> For example, with a VFC file, you need
> another 2-byte "header" for each record in
> the file. [...]

No, those bytes are in the data in the file.
They only need to be interpreted properly,
which will be done if the file attributes are
restored along with the data.

To see the "record boundaries" yourself,
compare the output from "DUMP /RECORDS" with
that from plain "DUMP" on a suitable file.
The only requirement is that the raw file
data be stored and retrieved, rather than
letting anyone try to interpret the records.

> No doubt this type of problem has been
> solved many times before, [...]

Info-ZIP Zip and UnZip, for example.

[Craig A Berry]

Well, maybe, Steven. But if it's sending you a set of differences rather than an entire file, I think it would be pretty tough to know when the VFC bytes are control bytes and when they are just data. Which is why I'm hoping you'll do the port so I won't have to :-).

[Steven Schweda]

> if it's sending you a set of differences [...]

It may be _storing_ a set of differences, but
I'd expect it to be sending you a
reconstructed file, with all the bytes
reconstructed properly. Assuming that the
critical file attributes do not change, or
that the changes are also maintained, what
could go wrong?

> I think it would be pretty tough to know
> when the VFC bytes are control bytes and
> when they are just data.

No, it's easy. The first two bytes are the
first byte count. After that, just count the
bytes to find the next byte count. And so
on. (It might be hard if you didn't start at
the beginning, so don't do that.)

> Which is why I'm hoping you'll do the port
> so I won't have to :-).

It might be more educational for you. And
I'm currently engaged in a tedious effort to
push some VMS changes into the next cdrtools
(cdrecord and friends) release.

[Craig A Berry · Accepted solution]

>It may be _storing_ a set of differences, but
>I'd expect it to be sending you a
>reconstructed file, with all the bytes
>reconstructed properly.

Subversion does not conform to that expectation. It sends differences in both directions over the network.

>> I think it would be pretty tough to know
>> when the VFC bytes are control bytes and
>> when they are just data.

>No, it's easy. The first two bytes are the
>first byte count. After that, just count the
>bytes to find the next byte count. And so
>on. (It might be hard if you didn't start at
>the beginning, so don't do that.)

I have not yet looked into the details of what Subversion sends across the network, but more than likely you get a byte offset and a chunk of data. If the byte offset puts you in the middle of a record in a record-oriented file with variable-length records, things could get interesting.

[Steven Schweda]

> Subversion does not conform to that
> expectation. [...]

None of that matters. Any file on VMS
comprises a stream of bytes, and meta-data
which explain how to interpert those bytes.

Any system which can correctly reproduce the
byte stream and the meta-data can do the job.

I don't know if Subversion can be persuaded
to do the job without adding features, but
the way that the VMS file system interprets
the byte stream is of no consequence to the
actual byte stream. Bytes are bytes. Look
again at that DUMP output. One byte looks
about the same as the next one. Records (and
their byte counts) exist only in your
interpretation on those bytes, not in the
bytes themselves.

[Patrick Traill]

I admit I had not been thinking about VFC or for that matter indexed-sequential files. I was principally concerned with storing platform-independent source-code.

In general, how VMS files should be represented in depends on what you want to do with them, and especially on whether you want to use them on non-VMS platforms.

One needs to remember that a VMS file (as presented by RMS) is conceptually a stream of records, with optional redundant(?) indices. There are at least three views of a VMS file: programmatic, presentation and storage.

1) The view an application programme has of a file (as presented by RMS or some programming language) is a stream of records. These may be of fixed or variable length and contain formatting (VFC) information, which is not regarded as part of the application data (anything else?). There may also be the ability to retrieve records based on content, using indices maintained by RMS, which may be reconstructed from the record stream and index specifications. The record stream view is sufficient(?) to reconstruct the entire file, provided we include the file header information and index specifications.

2) The view a user has of a readable file presented by commands such as TYPE is of a stream of bytes. This is not in general sufficient to reconstruct the record stream, since any byte used as a delimiter could equally well occur in a record.

3) The view that the RMS routines have of a file is of the raw data as stored in the On-Disk Structure via the QIO interface to the ACP. This may be regarded as a random access array of bytes or bits or indeed as a stream of bytes.

To store a file in Subversion, it must be represented as a byte stream, which is trivial for the presentation and storage view. In both these cases, the only Subversion attributes required would be for the file-header information. For the record-stream view, however, one needs to add record lengths and formatting information to the byte stream. The most flexible format would be a variable-length record header. One would also need to add the index specifications to the Subversion attributes.

For many text files, especially programme source, the presentation view will be adequate, and the ambiguity about the record stream acceptable. Such files are also easily processed on other platforms.

To process a record stream on a non-VMS platform would need special purpose software. Although not terribly complicated, it would have to be supplied specially.

To process the raw view of a sequential or direct-access file is also manageable, but it is forbiddingly complicated for indexed-sequential files, even leaving aside possible versioning issues.

On the other hand, storing the record stream of an indexed file makes reconstructing it much costlier, as the indices need reconstructing.

[Selden Ball]

Falk,

Like others, I'm very interested in a subversion client for OpenVMS, in our case on an Alpha running OpenVMS v7.2-3.

Have you had any luck building it?

[Mats Backlund]

Hi,

I have tried the Java variant of subversion called SVNKit on OpenVMS which seems to be an easier way to go rather than trying to compile a native subversion client. I have JDK 1.4.2-4 installed on the VMS and downloaded the org.tmatesoft.svn_1.1.0.standalone.zip file found on http://svnkit.com/download/index.php
The biggest challenge with the SVNKit is to set all logicals for Java so that UNIX-style filenames and similar things work correctly when running SVNKit on OpenVMS. I have succeeded to add files to a repository on Windows and also updated my working folder on VMS from Windows. The command line syntax for SVNKit is the same as when running the standard command line client (svn up, svn co http://xxxx, svn cleanup, etc).

[Ian Miller.]

If you could write some notes on what you did I think serveral people would be interested in how you got it to work.

[Patrick Traill]

Dear Mats,

This sounds encouraging.

If you manage to write it up as requested, could you perhaps also mention what view of the VMS file system it gives you.

I presume it will be suitable for source code, but clarity on this point would be useful.
(See my reply of Oct 9, 2006 09:50:46 GMT for come of the issues).

[ptrskg]

Hi Falk and others,

I have during the week worked with SNVKit, a Java implemetation of the SVN client, on OpenVMS with some success and also sent patches to TMate software that hopefully will be included in the next version of this package. (http://svnkit.com)

I haven't tested everything but I have it working with remote servers, svn- and http-protocol, and also with a local repository. When I tried https it failed and I think it is related to Java/VMS stuff.
I am running 8.2-1 on an Integrity with Java 5.0-1 and will soon try it on my Alpha.

I think this can be something to use and as soon as we have a stable environment we will move our code from CVS to Subversion. The server will be on Linux...

Peter

[Heinz Huber]

Using SVNKit:
What logicals did you set?
I'm having problems with filenames containing more than one dot (.). All dots except the last one are converted to underscores (_).

My current settings (worked for CVS):
$ set proc/parse=extended
$ define/job decc$efs_charset enable
$ define/job decc$efs_case_preserve enable

[Larry Bohan]

just to throw this out there, a very small, bare-minimum,
working example from Alpha VMS 8.3 (all patches/eco's, ods5 working device),
using Java 1.5.0 against a linux SVN server.

Caveat: i haven't tried much more than 'svn list' so far.

I started getting SVNkit going under a GNV (2.1-2) bash shell,
then worked my way back to getting SVNkit working from
a simple VMS (dcl) login.

I haven't worked with Java very much on VMS, or elsewhere.

I was able to get this minimal example to fly, with almost
no special JAVA$ logicals. in the real world, I'd expect
that wouldn't be the case. especially wrt. adds/commits
and/or various merges/diffs.


$ @jsvn2.com "ls" "svn://someserver/someproject/branches/qa/"
Authentication realm: <> AcmeCorp
Username:
Password for '':
v00_003_0006/
v00_003_006/
v01_000_0013/
v01_000_0014/
v01_000_0015/

$ sh log/job decc$*

(LNM$JOB_864A5140)

"DECC$ARGV_PARSE_STYLE" = "ENABLE"
"DECC$EFS_CASE_PRESERVE" = "ENABLE"
"DECC$EFS_CHARSET" = "ENABLE"
"DECC$PIPE_BUFFER_SIZE" = "32767" ! NEEDED?
"DECC$POSIX_SEEK_STREAM_FILE" = "ENABLE"

$ pipe sh log/job java$* | sear/mat=nor sys$pipe sys$comm

(LNM$JOB_864A5140)

"JAVA$FILENAME_CONTROLS" = "-1"

$ sh proc/all
...etc...
Parse Style: Extended
Case Lookup: Blind
...etc...

$ type jsvn2.com

$ set noon
$ CP = "/toolsdisk/svnkit-1_1_6_3855/svnkit.jar"
$ CP = CP + ":/toolsdisk/svnkit-1_1_6_3855/svnkit-cli.jar"
$ CP = CP + ":/toolsdisk/svnkit-1_1_6_3855/trilead.jar"
$ CP = CP + ":/toolsdisk/svnkit-1_1_6_3855/jna.jar:."
$ OPT = "-Djava.util.logging.config.file=/toolsdisk/svnkit-1_1_6_3855/logging.properties"
$ define/nolog/user sys$input sys$command
$ java "''OPT'" -cp "''CP'" "org.tmatesoft.svn.cli.SVN" "''P1'" "''P2'"

[Ben Armstrong]

I can confirm that DECC$PIPE_BUFFER_SIZE = 32767 is needed. Before I defined this, an SVN diff would give me a stack dump in java$java_shr UNIXPROCESS_MD remapPath (I will not clutter the thread with the whole traceback, but can post it if anyone wants to investigate further).

But I have a more fundamental problem. Without privs turned on, jsvn creates read-only files when I do a checkout which then cannot be renamed, e.g.

$ jsvn co svn://ntts/dyserver
svn: Error writing entries file for '/DKC200/DYMAX/BG/./dyserver'
svn: Cannot rename file '/DKC200/DYMAX/BG/./dyserver/_svn/tmp/entries' to '/DKC200/DYMAX/BG/./dyserver/_svn/entries'

This would appear to be a difference in assumed Unix filesystem semantics from VMS. In Unix you ought to be able to set a file read-only and then rename it. It is the permissions on the *directory* that govern whether or not you can rename files. Not so on VMS. If you set a file read-only, it cannot be renamed. I don't know how to fix this. Obviously endowing all developers with system privileges so that they can rename their own read-only files is not a solution. Can someone please help?

Thanks,
Ben

[Ben Armstrong]

I have further confirmed basic Subversion client functionality working with OpenVMS V8.3 Alpha, Java 1.5.0-3 and SvnKit 1.1.7 (released 11 Apr 2008, 19:07! lucky timing on my part, I guess).

I have tried:
jsvn import
jsvn checkout
jsvn add
jsvn commit
jsvn update
jsvn diff

Also: checkout twice and commit a conflicting change

jsvn update ! reports C & adds merge markers
resolve the conflict
jsvn resolved
jsvn commit

[Larry Bohan]

I was (seemingly) able to work past the check-out read-only problem using the following DECC$ feature logicals. (below)

it was/is of interest to me, to find out
what decc$, and java$ feature logicals
folks have found to useful/helpful.
(John Malmberg has a conference (PORTING_TO_VMS) on the encompasserve Notes server, that may be of interest)

Are you using SVNkit primarily from DCL
or from the GNV bash shell? (just curious)

See also:
http://www.openvms.compaq.com/doc/83final/5763/5763pro_005.html
(the formatting is screwy, when usingFireFox.
slightly better when using the IE-Tab add-on)

DECC$UMASK

DECC$UMASK specifies the default value for the permission maskumask. By default, a parent C program sets theumask from the RMS default permissions for the process. A child process inherits the parent's value forumask.

To enter the value as an octal value, add the leading zero; otherwise,it is translated as a decimal value. For example:

$ DEFINE DECC$UMASK 026

DECC$FILE_PERMISSION_UNIX

With DECC$FILE_PERMISSION_UNIX enabled, the file permissions for newfiles and directories are set according to the file creation mode andumask. This includes mode 0777. When an earlier version of the file exists,the file permissions for the new file are inherited from the earlierversion. This mode sets DELETE permission for a new directory whenWRITE permission is enabled.

With DECC$FILE_PERMISSION_UNIX disabled, modes 0 and 0777 indicateusing RMS default protection or protection from the previous version ofthe file. Permissions for new directories also follow OpenVMS rules,including disabling DELETE permissions.

[Ben Armstrong]

I'm using SVNkit from DCL.

Even with a very permissive DECC$UMASK (000) and DECC$FILE_PERMISSION_UNIX enabled, I still get the same error. I guess regardless of my umask, jsvn wants to set the file permissions read-only for the entries file. Since write permission isn't specified, implied delete isn't relevant. Nor is inheriting permissions from an earlier version of the file relevant since in a checkout, all files are new.

So while I appreciate the tips, I still don't have a solution yet from the information you provided. I will have a look at the doc you referred to, though, and see if I can figure it out.

Thanks,
Ben