topic Re: GNV and CRTL in Operating System - OpenVMS

GNV and CRTL

Martin Vorlaender — Mon, 22 Nov 2010 08:33:00 GMT

While building GNV from scratch I stumbled upon a CRTL behavior that I can't understand.

Environment: OpenVMS I64 V8.3-1H1 + VMS831H1I_UPDATE V8.0, HP C V7.3-019.

After fiddling a bit with the GNV sources, I took a GNU diff against the sourceforge source files and started over. To keep the SF source tree clean, I defined a GNV_ROOT consisting of GNV_SPECIFIC (my build directory) and GNV_COMMON (the SF source tree), and duplicated GNV_COMMON's directory structure in GNV_SPECIFIC. Then I SET DEFAULT GNV_ROOT:[000000], SET PROCESS/PARSE=EXTENDED/CASE=SENSITIVE, and called GNU patch with my diff file to carry over the changes to the new directory structure. Everything's fine for files like crtlsup/makefile.com or build/FEATURES.COM, but for file/file-4.09/src/compress.c patch says it can't find the input file. I traced this to a failing stat call (PCH\intuit_diff_type\%LINE 9402, errno=2 = ENOENT - No such file or directory). After some experimentation I found that to support unix paths that contain directory names with dots I have to define DECC$EFS_CHARSET. But the compress.c patch still fails under that...

Any other ideas why stat can't find this file?

TIA
Martin

Re: GNV and CRTL

Martin Vorlaender — Mon, 22 Nov 2010 09:48:29 GMT

Another data point: when I SET DEFAULT GNV_ROOT:[file.file-4^.09.src], and call GNU patch there (stripping 4 path elements), it works!

I don't get it...

Re: GNV and CRTL

Craig A Berry — Mon, 22 Nov 2010 17:21:31 GMT

I don't think the process settings for case handling and extended parsing have a one-to-one correspondence to what the CRTL does. For example, if case matters, you'll likely need DECC$EFS_CASE_PRESERVE and/or DECC$ARGV_PARSE_STYLE. In your problem example, though, it looks like it's all lower case.

Do check, since you have case sensitivity turned on, that the actual file and directory names on disk are the same as specified in the patch, e.g., that you have src.DIR;1, not SRC.DIR;1.

You might check whether you have the logical name "file" defined or try defining DECC$UNIX_PATH_BEFORE_LOGNAME. Make sure you do NOT have DECC$EFS_NO_DOTS_IN_DIRNAME defined.

I actually would have expected DECC$EFS_CHARSET to be enough in this example. Since it isn't, I would try a big hammer like:

$ define DECC$UNIX_LEVEL 30

and see what happens. You could also edit the patch to insert caret escapes, so

file/file-4.09/src/compress.c

becomes

file/file-4^.09/src/compress.c

which really shouldn't be necessary but might work.

Navigating to the subdirectory as you have done is a good one-off solution for one file, but wouldn't help if the patch had numerous files from several directories, so I hope we get to the bottom of this.

Re: GNV and CRTL

Hoff — Mon, 22 Nov 2010 18:18:20 GMT

IIRC, there was a particular link command that was used for parts of gnv; that included a lib$initialize psect module into the image that seeks to establish the DECC$ hackery correctly, IIRC.

Re: GNV and CRTL

H.Becker — Mon, 22 Nov 2010 21:18:20 GMT

What Hoff is refering to is crtlsup/src/vms_crtl_init.c

From what I read, you want to have/use GNV diff and patch. Patch seems to be missing in GNV, and your versions of these GNU tools for VMS seem to miss the necessary initialization with the DECC$ features. The above module should help or point you into the right direction..

Re: GNV and CRTL

Craig A Berry — Mon, 22 Nov 2010 21:44:58 GMT

vms_crtl_init.c is visible here:

http://gnv.cvs.sourceforge.net/viewvc/gnv/gnv213/crtlsup/src/vms_crtl_init.c?revision=1.1.1.1&view=markup

It looks like there is a version of GNU patch in the latest GNV:

http://gnv.cvs.sourceforge.net/viewvc/gnv/gnv213/patch/

but there have been various ports kicking around so I don't know if that's the one Martin is using.

Re: GNV and CRTL

Martin Vorlaender — Tue, 23 Nov 2010 14:44:12 GMT

Craig, Hoff, Hartmut,

thanks for the confirmation that it should have worked ;-) In detail:

I had carefully looked after the case of directory names (having been bit by that several times in this project). So yes, the whole path's case is exactly as it appears to patch:

$ dir [.gnv213.file.file-4^.09.src]compress.c

Directory DISK$BUILD:[build.gnv213.file.file-4^.09.src]

compress.c;1

Total of 1 file.

(BTW: GNV_SPECIFIC is defined as 'f$trnlnm("DISK$BUILD")'[build.gnv213.] )

I also fell into the logical name trap (though with MANPATH defined by CDE, and the man kit containing a manpath.OBJ file). But no, in this case "file" is not defined as a logical.

Re: DECC$UNIX_LEVEL : In my despair to get it to work (without testing any one of the dozens of feature logicals) I indeed set that logical to 30 once - no change.

Re: Inserting a caret into the Unix path : I tried that, though not with this patch, but in my general experiments with stat (see attached).

Re: Using vms_crtl_init / GNV's patch : I'm in the process of building the GNV environment from scratch, so no GNV utilities, please (except where the build process calls them). I have no idea where my gpatch.exe came from - it gets copied from system to system :-) Most probably it's the Freeware CD version. To trace the error, I fetched the sources from Freeware #5 and compiled it /DEBUG. So, no decc$feature_set here. BUT: the GNV environment contains FEATURES.COM which I called when setting it up. This defines a whole slew of DECC$* feature logicals. Should have the same effect, shouldn't it?

http://gnv.cvs.sourceforge.net/viewvc/gnv/gnv213/build/FEATURES.COM?revision=1.1.1.1&view=markup

(plus, of course, my definition of DECC$EFS_CHARSET).

Thanks,

Martin

Re: GNV and CRTL

Craig A Berry — Tue, 23 Nov 2010 15:46:18 GMT

I guess the next thing to do would be to make a tiny C program that does only a stat and see if you can reproduce the ENOENT that way. If so, then run it under SET WATCH FILE and see what it's actually looking for and can't find.

Re: GNV and CRTL

Hoff — Tue, 23 Nov 2010 17:02:33 GMT

This whole area of C has been a moving target, particularly around the logical name morass. Old C code can variously need tweaks here, and (when this arises) usually with the logical name settings. I'd not assume that old code works.

You'll need to decode the FEATURES.COM settings for your environment; that list of logical names is incomplete.

Don't turn on case sensitivity unless you absolutely need that. That switch will find filename and filename casing assumptions all over the place.

GNV has problems with mixed environments; mixing VMS file specifications and Unix path syntax is problematic. I'd suggest using (only) Unix syntax, as that path seems to be the most reliable.

Utility doesn't come from simplicity or surface aesthetics. Utility comes from forging complexity into something you can comfortably wield.

The GNV initialization code stuff establishes an lib$initialize initialization psect that sets the defaults for the executable; you can recreate this code yourself easily, and establish the defaults in a somewhat modular fashion. (This is the workaround for most of the DECC$* logical name morass.)

There are diff, patch and xxd tools in the OpenVMS vim kit over at Polarhome, if that's of interest.

I'd probably get (most of) the sourceforge svn out of the picture and run the repository with git; git-svn or a git svn clone command (if you're departing from svn) can help with that.

http://www.kernel.org/pub/software/scm/git/docs/git-svn.html

git svn clone http://PROJECT.svn.sourceforge.net/svnroot/PROJECT PROJECT.git

There are various issues with the default GNV set-up; have a look at the environment over at DECUServe for some of the fixes and updates that can be required.

And following the same vein, the DECUServe PORTING_TO_VMS notes conference has extensive information on C portability and the DECC$* settings.

I'd hope that the GNV project will decide to start using Sourceforge as something more than a file server and to engage the community, too; it'll be interesting to learn if this is a future during that upcoming GNV session.

It's often easier to fix the file formats (switching to stream LF) and migrate the VMS files to a platform with tools for the necessary work, then haul the stuff back to VMS to build it.

Build the reproducer, too. Narrow this stuff down.

I've posted some of my adventures with GNV. Here's one:
http://labs.hoffmanlabs.com/node/1481

Here are some of the topics associated with the DECC$* logical name settings:
http://labs.hoffmanlabs.com/node/1513
http://labs.hoffmanlabs.com/node/1515

"Utility doesn't come from simplicity or surface aesthetics. Utility comes from forging complexity into something you can comfortably wield." Unfortunately and problematically, both GNV and the C Unix compatibility environment and the current code-drop use model over at Sourceforge are presently al; rather unwieldy.

Re: GNV and CRTL

Craig A Berry — Tue, 23 Nov 2010 17:50:45 GMT

Martin, just as a reminder that what goes around comes around, note the following in src/vms/vmsutil.c of the GNU patch distro on the Freeware CD v5.0:

* Stolen from the VMSperl source by Martin Vorlaender
* Um, that's actually GPB'd (GNU Public Borrowed) under the terms of the
* GNU Public License.

And a few lines further down:

*
* Wrapper routines added by Craig Berry based in part on code by Charles
* Lane.
*

So what's happening is our fault :-). As Hoff said, don't assume old code will work unmodified, even (especially?) if it's your own!

More specifically, there is a my_stat() wrapper around stat() that calls a home-grown converter routine to convert a filespec in Unix syntax to VMS syntax. The converter routine contains the following line in the part that handles the directory portion of the path:

else *(cp1++) = '_'; /* fix up syntax - '.' in name not allowed */

So we're explicitly defeating any attempt to make a dot in a directory name visible to the filesystem by converting it to an underscore. Remember, the target 10 years ago was VMS 6.x and early 7.x. There was limited or no capability in the CRTL to accept Unix-style pathnames, convert between Unix and VMS syntax, etc. ODS-5 did not exist and a dot in a directory name was a hard syntax error with no features to work around that.

I think your best bet may be to start over with the GNU patch from the GNV kit (or even just the authoritative GNU sources).

Re: GNV and CRTL

Martin Vorlaender — Tue, 23 Nov 2010 19:44:53 GMT

Ouch!

Of course I didn't think of stepping into the stat call...

Time to update the GPB'ed portions of gpatch, I guess.

Thanks a lot!

Over and out...

Re: GNV and CRTL

H.Becker — Tue, 23 Nov 2010 20:11:02 GMT

>>> I'm in the process of building the GNV environmet from scratch, so no GNV utilities, please (except where the build process calls them).

Is this really "from scratch"? It seems good enough to build GNV from the known source tree and then use its tools to create a branch.

On the other hand, did you try the VMS tools for that: diff/slp and edit/sumslp?

Re: GNV and CRTL

Martin Vorlaender — Wed, 24 Nov 2010 04:46:24 GMT

Hartmut,

no, this isn't from scratch in the sense that I'm not starting with the SF sources.

The "Building GNV" page at http://gnv.sourceforge.net/building.html starts from the source files installed with the binary kit, i.e. a working GNV environment is assumed to build GNV from source.

As I find this to be a bit absurd, I tried to *really* build from the sources, i.e. download the source files from SourceForge onto a VMS system equipped with just the compilers (NB: it doesn't say anywhere you need BLISS installed [to build vitpu]). That's what I meant when writing "from scratch".

I'm comfortable with GNU patch usually, and haven't ever used SUMSLP, so I never thought of using that.

cu,
Martin

Re: GNV and CRTL

H.Becker — Wed, 24 Nov 2010 17:57:06 GMT

To me Posix on VMS was the unwanted child, GNV seems to be wanted somehow, but neglected.

There are cvs commands shown on sourceforge, which do not work: That means you can't take a snapshot or check out any module via cvs, except you are registered/permitted. You just can browse and view (download) each single file. That may make sense when there is no cvs for VMS. Is there?

But there is a GNV tarball, gnv.tar.gz, which contains both source trees. Why someone would want both at the same time is unknown to me. This file shows under "browse code". Then you need just a recent tar utility on VMS.

So you can get a full source tree without installing the binaries. Whether this is the same tree as the one which comes with the binaries is unknown to me.

I didn't see any readme or so, but I would just kick off a "@[.build]build all". (I haven't tried it, I currently don't have access to a system with enough resouces andd a BLISS compiler).

Re: GNV and CRTL

Martin Vorlaender — Thu, 25 Nov 2010 05:05:29 GMT

>>>
There are cvs commands shown on sourceforge, which do not work: That means you can't take a snapshot or check out any module via cvs, except you are registered/permitted.
<<<

I used TortoiseCVS for Windows to checkout the source tree (which is integrated into Exploder) but you made me curious, so I tried the commands from http://sourceforge.net/projects/gnv/develop . The first one asks for a password, but it's okay to just press RETURN there. For the second one, you have to look up via "browse code" that the module name is "gnv213". Then it checks out the source tree okay.

>>>
I didn't see any readme or so, but I would just kick off a "@[.build]build all".
<<<

The "doc" folder has a "Read Me First" in TXT, PS, PDF, and HTML. It's for version 1.6 (July 2004), though. And the PDF is broken. But that folder at least has current release notes. And there's the "Building GNV" web page I mentioned.

You'll need multiple rounds of build.com, as several build files use GNV commands (e.g. bash, make, which, dcl, patch) - which of course is not a problem if you're starting with GNV already installed...

And, of course, neither the kitting commands nor GNV$STARTUP.COM is included.

cu,
Martin

Re: GNV and CRTL

Martin Vorlaender — Mon, 29 Nov 2010 10:24:25 GMT

Just FYI:

Conditionalizing most of vmsutil.c:do_tovmsspec() with "#if __CRTL_VER < 70000000 || defined (LINK_TARGET_PRE7)" yields:

- gpatch can now cope with dots in directory names, i.e. the stat call isn't failing any more

- gpatch doesn't understand VMS filename syntax on the command line any more (e.g. for the -i filename).

Yes, it would have been better to get a current do_tovmsspec() from the perl 5.12.2 sources, but laziness prevails (for now, at least).

cu,
Martin