topic Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]? in Operating System - OpenVMS

CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Steven Schweda — Mon, 07 Mar 2011 23:48:30 GMT

Here's an interesting quirk into which I
recently ran. I was trying to build a recent
OpenSSL kit (1.0.0d) using 64-bit pointers.
(CC /POINTER_SIZE=64=ARGV [...].)

After getting past most of the easy problems,
I ran into some test failures whose cause was
unexpected, perhaps even novel. Several of
the [.apps]*.c programs scan their
command-line argv[] array using code like
this:

[...]
int MAIN(int argc, char **argv)
[...]
char **args;
[...]
args = argv + 1;
[...]
while (*args) [or similar]
[...]
args++;
[...]

That is, they look for a NULL terminator at
(past) the end of "argv". (I might have
expected to see this sort of thing with
envp[], but I'd've figured that argc was there
for a reason, and I'd've used that for
determining where argv[] ended. But it's not
my code.)

This NULL-terminator expectation seems to work
well enough on VAX and IA64, and on Alpha with
32-bit pointers, but, on Alpha with 64-bit
pointers, it breaks down. In that case, these
programs frequently see one extra garbage
argument on their command lines, leading to
symptoms like:

Error opening recipient certificate file ÀØÜz

where "ÀØÜz" is what appears to follow the
usual (real) command-line arguments in argv[].

The obvious fix is to re-write these programs
so that they use argc, and don't look for a
NULL terminator at the end of argv[]. But,
it's apparent that some people seem to expect
this thing to be NULL-terminated, and in this
situation, it's not. For all I know, there's
some POSIX standard (or something) which
demands that argv[] be NULL-terminated, in
which case, this would seem to be a bug. In
any case, there would seem to be a plausible
argument to be made for making this work as
well on Alpha as it seems to on IA64 (whether
that's from good design or just good luck).
(And if it is only good luck, then it might be
good to add a little reinforcement to the
luck.)

alp $ cc /version
HP C V7.3-009 on OpenVMS Alpha V8.3

With no patch access these days, this is all
of only academic interest to me, but I thought
that it might be of some interest to a broader
audience.

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

John Gillings — Tue, 08 Mar 2011 03:14:54 GMT

Steven,

Interesting. I'm curious to know if this is one of those things that fell out of someone's implementation, that are now widespread enough to warrant standardising.

Since your pointers are now 64 bit, is it possible there's a 32 bit null, followed by junk?

If I've translated your magic string "Ã Ã Ã z" correctly, it's hex 7ADCD8C0, which on my V8.3 system appears to be inside DCL. Does argv always map to the same location? (I don't have a C compiler to experiment with).

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Craig A Berry — Tue, 08 Mar 2011 03:28:53 GMT

The standard at

http://c0x.coding-guidelines.com/5.1.2.2.1.html#170

or 5.1.2.2.1 of C99 if you're following along elsewhere, says

"â argv[argc] shall be a null pointer."

So I think you've found a bug.

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Craig A Berry — Tue, 08 Mar 2011 03:35:04 GMT

Also, the POSIX specs on exec and friends

http://pubs.opengroup.org/onlinepubs/009695399/functions/execve.html

say, "The argument argv is an array of character pointers to null-terminated strings. The application shall ensure that the last member of this array is a null pointer. "

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Steven Schweda — Tue, 08 Mar 2011 04:15:28 GMT

> If I've translated your magic string "Ã Ã Ã z"
> correctly, [...]

That's not really the pointer itself, but the
"string" it pointed to, although it's not far
off. My fprintf() diagnostic output looked
like this:

[...]
cms. argc = 10.
[Here, it's looking at the non-option
(file-name) arguments. "argc_x" was my
temporary count (-down-to-one) of valid
remaining arguments.]
cms(1). argc_x = 3, *args: >smime-certs/smrsa1.pem<.
file_ctrl(). ptr: %x00000000004e077a >smime-certs/smrsa1.pem<.
cms(1). argc_x = 2, *args: >smime-certs/smrsa2.pem<.
file_ctrl(). ptr: %x00000000004e0791 >smime-certs/smrsa2.pem<.
cms(1). argc_x = 1, *args: >smime-certs/smrsa3.pem<.
file_ctrl(). ptr: %x00000000004e07a8 >smime-certs/smrsa3.pem<.
cms(1). argc_x = 0, *args: >Ã Ã Ã z<.
file_ctrl(). ptr: %x000000007adcd870 >Ã Ã Ã z<.
Error opening recipient certificate file Ã Ã Ã z
[...]

("file_ctrl()" is the function which actually
tried to open the file by name.)

So, unless I bungled something, that last
"ptr:" value (which looks significantly
different from the earlier, valid ones) is
what everyone was hoping would be all zeros.

> [...] Does argv always map to the same
> location? (I don't have a C compiler to
> experiment with).

I believe that it was pretty reliable, but
once I found out what was happening, I
stopped paying close attention to the
details. I figured that someone must know
who builds argv[], and how, so if this was an
actual bug, then it should be pretty easy to
investigate and/or fix.

> [...] (I don't have a C compiler to
> experiment with).

That's a pretty shameful admission, isn't it.

> The standard at [...]

Hmmm. Those certainly are suggestive.
Perhaps it was always thus, and I, blinded by
the existence of argc, never noticed. Or,
perhaps whoever's setting this stuff up
thinks that it's being done properly, but
apparently it's not. Certainly, most of the
code I've seen looks at argc, not for a NULL
at the and of argv[], and most VMS users
don't compile most stuff with a 64-bit
argv[], so a problem like this might easily
have been missed.

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Craig A Berry — Tue, 08 Mar 2011 21:22:47 GMT

Here's a simple reproducer. Works ok on Itanium, but fails on Alpha (though not a particularly up-to-date on ECOs Alpha). Hard to say that it's a compiler or CRTL problem for sure since something might have already gone wrong before language support took over from the image activator.

$ create argv_of_argc.c
#include
#include

int main(int argc, char *argv[])
{
void *p = argv[argc];

if (p != NULL)
printf("not ");

printf("OK - argv[argc] should be NULL (got %p)\n", p);
}
^Z
$ cc/pointer_size=32 argv_of_argc
$ link argv_of_argc
$ r argv_of_argc
OK - argv[argc] should be NULL (got 0)
$ cc/pointer_size=64=argv argv_of_argc
$ link argv_of_argc
$ r argv_of_argc
not OK - argv[argc] should be NULL (got 7FF57810)
$ cc/vers
HP C V7.3-009 on OpenVMS Alpha V8.3

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

John Gillings — Tue, 08 Mar 2011 21:33:36 GMT

>That's a pretty shameful admission,
>isn't it.

au contraire!

Having spent almost 20 years as a tech support "linguist", dealing with all manner problems reported in the full spectrum of programming languages, C was, by far, the worst for programmers reporting "I've found a bug" when the real issue was they didn't understand what they'd really asked the compiler to do (next in line was Basic, but that's another story)

Far too heavy on obscure syntax "swiss army knife" control structures, and far too light on easily understood data structure definitions.

Indeed, this report is a perfect example of the fundamental problem with the language. Proponents of C would laud the "freedom" to code something as basic as processing an argument list in different ways.

I'd expect a programmer to be able to glance at and immediately recognise argument parsing without having to figure out which variation the programmer has chosen.

The dependence that a programmer is supposed to "just know" that an argument list is terminated by a null, along with the volumes of other critically important language-lawyer trivia is a recipe for building in bugs.

All programming languages have ideosyncratic faults, but C takes the art form to a whole new level. The only reason the language survives at all is the strategic history of giving compilers away free to universities in the late 1970's, and the resulting momentum from programmers who never knew any better. (At the same time a BLISS compiler license was US$20K per year)

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Lucifer Megacruel — Wed, 09 Mar 2011 11:26:13 GMT

Hi Steve,

I am not trying to reduce the severity of the problem. But a more POSIX ey way of doing it these days I guess is to use the getopt family of functions than messing with the command line options in the application anyway :). Can you just see if that works ?

--Lucifer

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Steven Schweda — Wed, 09 Mar 2011 12:02:02 GMT

> [...] Can you just see if that works ?

Perhaps I could, but I won't. It's not my
code, and I really doubt that the OpenSSL
folks would like to drag all the getopt stuff
into their product, even if it would help
work around this (VMS-specific) problem. (I
know that _I_ wouldn't.)

> [...] a more POSIX ey way [...]

The _most_ POSIX-y way of doing anything is
usually to make the C infrastructure
(compiler, RTL, ...) follow the POSIX
standard, which seems not to be happening
here.

My initial work-around was to re-write the
command-line processing code to look at argc,
but, as the head OpenSSL guy suggested, it's
a smaller change to check
if (argv[ argc] != NULL)
in a central location (which exists), and
then, if necessary, make a properly
terminated duplicate argv[], and then pass
that along to the consumers, instead of the
original, defective argv[]. It's ugly, but
it works (and it's more compact than "fixing"
the several consumers).

I sent a short message to
OpenVMS.Programs@you-know-where, citing this
thread, so I'm sure that the appropriate
engineers will soon be all over this problem.
Perhaps someday I'll even be able to get hold
of the patch which fixes it, but I'm not
holding my breath.

> au contraire!
> [...]

With great power comes great responsibility.

> [...] a programmer is supposed to "just
> know" [...]

Or just read one of those standards
documents. Of course, it helps if the
underlying implementation is bug-free.

> [...] The only reason the language survives
> at all [...]

There's a reason that people try to gain
market share for a new product. Momentum,
once gained, can require significant force
and/or time to extinguish. (Ask Newton.)

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Lucifer Megacruel — Wed, 09 Mar 2011 12:52:26 GMT

Hi Steve,

The _most_ POSIX-y way of doing anything is
usually to make the C infrastructure
(compiler, RTL, ...) follow the POSIX
standard, which seems not to be happening
here.

I am not denying that, let me take a look at it deeper, may be when i have some more time.

--Lucifer

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Jeffrey Goodwin — Wed, 09 Mar 2011 13:35:49 GMT

The problem was on IA64 at one time:

IA64 Dec C compiler for OpenVMS: Includes fix for problem related to specifying /POINTER_SIZE=LONG=ARGV, the argv argument to main, which comprised of long pointers, was not terminated with NULL.

This problem has been corrected HP-I64VMS-C-V0703-019ECO1-1

I would seem the problem was the same root issue. I didn't see a matching Alpha patch.

-Jeff

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

WW304289 — Wed, 09 Mar 2011 15:08:32 GMT

"
For all I know, there's
some POSIX standard (or something) which
demands that argv[] be NULL-terminated [...]
"

It is in the C standard. From ISO/IEC 9899:201x

5.1.2.2.1 Program startup

1 The function called at program startup is named main. The implementation declares no
prototype for this function. It shall be defined with a return type of int and with no
parameters:

int main(void) { /* ... */ }

or with two parameters (referred to here as argc and argv, though any names may be
used, as they are local to the function in which they are declared):

int main(int argc, char *argv[]) { /* ... */ }

or equivalent;9) or in some other implementation-defined manner.

2 If they are declared, the parameters to the main function shall obey the following
constraints:

â The value of argc shall be nonnegative.

â argv[argc] shall be a null pointer.

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Dennis Handly — Sat, 12 Mar 2011 04:01:46 GMT

>on Alpha with 64-bit pointers, it breaks down. In that case, these programs frequently see one extra garbage argument on their command lines

Have you tried printing that address? Perhaps only 32 bits is zero?

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Steven Schweda — Sat, 12 Mar 2011 04:20:31 GMT

> Have you tried printing that address?

Have you tried reading this thread?

> Perhaps only 32 bits is zero?

We're little-endian here, so values like:

> [...] ptr: %x000000007adcd870 [...]
(Mar 8, 2011 04:15:28 GMT)

or:

> [...] (got 7FF57810)
(Mar 8, 2011 21:22:47 GMT)

aren't even zero at the beginning (the
low-significance end).

But thanks for playing.

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Robert Brooks_1 — Sat, 12 Mar 2011 04:54:42 GMT

This problem has been corrected HP-I64VMS-C-V0703-019ECO1-1

I would seem the problem was the same root issue. I didn't see a matching Alpha patch.

---

I've seen some internal-to-HP communication on this one, and apparently VMS Engineering states that an Alpha patch is in the works.
I have no idea when it's due to hit the streets.

-- Rob

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Steven Schweda — Sat, 12 Mar 2011 05:19:28 GMT

> This problem has been corrected
> HP-I64VMS-C-V0703-019ECO1-1

Hey. Sure enough. That's so old that even
_I_ have it:

IT $ cc /version
HP C V7.3-019 on OpenVMS IA64 V8.3-1H1

which explains why I saw the problem only on
Alpha (V7.3-009).

I probably read the ".txt" file when I
downloaded the patch, but figured that I'd
never care, and promptly forgot all about it.
(It's not my fault. This 64-bit pointer
stuff was all someone else's idea. I may
still never care for my own purposes.)

> [...] an Alpha patch is in the works.
> I have no idea when it's due to hit the
> streets.

That's nice to know, but, of course, a lowly
hobbyist peon like me may never see it, as I
can no longer gain access to those streets.

Re: CC /POINTER_SIZE=64=ARGV, Alpha, NULL termination of argv[]?

Dennis Handly — Sat, 12 Mar 2011 13:59:21 GMT

>Have you tried reading this thread?

Oops, sorry I missed that.
It would have helped if you printed them as increasing:
"argv[%d] = ptr: %p, %s\n", i, &argv[i], argv[i]

>aren't even zero at the beginning (the low-significance end).

Yes.