"test" evaluation of variables

[Jeff_Traigle]

I don't think this is a real issue, but I wanted to double-check with folks on here before I call the job done in case there's something I'm not aware of.

A developer reported that the test command stopped working properly. When a variable containing a string value is used with a numeric operator, the test fails because of the type mismatch for the operator. All well and good as far as I'm concerned since that's exactly what should happen in my experience.

For some reason he was expecting test to evaluate the variable to a 0 value during the numeric operation if it was a string. I'm not a programmer, but I don't recall this behavior in any language I used throughout college... FORTRAN, Pascal, C... so I'm not sure where his notion came from.

He also claims this worked fine in 2003 with the same scripts, but the test command hasn't been touched by any patches since then so I'm skeptical of this.

I told him about typeset to enforce type checking during assignment of a value to the variable. He's also used a sed command to validate the variable value is numeric in one script. He's wanting an environmental approach so he doesn't need to check all the scripts they have for this "problem". I'm not aware of any. As far as shell scripts are concerned, this is handled only on a variable by variable basis that I've ever seen.

[A. Clay Stephenson · Accepted solution]

The atoi() function and its cousins (e.g. atol(), strtol(), strtoul()) which convent strings to integers have this characteristic:

A string is scanned up to the first non-numeric character (or in the case to strto(u)l up to the first chacterer inconsistant with the integer base) and does a numeric conversion. If no conversion is possible, zero is returned. For example, if "234Tom" were supplied to atoi(), it would return 234. If "Tom234" were supplied, atoi() would return 0. There is therefore some precedent for his argument and there have been a ferw bugs in the shell. For example, a value beginning with "0" would be intepreted as octal so that the value "087" would be treated by the shell as an invalid value. However, aside for am obvious numeric bug like that, checking the contents of a variable before doing a numeric comparison is the sole responsibility of the programmer. Man atoi for details.

[Bill Hassell]

And to amplify just a bit more on Clay's comments, there was a patch to the POSIX shell concerning numeric values. Patch PHCO_25597 dated 2002 and PHCO_34240 dated 2006, cumulative patches for ths POSIX shell causes numbers to be processed using the octal and hex ISO C conventions, specifically a leading zero means octal digits follow and 0x means hex digits follow. When the patch was first released, it broke some standard HP-UX startup scripts which contained invalid octal numbers like 089 which gave test errors on startup.

NOTE: there is a significant difference between the two patches!! 25597 introduces the ISO C behavior but 34240 makes the behavior optional based on UNIX95. From the 25597 README (concerning startup script rc.utils):

For example: rc.utils might have configured as follows:
let ROWS="$rows"
let COLS="$cols"
Now, if the rows/cols variables start with a 0, the shell will interpret the data as octal numbers resulting in incorrect interpretations or errors. Hence we advise users to change the rc.utils as follows:
let ROWS="${rows##*(0)}"
let COLS="${cols##*(0)}

and also...

"The problem can be reproduced as follows:
a. Type the following commands on command prompt.
echo $((0x10))
let "a=0x10"; echo $a
((a=0x10)) ; echo $a
The shell will give an error
"sh: 0x10: The specified number is not valid for this command."
instead of printing 16.

b. Type the following command on command prompt.
echo $((010))
let "a=010"; echo $a
((a=010)) ; echo $a
The shell gives output "10" instead of "8".

-----------------------------

However, it looks like this caused too many scripts to break so the UNIX95 flag was enhanced for the POSIX shell to turn on the behavior, but leave the misinterpretation (based on ISO-C standards) broken as it was before. So there is a workaround -- From the patch README for 34240:

"Numbers beginning with 0 or 0x/0X are treated as octal and hexadecimal respectively, if and only if, UNIX95 flag is defined. Since the fix for JAGad93413 is to make sh-posix(1) standards compliant, the commands - let "..", $((..)), ((..)) works as per ISO C standards by default, and is the only behavior available for 11.22 and above releases.

However, in order to maintain compatibility with already existing scripts on 11.11 and 11.00 the above fix is enabled only if environment variable UNIX95 is defined in the shell. It is also expected that customers writing standard compliant programs/scripts will enable this variable.This allows them to make use of full ISO-C standard behavior.If UNIX95 is not defined, the above mentioned commands will not recognize numbers beginning with 0 and 0x/0X as octal and hexadecimal respectively."

Note that ksh has always interpreted the octal and hex numbering correctly. If a script works OK with POSIX shell but fails on a numeric test in ksh, that is very likely the problem, and depending on whether patch 25597 is installed, the script may break. Of course, the coding is defective and as seen from the above example for ROWS and COLS, the defect may be dependent on other sources of numbers. I'd go for the coding fix simply to maintain standards compatibility.

[TwoProc]

The behavior he's hoping for is how string conversions behave in C. But, I've not seen that work in the various shells that way.

[Jeff_Traigle]

We're at patch PHCO_27345 for the POSIX shell so I'm guessing they just never had garbage input for the variable causing the trouble in all this time.

Otherwise, pretty much what I figured.

As always, the insights are greatly appreciated.