topic Re: DCL CHECKSUM sometimes doesn't select correct architecture in Operating System - OpenVMS

DCL CHECKSUM sometimes doesn't select correct architecture

Benjamin Levy — Sun, 15 Feb 2009 22:24:48 GMT

Has anyone seen this one...?

Beginning with VMS V8.2, CHECKSUM is now documented, and has switches /ALPHA and /I64. Default is documented in the HELP as being /ALPHA on an ALPHA.

Try executing the following:

$ arch = f$getsyi("ARCH_NAME")
$ sho sym arch
$
$ open/write checksum_test file_created_by_dcl.txt
$ write checksum_test "Testing Checksum"
$ close checksum_test
$!
$ checksum file_created_by_dcl.txt
$ write sys$output -
"Checksum with no architecture specified is ''checksum$checksum'""
$!
$ checksum/alpha file_created_by_dcl.txt
$ write sys$output "Checksum with /ALPHA is ''checksum$checksum'""
$!
$ checksum/i64 file_created_by_dcl.txt
$ write sys$output "Checksum with /I64 is ''checksum$checksum'""

$ @checksum_test
ARCH = "Alpha"
Checksum with no architecture specified is 269859666"
Checksum with /ALPHA is 1510215701"
Checksum with /I64 is 269859666"

Even though I am running this on an Alpha, the CHECKSUM utility is defaulting to /I64

Re: DCL CHECKSUM sometimes doesn't select correct architecture

Robert Walker_8 — Mon, 16 Feb 2009 00:05:58 GMT

Hi Ben,

Not for one to experience VMS on IA64, but have gone through VAX and Alpha migrations some years ago.

One would normally have to put a little logic into the DCL. Either say

$if f$getsyi("ARCH_NAME") .eqs. "Alpha" then
$ checksum/alpha
$else
$ checksum/i64
$endif

or depending upon output of ARCH_NAME one may get away with

$checksum/'f$getsyi("ARCH_NAME")'

It is probably better to set up a variable at top for Arch type. This way you can have scripts set to run in mixed-clusters, and thus reduce the need to keep two sets of scripts up to date.

Robert.

Re: DCL CHECKSUM sometimes doesn't select correct architecture

John Gillings — Mon, 16 Feb 2009 01:05:03 GMT

Ben,

/IA64 and /ALPHA are only relevant with /IMAGE. They're used to calculate image checksums for the "other" architecture. See HELP CHECK/ALPHA and HELP CHECK/I64

I'm not sure why you're seeing a difference in the calculated checksums. Here's an example, on the same file (an Alpha image)

From Alpha:
$ checksum alphatest.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "2823409345"
$ checksum/alpha alphatest.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "2823409345"
$ checksum/i64 alphatest.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "2823409345"

From I64
$ checksum alphatest.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "2823409345"
$ checksum/i64 alphatest.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "2823409345"
$ checksum/alpha alphatest.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "2823409345"

Similarly, executing against an I64 image

From Alpha:
$ checksum i64test.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1570401477"
$ checksum/alpha i64test.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1570401477"
$ checksum/i64 i64test.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1570401477"

From I64:
$ checksum i64test.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1570401477"
$ checksum/alpha i64test.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1570401477"
$ checksum/i64 i64test.exe
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1570401477"

Using your test, I get consistent results on I64, but not on Alpha (same as your Alpha results)

Alpha:
$ checksum file_created_by_dcl.txt
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "269859666"
$ checksum/alpha file_created_by_dcl.txt
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1510215701"
$ checksum/i64 file_created_by_dcl.txt
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "269859666"

I64
$ checksum file_created_by_dcl.txt
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1510215701"
$ checksum/alpha file_created_by_dcl.txt
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1510215701"
$ checksum/i64 file_created_by_dcl.txt
$ show sym checksum$checksum
CHECKSUM$CHECKSUM = "1510215701"
$ dir/full file_created_by_dcl.txt

I suspect this may have something to do with the file being VFC

When used for its intended purpose, the architecture qualifiers appear to work correctly:

Alpha:

$ checksum/image alphatest.exe
file DSA1:[GILLINGS]ALPHATEST.EXE;1
image section %D'1' checksum is %X'200D1419'
image section %D'2' checksum is %X'010C0305'
image section %D'3' checksum is %X'B1C4C909'
image section %D'4' checksum is %X'585945C2'
image header checksum is %X'00000019'
checksum of all image sections is %X'C89C9BD7'
$ checksum/image i64test.exe
file DSA1:[GILLINGS]I64TEST.EXE;1
%IMGACT-F-NOTNATIVE, image is not an OpenVMS Alpha image
$ checksum/image/i64 i64test.exe
File DSA1:[GILLINGS]I64TEST.EXE;1
Checksum program segment 0: %XB8350076
Checksum program segment 1: %X22C8DC48
Checksum program segment 2: %XE316BCEE
Checksum program segment 3: %X967AAF38
Checksum dynamic segment %X9991F29D
Elf header checksum: %XB27563D0
Elf program header checksum: %X59BCEBA7
Elf section header checksum: %X4CE2E954
Elf (object/image) checksum: %XA846BF48

I64:
$ checksum/image i64test.exe
File DSA5:[GILLINGS_J]I64TEST.EXE;1
Checksum program segment 0: %XB8350076
Checksum program segment 1: %X22C8DC48
Checksum program segment 2: %XE316BCEE
Checksum program segment 3: %X967AAF38
Checksum dynamic segment %X9991F29D
Elf header checksum: %XB27563D0
Elf program header checksum: %X59BCEBA7
Elf section header checksum: %X4CE2E954
Elf (object/image) checksum: %XA846BF48
$ checksum/image alphatest
File DSA5:[GILLINGS_J]ALPHATEST.EXE;1
%CKSM-W-NOTELF, not an Elf object/image file
$ checksum/image/alpha alphatest
file DSA5:[GILLINGS_J]ALPHATEST.EXE;1
image section %D'1' checksum is %X'200D1419'
image section %D'2' checksum is %X'010C0305'
image section %D'3' checksum is %X'B1C4C909'
image section %D'4' checksum is %X'585945C2'
image header checksum is %X'00000019'
checksum of all image sections is %X'C89C9BD7'

If there's a bug here, it's that CHECKSUM allows the architecture qualifiers without /IMAGE. I've attached a CHECKSUM.CLD with the additional constraint:

DISALLOW ((alpha or i64) AND NOT (image or object))

If you use this CLD, the command CHECKSUM/ALPHA (without including /IMAGE) will result in:

$ checksum/alpha alphatest.exe
%DCL-W-CONFLICT, illegal combination of command elements - check documentation
\ALPHA\

Reading further, it may be slightly more correct to change the constraints to:

DISALLOW ALPHA AND NOT IMAGE
DISALLOW I64 AND NOT (IMAGE OR OBJECT)

Re: DCL CHECKSUM sometimes doesn't select correct architecture

Benjamin Levy — Mon, 16 Feb 2009 03:45:03 GMT

There is, in my opinion, a bug here.

If I take a command file on Alpha V7.3-2 with CHECKSUM, without qualifiers, and run it on a file in VFC format, and then do exactly the same thing on Alpha V8.3, the checksums do not match even though the file is identical. In theory HP could say this isn't a bug because CHECKSUM wasn't supported on V7.3-2, but I think that is a stretch - when they started supporting it on V8.2 they attempted to make it backwards compatible with the previous undocumented feature.

I have procedures that check that files on various nodes have the same checksum, and these procedures no longer work. It will be necessary to add vms-version specific logic to these procedures, to perform CHECKSUM on VMS V7.x and CHECKSUM/I64 on VMS V8.x.

I agree that the /ALPHA and /I64 should only be necessary when analyzing images, unfortunately that is not the case - my procedures which are not looking at images, will not work without /ALPHA added.

Interestingly, on Alpha V8.3, if I type CHECKSUM filename and specify a file in VFC format, it does not even invoke CHECKSUM.EXE, but rather I64_CHECKSUM.EXE. This makes me wonder if the underlying bug is in the CLD tables.

Re: DCL CHECKSUM sometimes doesn't select correct architecture

John Gillings — Mon, 16 Feb 2009 04:11:54 GMT

Ben,

The CLDs are all in SYS$UPDATE. If you think the problem is there, you can experiment yourself. If you find a fix, send it back to HP.

The significant differences between them are:

$ diff IA64_CHECKSUM.CLD,alpha_checksum.cld /para/ignore=(comment,trail,form,blank)/comment="!"
-----------------------------------------------------------------------------------------------------------------------------------
File DSA1:[GILLINGS]IA64_CHECKSUM.CLD;1 | File DSA1:[GILLINGS]ALPHA_CHECKSUM.CLD;1
-------------------------------- 49 -------------------------------------------------------------- 55 -----------------------------
DEFINE SYNTAX alpha_checksum | DEFINE SYNTAX ALPHA_CHECKSUM
image alpha_checksum | image checksum
DEFINE SYNTAX i64_checksum | DEFINE SYNTAX I64_CHECKSUM
image checksum | image i64_checksum
DEFINE VERB checksum | DEFINE VERB CHECKSUM
IMAGE checksum | image i64_checksum
-------------------------------- 62 -------------------------------------------------------------- 68 -----------------------------
QUALIFIER image | QUALIFIER image, syntax=alpha_checksum
-------------------------------- 71 -------------------------------------------------------------- 77 -----------------------------
DISALLOW ANY2 (alpha, vax) | DISALLOW ANY2 (alpha, vax, i64)
DISALLOW ANY2 (file, image, object) | DISALLOW ANY2 (file, image, object)
DISALLOW (algorithm AND (image OR object)) | DISALLOW (algorithm AND (image OR object))
DISALLOW (object AND (alpha OR vax)) | DISALLOW (algorithm AND (alpha OR vax))
DISALLOW (show AND (alpha OR vax)) | DISALLOW (object AND (alpha OR vax))
DISALLOW (file AND ( headers OR segments OR sections OR | DISALLOW (show AND (alpha OR vax))
DISALLOW (object AND segments ) | DISALLOW (file AND ( headers OR segments OR sections OR
| DISALLOW (object AND segments )
-----------------------------------------------------------------------------------------------------------------------------------

Number of difference sections found: 3
Number of difference records found: 15

DIFFERENCES /IGNORE=(TRAILING_SPACES,FORM_FEEDS,COMMENTS,BLANK_LINES)/COMMENT_DELIMITERS=(EXCLAMATION)/PARALLEL-
DSA1:[GILLINGS]IA64_CHECKSUM.CLD;1-
DSA1:[GILLINGS]ALPHA_CHECKSUM.CLD;1

Re: DCL CHECKSUM sometimes doesn't select correct architecture

John Gillings — Mon, 16 Feb 2009 04:14:48 GMT

Apologies for the mangling ITRC has done the the DIFF output.

Please check your results after CONVERTing your VFC files to VAR and/or STREAM_LF. I seem to recall CHECKSUM having trouble with VFC's in the past.

Re: DCL CHECKSUM sometimes doesn't select correct architecture

x2084 — Mon, 16 Feb 2009 11:18:23 GMT

It's a bug.

The I64 version is written in C and uses C I/O routines. The CRTL when reading VFC files includes the control field in the returned data. That obviously changes the checksum. If you convert your file to VAR you will see identical checksums and they are 1510215701.

This is fixed in 8.3-1H1, but that doesn't help you on Alpha. If you want that fixed in supported Alpha versions, please contact HP.

As a workaround it help you to force the activation of the Alpha image by
$ define/user i64_checksum checksum

I don't want to argue about including the VFC into a checksum or not: Alpha doesn't, so you want to have the compatibility. The XOR algorithm is called a data checksum. It operates only on the data read in with record I/O. You may want to use one of the file checksums, CRC or MD5, which work on all bytes of the file. The MD5 or CRC checksums are known to produce better (in terms of uniqueness) checksums as XOR.

Re: DCL CHECKSUM sometimes doesn't select correct architecture

Benjamin Levy — Tue, 17 Feb 2009 22:02:39 GMT

Thanks to all the responders.

I now clearly understand that this is a bug, and that I either need to work around it or escalate to HP for a fix.

Given the relatively moderate number of command files I would need to edit, I plan to work around this problem by editing the command files.

Converting to use CRC or MD5 checksums clearly has some advantages, but is not at present an option since it would not work on VMS V7.3-2, and it would require a time-intensive conversion of my data files which presently hold decimal checksum values. Same problem applies to conversion of the files from VFC to VAR.

My most likely edit would be to use F$GETSYI to determine the architecture, and then
$ define/user 'arch'_checksum checksum
this would seem to be compatible with all versions and architectures and allow compatibility forward and backward.