topic Re: Ooops, I needed to change the transfer vectors. in Operating System - OpenVMS

Setting transfer vector to not activate debugger for Alpha image (.EXE)

GuentherF — Thu, 01 Sep 2011 17:28:44 GMT

I am looking for that procedure which allows me to change the transfer vector for an Alpha image so that an image linked /DEBUG would not start in the debugger.

Thanks,

Guenther

P.S. If this forum software would be NOTES I would have found it already. Sigh!

Edited to state correctly what I 'really' wanted.

Re: Setting DEBUG flag for Alpha image (.EXE)

Hoff — Thu, 01 Sep 2011 17:16:42 GMT

http://labs.hoffmanlabs.com/node/848

http://www.mppmu.mpg.de/~huber/vmssig/src/C/DEBUG_TOG.C

Given tthat this "forum software" is "well written" and "thoroughly tested" and "chosen specifically for customer needs" and "not prone to corrupting text", it seems mysterious that I cannot include source code.

Edit: I have posted a Stream-LF copy of the DCL procedure accessible at the first URL listed above, and have also added a link over to Hein's original posting from the comp.os.vms newsgoup.

Ooops, I needed to change the transfer vectors.

GuentherF — Thu, 01 Sep 2011 17:33:00 GMT

Thanks Hoff!

I made a mistake in my original post: Actually I wanted to change the transfer vector only so an image linked /DEBUG would not start in the debugger.

I found a proprietary procedure which does what I want. But I can't post it here. Maybe someone else has a tiny DCL procedure that is public for future generations?

Thanks,

Guenther

Edit changed the subject in my original post.

Re: Ooops, I needed to change the transfer vectors.

Hoff — Thu, 01 Sep 2011 17:53:56 GMT

A transfer vector patch tool and another debug-flag tool are available in this comp.os.vms thread:

https://groups.google.com/d/topic/comp.os.vms/V2B-4u7pq2A/discussion

Re: Ooops, I needed to change the transfer vectors.

GuentherF — Thu, 01 Sep 2011 18:20:34 GMT

Thanks Hoff!

That is a good pointer. It always is with Norm's name showing up. :-)

/Guenther

Re: Ooops, I needed to change the transfer vectors.

Jur van der Burg — Thu, 01 Sep 2011 18:54:23 GMT

This is what I use.

Jur.

$ if p1 .eqs. "" then inquire p1 "Filename"
$ on error then goto end
$ open/read/write file 'p1'
$ read file data
$ if f$cvsi(0,32,f$extract(112,4,data)) .ne. %x340 then goto move
$ if f$cvsi(0,32,f$extract(116,4,data)) .eq. %xffffffff then goto update
$! No previous entry for debugger in vector, insert one
$move:
$ data[8*132,32] = f$cvsi(0,32,f$extract(124,4,data))
$ data[8*128,32] = f$cvsi(0,32,f$extract(120,4,data))
$ data[8*124,32] = f$cvsi(0,32,f$extract(116,4,data))
$ data[8*120,32] = f$cvsi(0,32,f$extract(112,4,data))
$ data[8*116,32] = %xffffffff
$ data[8*112,32] = %x340
$update:
$ flags = f$cvsi(0,8,f$extract(80,1,data))
$ data[8*80,8] = (flags/2) * 2 + 1
$ write/symbol/update file data
$end:
$ if f$logical("file") .nes. "" then close file
$ exit

Re: Ooops, I needed to change the transfer vectors.

John Gillings — Fri, 02 Sep 2011 00:45:03 GMT

Just for completeness and future generations... patches like this are only necessary on VAX or Alpha. On IA64 systems, there's a DCL command to turn image flags on or off. One wonders why they didn't make it work on Alpha too :-(

$ HELP SET IMAGE

SET

  IMAGE

    /FLAGS

          /FLAGS=(keyword[,...])

       Specifies which image attribute flags to turn on or off. The
       original flags are set by the OpenVMS I64 Linker at image
       link time. The possible keywords are listed below with a brief
       description. For more information about the image link flags, see
       the HP OpenVMS Version 8.2 Release Notes.

                                    WARNING

          Beware of modifying the flag values unless you are very
          knowledgeable about the internals of the image.

       Keyword          Description

       [NO]CALL_DEBUG   Call Debugger at startup.
       [NO]DBG_IN_DSF   Debug records in debug symbol file.
       [NO]DBG_IN_IMG   Debug records in image file.
       [NO]EXE_INIT     Image has a pointer to EXE$INITIALIZE.
       [NO]IMGSTA       Call SYS$IMGSTA.
       [NO]INITIALIZE   Image has a pointer to LIB$INITIALIZE.
       [NO]MAIN         Image has a main transfer address.
       [NO]MKTHREADS    Enable multiple kernel thread use.
       [NO]NOP0BUFS     No P0 buffers for RMS image I/O.
       [NO]P0IMAGE      Image is loaded only to P0 space.
       [NO]SIGNATURES   TIE Signatures are present.
       [NO]TBK_IN_DSF   Traceback records in debug symbol file.
       [NO]TBK_IN_IMG   Traceback records in image file.
       [NO]UPCALLS      User thread upcalls are enabled.

(re Hoff and inserting code... Although I agree with most of your opinions about this woeful interface, one thing that is an improvement for me is posting code examples. I have an icon called "Insert code" which brings up a dialog box - copy and paste even works! - maybe it's Internet Explorer on Windows only?)

Re: Ooops, I needed to change the transfer vectors.

H.Becker — Fri, 02 Sep 2011 07:07:36 GMT

>>> One wonders why they didn't make it work on Alpha too :-(

"They" made it work on Alpha, too :-) But there was no project, enhancement request,funding to make it work for Alpha images. On the other hand, the VMS extensions for ELF specify some structures to mark an image as manipulated and to save the original settings. There is no such thing in the Alpha Object/Image specification. "They" felt that there should be at least a flag to show that someone manipulated the image with the supported tools, that is "set image".

Re: Ooops, I needed to change the transfer vectors.

Hein van den Heuvel — Sun, 04 Sep 2011 02:38:03 GMT

>> "They" felt that there should be at least a flag to show that someone manipulated the image with the supported tools,

1) Couldn't the presence of debugger records in the image serve as a flag that it must have been manipulated?

2) If a feature goes against the grain of proper engineering, but is likely to please some customers, then IMHO one should simply add a /FORCE or /PRETTY_PLEASE_WITH_A_CHERRY_ON_TOP option where the help indicates reservations about its usage.

fwiw,

Hein

Re: Ooops, I needed to change the transfer vectors.

Hoff — Sun, 04 Sep 2011 13:29:30 GMT

1) Couldn't the presence of debugger records in the image serve as a flag that it must have been manipulated?

Could be.

Tracking the integrity of operating system has been an issue for eons and for all operating systems, and long before SET IMAGE and related commands arived in VMS, and there have been previous products and implementations. (Applications have similar issues, of course.)

"They" implemented portions of a scheme for integrity-detection logic for the files comprising VMS itself, and the first parts were deployed in OpenVMS V8.2. See SYS$UPDATE:VMSKITBLD.XSD for some related details. (In retrospect, "they" would have found MD5 as part of this implementation would be replaced with a newer SHA scheme, but the intentional use of XML allowed "them" to make these sorts of changes.) "They" also knew there was more work involved than that tracking, of course. Modifying the image headers to flag these changes is possible, but not robust against nefarious changes. Given the flags for the image are themselves at risk of modification, this gets into the discussions of digital signatures and of digitally signing files, too.

As for the question here, it wouldn't be at all difficult to activate a simple (and an "insecure") SET IMAGE tool or analogous on OpenVMS Alpha and even over on OpenVMS VAX. The Alpha and VAX image headers are comparatively simpler than the ELF headers, and the work involved is already known and discussed here. (But given that the tools are already available, this DCL change would only provide an easier UI and a way to avoid fetching (or writing) the tools, and (the costs) would increase the testing and support requirements for any future releases of VMS.)

Re: Ooops, I needed to change the transfer vectors.

H.Becker — Mon, 05 Sep 2011 11:45:43 GMT

... and you can always link/dsf to run the image by default without the debugger but to have the full debug support on request when running it with /debug.