RDB/SECURESHR image activation problem with "kept" debugger & heap analyzer

[Mark Corcoran]

I wonder whether or not anyone else might have encountered this issue, and if so, whether or not it is something that can (readily) be fixed?

I've been trying to assist in debugging a memory leak issue in a number of processes.

On the development system, a debug version of one of the executables has been created.

If I run this /DEBUG in the "normal" debugger, I can hit G at the DBG prompt, and eventually the program springs into life.

However, in order to try and track down the memory leak, I want to use the Heap Analyzer.

It took me a while to work out how to do this -my reading of the debugger manual indicated that a server process needed to be created on the VMS node, with special debugger PC client software extracted from a saveset and installed on the PC (I later found out about SET DISPLAY /CREATE /NODE=pc_ip_addr /TRANS=TCPIP).

I can get the debugger display to be exported to the PC (DEBUG /KEEP after the SET DISPLAY), then the Heap Analyzer window opened up when I issue RUN /HEAP image.exe at the DBG prompt).

I click on the Start button on the H.A. window, and a lot of updates then take place to the screen.

I think these are memory allocations from various shareable images, before the code's main() function is called, mainly because I see very very fragmented allocation of memory by pthreads/DECthreads, and eventually at the debugger window I get %DEBUG-I-INITIAL and %DEBUG-I-NOTATMAIN messages).

I enter the GO command, and some more updates take place on the Heap Analyzer window, whilst the debug window shows Call Stack 0, LIB$INITIALIZE.

Eventually, the debug window shows a break(point) in the main() function, so I type GO again.

Now the program starts to perform its own initialisation, and using SMG draws outline boxes on my original telnet session that I issued the DEBUG /KEEP on.

Finally, the debug window shows that the program has handled an exception, and returned to the DBG prompt.

Looking in the log file that the process writes to, it shows the following messages:

%RDB-E-UNAVAILABLE, Rdb/Dispatch is not available on your system
-RDB-I-TEXT, Error activating image DSA20:[SYS1.SYSCOMMON.][SYSLIB]SECURESHR.EXE
, Shareable images must be installed to run privileged image
%DBAT-I-M010ATTDBSERR, Set Transaction Error - Attaching to database
%RDB-E-UNAVAILABLE, Rdb/Dispatch is not available on your system
-RDB-I-TEXT, Error activating image DSA20:[SYS1.SYSCOMMON.][SYSLIB]SECURESHR.EXE
, Shareable images must be installed to run privileged

I'm not sure if the DBAT is an RDB facility whose message is being output by the .EXE, or if it is the .EXE's own, and indicating in which "subsystem" an error occurred.

I'm a bit confused by the reference to SECURESHR.EXE though...

SECURESHR *is* an installed image:

$ pipe install list |sea sys$Input secureshr
SECURESHR;1 Open Hdr SharAddr Lnkbl Resid

SECURESHRP;1 Open Hdr SharAddr Prot Lnkbl Resid

SECURESHR_JACKET;1

After having had a quick rummage through comp.os.vms, it occurred to me that it might be a privilege issue, so I looked at the account settings in SYSUAF (not my own).


Authorized Privileges:
CMKRNL GROUP GRPNAM NETMBX PRMGBL SECURITY
SYSGBL SYSLCK SYSNAM TMPMBX WORLD
Default Privileges:
GROUP GRPNAM NETMBX PRMGBL SYSGBL SYSLCK
SYSNAM SYSPRV TMPMBX

I tried tearing everything down, then (for the purposes of testing) changing the account privs and defprivs to ALL.

I logged back in the modified account, used the SET DISPLAY, DEBUG /KEEP etc., and tried again.

Still no joy, exactly the same problem.

FWIW, version information:

$ @sys$library:rdb$shover
Current PROCESS Oracle Rdb environment is version V7.0-21 (MULTIVERSION)
Current SYSTEM Oracle Rdb environment is version V6.1-04 (STANDARD)
$ write sys$output f$getsyi("VERSION")
V7.2-1

Given that I don't have this problem when using the "normal" debugger, I would guess that this is some kind of "kept" debugger vs RDB (RTL) shareable image interaction issue.

If it is a version compatability issue, then it would help as ammunition to justify the case for upgrades (some other software that has to be upgraded across multiple platforms types has failed 4 times so far; there's even more reluctance to upgrade "base" (O/S and layered product) software as a result).

Any thoughts, anyone?

[Richard J Maher]

Hi Mark,

Rdb is known to give that/similar error when an executable has been installed with privileges and Rdb is trying to call out to a shareable image for an external function/procedure. (This is a completely erroneous restriction on Rdb's part but that's besides the point)

The usual "solution" is to give the user the matching privileges as those the image is installed with, but in this case you say the user was given default ALL privs (BTW does ALL grant SECURITY?)

What is the image being run and do you have a list of privs it was installed with?

Cheers Richard Maher

[John Gillings]

Mark,

PRIVINSTAL messages are often inconsistent and confusing (I think it may be deliberatly obsuring useful information to make life hard for people trying to crack systems). For some reason SECURSHR and SECURSHRP get mentioned when it's actually another image entirely.

The logic here is that a privileged image is being activated dynamically, but, since you have DEBUG control, there may be a way to exploit the privileges. As Richard has suggested, give privilege to the process running the image, I'd also give them to the process running DEBUG for good measure.

Tracking the image can be tricky, try:

$ SET WATCH/CLASS=MAJOR

This may give a better indication. An even bigger hammer is to turn on the UAF AUDIT flag on the username running the image (but expect LOTS of output in the security journal!)

If you have all privileges, another approach to this type of problem is to try to avoid activating any installed images. Define logical names for everything you're likely to activate, including a version number, for example:

$ DEFINE LIBRTL SYS$SHARE:LIBRTL.EXE;

This will bypass the installed image. It may not work for all types of installed image, it will also completely change the memory layout of your running program, as RESID images will now be mapped to process space. This will probably affect your memory leaks.

(Lucky you have Richards attention, he has lots of experience debugging protected images, and running up against the arcane and confusing rules surrounding them)

[Mark Corcoran]

Richard: Yup, I granted all privs to the account.

The account already had the SECURITY priv, but MOD /PRIV=ALL and /DEFPRIV=ALL does give SECURITY, even if the account doesn't already have it.

As regards what is the image being run and what privs is it installed with, I wonder if we're talking at cross-purposes...

An executable compiled and linked /DEBUG (and possibly other qualifiers, I don't know) was built by our off-shore suppliers.

Due to firewall restrictions, they cannot export the debugger display to their PCs, so I'm logging in to one of the developers' accounts, and running the executable for him (within the confines of the kept debugger and heap analyzer).

The executable itself is being run sort of in the same way as you might run an image - i.e. $ RUN FRED.EXE but within the kept debugger.

The exectuable is making some RDB call (unfortunately, not clear what, because the debugger can't see the sources; not sure if this is because logicals are not defined or are defined but point to the wrong location, or what).

The RDB call appears to encounter an exception, and the exception handler writes into the log file evidence suggesting it is a shareabile image access issue.

This is what refers to SECURESHR.EXE, and looking in VMSIMAGES.DAT, SECURESHR.EXE isn't specifically installed with any privileges:

/OPEN /HEADER /SHARED=ADDRESS_DATA /RESIDENT

Are you saying that it *should* in fact be installed /PRIV=(something_or_other) ?


John:
After modifying the account in SYSUAF to /PRIV=ALL/DEFPRIV=ALL, I've then logged into it, and run up the debugger, exporting the display to an X-term server on my PC, then issued RUN /HEAP debug_version_of_the_image.EXE

I'd already tried turning the AUDIT flag on for the account, in the hope that it might gave an indication as to some failure, but I couldn't see anything from the audit journal where the status was anything other than %SYSTEM-S-NORMAL).

I've tried the SET WATCH this morning (I was aware of it, had not thought to/forgotten to do this in my earlier tests).

Again, nothing really obvious, though whether or not the fact that the debug image being run uses SMG$ routines, and maybe results in me missing some of the SET WATCH output (because of scrollports etc.), I don't know.

If I bypass the installed SECURESHR.EXE (DEF /PROC SECURESHR SYS$SHARE:SECURESHR.EXE;1), the image being run in the debugger still indicates in the log file:

%RDB-E-UNAVAILABLE, Rdb/Dispatch is not available on your system
-RDB-I-TEXT, Error activating image DSA20:[SYS1.SYSCOMMON.][SYSLIB]SECURESHR.EXE
, Shareable images must be installed to run privileged image
%DBAT-I-M010ATTDBSERR, Set Transaction Error - Attaching to database
%RDB-E-UNAVAILABLE, Rdb/Dispatch is not available on your system
-RDB-I-TEXT, Error activating image DSA20:[SYS1.SYSCOMMON.][SYSLIB]SECURESHR.EXE
, Shareable images must be installed to run privileged


[Clutching at straws, when I defined SECURESHR to point to SYS$SHARE:SECURESHRP.EXE instead of SYS$SHARE:SECURESHR.EXE, the image reported in the log file stops being SECURESHR (or in this case, SECURESHRP), but RDMSHRP70.EXE instead]

[H.Becker]

>>>
-RDB-I-TEXT, Error activating image DSA20:[SYS1.SYSCOMMON.][SYSLIB]SECURESHR.EXE
, Shareable images must be installed to run privileged image
<<<

It is an RDB message. The VMS PRIVINSTALL text is just "shareable images must be installed to run privileged image". You may see this at main image activation time as

$ run m
%DCL-W-ACTIMAGE, error activating image F
-CLI-E-IMGNAME, image file SEMMEL$DKA0:[HARTMUT]F.EXE;1
-SYSTEM-F-PRIVINSTALL, shareable images must be installed to run privileged image
$

There is no problem with the filename in this error message. Once F is installed and a logical in exec mode points to it, the main program can be activated.

It looks like RDB dynamically activates a protected shareable image. It's very likely installed /open, so with SET WATCH/CLASS=MAJOR you will not see it.

I don't know anything about the heap analyzer. But I assume you have to define at least a couple of logicals pointing to images. Is there a logical for SECURESHR? make sure it is defined exactly as the file spec of the image which is installed, and that it is defined in exec mode.

Other than that, I would check with Oracle/RDB support.

[Richard J Maher]

HI Mark,

Installing a shareable image with privs is useless. What I asked to see and you don't need any source code to obtaining is the full install listing of the image your're running. If it is a special debugger that I have not used that runs as the main executable and then somehow bootstraps to the target image then by all means show the install listing for the debugger as well.

Another thing that I'm sure you can do without the source is go into the database and do a SQL> SHOW FUNCTIONS or SHOW PROCEDURES. My guess is that there is an external function or external procedure in there that is calling out to a shareable image (may be as part of a virtual (COMPUTED BY) column). Again, just guessing but Rdb is known to place additional bullshit restrictions on shareable activation (over and above VMS's) but do make sure that all target shareables are in fact installed.

Cheers Richard Maher.

[Mark Corcoran]

>I don't know anything about the heap analyzer. But I assume you have to define at least a couple of logicals pointing to images.

According to the debugger manual, there's 3 ways of starting the Heap Analyzer, and of these, only one requires defining a logical (for LIBRTL, to point to an instrumented version of it), - and that is only for when the image is a shareable image.

The image being debugged isn't a shareable image; it's a normal image that you just RUN, but it makes an RDB call which fails, and the error information returned appears to suggest that the SECURESHR.EXE shareable image is the one with access issues.


>Is there a logical for SECURESHR?
Not normally, and according to the debugger manual, not required.


>Other than that, I would check with Oracle/RDB support.
A colleague is going to (belatedly) do that this morning.


>What I asked to see and you don't need any source code to obtaining is the full install listing of the image your're running

I think we must be talking at cross-purposes - the image I'm running isn't an installed image...

It's you bog-standard RUN image.EXE sort of image, but which makes calls to RDB, to access database tables.

It would appear that the attempt to connect to the database, is failing, and any secondary status returned by RDB, is indicating that it has a problem activating the SECURESHR.EXE installed image.


>If it is a special debugger that I have not used that runs as the main executable and then somehow bootstraps to the target image then by all means show the install listing for the debugger as well.

Analysing the channels open by a process which enters the command DEBUG /KEEP at the DCL prompt, shows that it accesses:

SYS$LIBRARY:DEBUGSHR.EXE
SYS$LIBRARY:LIBOTS.EXE
SYS$LIBRARY:LIBRTL.EXE
SYS$LIBRARY:LBRSHR.EXE
SYS$LIBRARY:DPML$SHR.EXE
SYS$SYSTEM:DCL.EXE
SYS$LIBRARY:DCLTABLES.EXE
SYS$LIBRARY:CMA$TIS_SHR.EXE
SYS$LIBRARY:DCXSHR.EXE
SYS$LIBRARY:DECC$SHR.EXE
SYS$LIBRARY:SMGSHR.EXE

You did ask for the full install listing of the debugger, but I'm not sure if you simply meant (in this case) DEBUGSHR.EXE or all of the other listed images.

I'm hoping DEBUGSHR.EXE will suffice...

xxxxx> install list /full sys$library:debugshr.exe

DISK$SYS_xxxxx:.EXE
DEBUGSHR;1 Open Hdr SharAddr Lnkbl
Entry access count = 1808
Current / Maximum shared = 6 / 15
Global section count = 5
Resident section count = 0000



>My guess is that there is an external function or external procedure in there that is calling out to a shareable image

I'll reserve judgement on this, because I'm not sure why it should be a function or procedure, rather than just an ESQLC call to an RDB/SQL function.

My RDB colleague is going to raise this with Oracle this morning, although I have a suspicion I know what they are going to say.

If I get anything meaningful from them, I'll update here.

On the other hand, if you think there's anything else I can provide here that might assist, I'd be only too happy to do so.

[H.Becker]

>>>
The image being debugged isn't a shareable image; it's a normal image that you just RUN, but it makes an RDB call which fails, and the error information returned appears to suggest that the SECURESHR.EXE shareable image is the one with access issues.
<<<
Which matches the initial description that the problem shows after you enter GO at the initial breakpoint in main. At the breakpoint, the image activation of the main image and all shareable images it depends on is already done.

Again, to me it looks like rdb activates another image. (I have no idea whether they use lib$fis or any other undocumented procedure.) That activation fails and rdb tells you that SECURESHR.EXE is not installed. I suspect rdb activates a protected shareable image, so I would try the documented (12.1 Starting a Heap Analyzer Session):

$ DEFINE/EXEC/NAME=CONFINE LIBRTL SYS$LIBRARY:LIBRTL_INSTRUMENTED

before starting the main image.

[Richard J Maher]

Hi Mark,

> On the other hand, if you think there's
> anything else I can provide here that
> might assist, I'd be only too happy to
> do so.

If the *executable* image being run is not a privileged image in the sense that it is not INSTALLed with privileges (even if they be /PRIV=NOALL) then this is indeed different.

I'd still like to see the results of
SQL> SHOW FUNCTIONS
SQL> SHOW PROCEDURES
on the database(s) in question. This will include simple stored SQL functions as well as any EXTERNAL 3GL code calls. If you can wittle the list down to just the external ones then you can do a SHOW FUNCTION MY_EXTERNAL and see what shareable image it invokes and what Logical Name translation it uses.

Yes Rdb uses LIB$FIS and the image-activator rules for logical names and installation etc are obviously followed but, once again, I am surprised that the mainline executable image is *not* installed with privs. How about showing us the $run myimage command followed by a $show log myimage and an $install list/all myimage

Cheers Richard Maher

[Richard J Maher]

btw, SYS$SHARE:RDB$COSIP.EXE is a known vector for SECURESHR.EXE and SECURESHRP.EXE and IIRC (electricity has gone up 50% here and I don't just leave my VMS boxes running any more :-) it is a UWSS that does the username/paswword checking and should be installed protected by Rdb system (or monitor?) startup.

Either way, make sure all 3 of the above are installed.

Cheers Richard Maher