How to debug a memory resident driver written in Macro32?

[XMAN_1]

Hello! Guys,

I am trying to debug a memory resident driver written in macro32. Its loading Ok using SYSMAN but crashing when doing SYSMAN IO CONNECT.

Note:Please also see the attached Clue Crash-01.txt file for more information.

Best Regards!

[Robert Brooks_1]

Did this driver work on a previous version of OpenVMS Alpha?

[John Gillings]

XMAN,

The bugcheck code is correct, your stack is invalid. The stack address in the dump is in S0 space, it should be in P1 space.

Best guess is some of your driver code has corrupted a stack frame, which has resulted in restoring a bogus stack pointer address on return.

Tricky to debug. I'd start with a careful code inspection of the IO CONNECT path - look at argument lists in and out, and also at stores to stack local storage. Perhaps there's an assumption about the size of a data structure that's been violated?

[Volker Halle]

XMAN,

try SDA> CLUE CALL to get a summary of the call frames. Or SDA> SHOW CALL and a couple of SHOW CALL/NEXT

The KSP - as saved in the HWPCB - is reported as FFFFFFFF.FFFFFFC0 - clearly invalid (if that saved value is true).
You may want to look around on the real KERNEL stack for a longword of FFFFFFC0. You can find the stack address/limits with:

SDA> EXA ctl$aq_stack;20 ! bottom of stacks
SDA> EXA ctl$aq_stacklim;20 ! top of stacks

The first quadwords shown are the allowed range of the KERNEL stack.

There is also a return address in R26, try
SDA> EXA/INS 7FFA1ED0-20;30

You need to try to determine the context of the routine being executed immediately before the crash. Sometime the register contents are helpful:

SDA> FORMAT 80D47700

Try to provide the data as an attached .TXT file and we'll see what can be done next...

Is this a new driver, did it ever work before ?

Volker.

[John Travell]

Halt Registers saved in Process HWPCB (by console):
KSP = FFFFFFFF.FFFFFFC0
I am trying to remember if this value is wholly trustworthy, or whether the console has stepped on it by the time we get to see this content. I think it is good data, but in this context utterly invalid as an SP.

How do we get to __RELEASE_KERNEL_SERVICE: ? It does not appear to be through EXE$CMODKRNL_C.
Note the following comments on the previous instructions.

R5 = 00000000.00000088
R7 = 00000000.00000038
R22 = 00000000.00000000

EXE$CMODKRNL_C+0019C: LDA R22,#X0800(R31) < Put 800 into R22
EXE$CMODKRNL_C+001A0: BIS R7,R31,SP < zap stack pointer
EXE$CMODKRNL_C+001A4: AND R5,R22,R22 < 800 AND 88 do not make zero
EXE$CMODKRNL_C+001A8: BEQ R22,#X000001 < branch because R22 is zero
EXE$CMODKRNL_C+001AC: BSR R4,#X000298 < not executed
__RELEASE_KERNEL_SERVICE: STQ R6,#X0030(SP) < die horribly. BUT...
the SP is inconsistent with having been written at EXE$CMODKRNL_C+001A0, and other register contents are also inconsistent with executing that code stream.

The implication is that there is a branch or jump, either to EXE$CMODKRNL_C+001A8:, or to __RELEASE_KERNEL_SERVICE: somewhere else, either in EXE$CMODKRNL_C, or in the loaded driver.

You probably need to learn, and use, Xdelta.

[Volker Halle]

XMAN, John,

the code, which has reported the KRNLSTAKNV crash, is the system service completion code (SERVICE_EXIT), which gets executed, when a system service completes. And the code stream all makes sense...

R7 is supposed to hold the inner mode SP

AND R5, R22, R22 will return R22=0 (with R5=88 and R22=800 to begin with)

so the branch is taken and we crash on the STQ instruction.

I'm pretty confident, that CLUE CALL will show more information and allow to follow the code flow until the time of the crash.

Let's wait and see...

Volker.

[John Travell]

Please excuse my brain fade, it is AND not ADD... There are no bits set in both 800 and 88.
CLUE tells us that R7 contains 00000000.00000038
The KSP in the Process HWPCB is FFFFFFFF.FFFFFFC0
The BIS R7,R31,SP will copy the entire content of R7 to the SP, and I cannot see at this moment how we come by the 78 difference.
The STQ R6,#X0030(SP) will try to write to SP+30, which will be either 00000068 or FFFFFFF0 depending on whether the KSP value in the HWPCB is a valid representation of the KSP at the time of the STQ, or has been modified by the console. (Maybe R7 has been changed instead!)
Perhaps I have forgotten too much in the last few years, but this looks to me as though the value in R7 is not what it should have been.

[XMAN_1]

Hello! Gurus,

Please do find the attached SDA text output's from the original source. I have revoked my changes and create a new build from the original code. This driver is a step-2 driver and was running flawlessly on 6.2-1H3 and 7.0.

Regards!

[John Travell]

Hmmm. A rather different crash to the KRNLSTAKNV you first reported.
This could be difficult to trace without the listings for this driver. What is the label you have at PRADCH$PRDRIVER+05C68, and where in your driver do you invoke that label as a the destination of a jump or branch ?
There is a good chance that immediately prior to PRADCH$PRDRIVER+05B90 you have a JSR R26(R26) to that label. If so, please look carefully at why that JSR is taken, and consider what could cause it.
In any case, please show us as much of the code as you can, especially around that area, either from the listing file, or from SDA.

[XMAN_1]

Hi! John,

Check the attached crash dump analysis.

Regards!

[Arch_Muthiah]

Xman,

There are some unsupported third party layered drivers will load without any problem, but we can do very limited operations on that.

Apart from this I doubt the SYSMAN commands and syntax you gave may be the cause for this crash.

Because during sysman IO connect command, the system will do some device config checking. if we don't give the correct device name in the SYSMAN IO CONNECT command, the "I/O database" will be damaged. I suspect that is what happened now for this "Inconsistent I/O data base" BUGCHECK.

Archunan

[Arch_Muthiah]

XMAN,

I doubt the SYSMAN command before analyzing your BUGCHECK details.

We knew there are two config table files, SYS$SYSTEM:SYS$CONFIG.DAT is for VMS suported devices and SYS$SYSTEM:SYS$USER_CONFIG.DAT file for all third-party, layered-product, and user-written device drivers. The device info passed in "SYSMANN IO connect" should match the info on these files. If not, the IO database will get damage and the bugcheck.

So before configuring any device using SYSMAN, better use SYSMAN rebuild command, which actually to rebuild the configuration tables attached to each of the adapter blocks by re-reading and parsing The REBUILD command will always rebuild all of the configuration tables, regardless of the type of bus.

You can make sure the device details using
SDA> CLUE config
then you can give SYSMAN commands with correct device details.

Archunan

[Volker Halle]

XMAN,

this INCONSTATE crash is what's called an INLINE bugcheck from your driver. The driver is doing some consistency check and decides to immediately crash the system, if this check fails.

The machine code you've shown from your source listing does NOT match the code stream from the crash ! You need the machine code listing from exactly the same version of the driver which was active when the system crashed. Or you need to look around at similar offsets for the same instruction stream.

There is a source code line

47944 bug_check (ERRHALT, FATAL, COLD);

ERRHALT is not INCONSTATE, so look for a bug_check(INCONSTATE,...) in your source code.

The crash happens in SYSTEM context (INT-bit in PS is set), but seems to be clearly related to the most recent SYSMAN command issued and it happens IN your driver.

If you look further down the stack, you see (bottom to top) EXE$CMKRNL_C calling IOGEN$SHARE, later EXE$PRIMITIVE_FORK_C calling your driver.

Believe me, there is nothing wrong with your IO database and probably also with your SYSMAN commands. There is some problem in your driver and you just need to analyze and fix that.

You didn't answer the question, if that driver has ever worked before on the same or older version of OpenVMS and/or a similar HW config.

Volker.

[XMAN_1]

Hi! Volker,

This driver was working fine on same machine with same config running on O/S OpenVMS 6.2-1H3 and 7.0. But when installed on 7.2-1, it starts crashing.

My first submission is from the source code where I made some changes like replacing the obsolete system calls with the new system calls.

Then I revoke my changes and rebuild the new binary files using the original driver code and submitted again in John's response.

The strange thing is its not giving me any errors while rebuilding process, so in my view it looks like a data/structure or data corruption error. Its just my view :)

Note for all Guru's:
Guys, Do you want me to close and reopen this thread again with the latest crash dump analysis as it looks like the 1st one is causing confusion?

Let me know guys!

And many thanks to all of you for your help, support and taking time in analyzing this problem!

You guys ROCK!

[Volker Halle]

XMAN,

I don't mind to continue looking at further crashes in the context of this topic. It's all the same problem anyway, just the crash symptoms may differ...

I assume you are aware, that crash analysis using a forum like this may not be a very effective way, but it works - I can tell from experience.

A couple of things have certainly changed in OpenVMS between V6.2-1H3 and V7.2-1. Try to find all intermediate new features manuals and release notes and read through them regarding any notes about user-written drivers.

Then try to diagnose and fix each crash step by step...

Volker.