Enabled Rdb AIJs causes BAS-F-MEMMANVIO in PowerHouse call to External Product

[Frederick Hoenisch]

After waiting for a patch and upgrading to Oracle's Rdb product to version 7.1-241, we re-enabled After Image Journaling (AIJ) on one of our databases.

When users perform a particular function (called an assessment) from a PowerHouse (PH) screen they would intermittantly get a stackdump error: "BAS-F-MEMMANVIO, memory management violation". This problem appears to increase in frequency as system loading increases.

After some investigating/head scratching we tried turning off AIJs (only thing left that we felt we changed) and the problem stopped.

We've recompiled the PowerHouse code which links to an external product (address/postal code checking software) and get the same problem.

We've opened calls with the three vendors for advise:
COGNOS (PH)
ORACLE (Rdb)
COMDATA (PC Lookup)

COMDATA provided me with some source code with some additional debug checking within the code to help us narrow down the problem. I recompiled it under BASIC 1.5 with the /CHECK=ALL qualifier. The resulting .OBJ was recompiled into the PowerHouse routine and our latest attempt to turn on AIJs yielded the following:

%BAS-F-MEMMANVIO, Memory management violation
-BAS-I-USEPC_PSL, at user PC=80C981A4, PSL=0000001B -SYSTEM-F-ACCVIO, access violation, reason mask=04, virtual address=00000000000
-BAS-I-FROLINSUB, from line 2999 in subprogram MMPREP %TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
0 FFFFFFFF80C9A984 FFFFFFFF80C9A984
DEC$BASRTL 0 000000000000EF9C 000000007C1D8F9C
----- above condition handler called with exception 0000000C: %SYSTEM-F-ACCVIO, access violation, reason mask=04, virtual address=00000000000
----- end of exception message
0 FFFFFFFF800A609C FFFFFFFF800A609C
0 FFFFFFFF80C981A4 FFFFFFFF80C981A4
QKDRIVER MMPREP MMPREP 1218 0000000000005834 00000000001122D4
0 FFFFFFFF80C9A894 FFFFFFFF80C9A894 QKDRIVER DRIVER_ROUTINE call_external
7563 0000000000000664 00000000000506C4 QKDRIVER DRIVER_MAIN driver_mainline
1318 0000000000000048 0000000000050048
0 FFFFFFFF8026FE94 FFFFFFFF8026FE94


Here's an extract of the MMPREP routine from line 2999 (provided with permission from COMDATA):


2999 EXTRA_INFO$ = EDIT$(EXTRA_WORK$,8% + 128%) ! STRIP SPACE
LSET MMISER_OUTREC_NONADDRESS = EXTRA_INFO$
IF TRACE$ = 'Y' THEN
PRINT 'LEAVING MMPREP ';MMISER_OUTREC_NONADDRESS;'+'
PRINT ' ADDRESS 1 ';MMISER_INREC_ADDRESS1
PRINT 'ADDRESS 2 ';MMISER_INREC_ADDRESS2
END IF

IF EXTRA_INFO$ <> SP THEN
LSET MMISER_OUTPAR_ERRORTEXT(0%) = '86 Extra Information'
ELSE
IF EDIT$(MMISER_INREC_ADDRESS1 + ' ' + &
MMISER_INREC_ADDRESS2,16%+32%+128%) <> &
EDIT$(ORIG_ADDR1$,16%+32%)
THEN
LSET MMISER_OUTPAR_ERRORTEXT(0%) = '80 Abbreviation'
END IF
END IF

! RESET PASSING PARAMETERS

INPAR$ = MMISER_INPAR_WHOLE
INREC$ = MMISER_INREC_WHOLE
OUTPAR$ = MMISER_OUTPAR_WHOLE
OUTREC$ = MMISER_OUTREC_WHOLE

GOTO 32767


Note: We've checked quotas as well and they appear to be adequate (as reported from PQUOTA utility).

Typical user's quotas are as follows:
Maxjobs: 0 Fillm: 8192 Bytlm: 1500000
Maxacctjobs: 0 Shrfillm: 0 Pbytlm: 0
Maxdetach: 0 BIOlm: 100 JTquota: 4096
Prclm: 6 DIOlm: 200 WSdef: 1024
Prio: 4 ASTlm: 206 WSquo: 20000
Queprio: 0 TQElm: 50 WSextent: 70000
CPU: (none) Enqlm: 32767 Pgflquo: 1500000

Any thoughts/comments would be appreciated.

VMS 7.3-2
Rdb 7.1-241
PH 7.10G1

The PC_LOOKUP product hasn't had any program updates for a number of years.

Yours truly,
Fred.

[Robert Gezelter]

Fred,

Does the program fault in the same place every time?

- Bob Gezelter, http://www.rlgsc.com

[Frederick Hoenisch]

Yes, but because it is a production application, we didn't have many opportunities to test, before disabling AIJs.

Of the failures - all were at the same point.

[Robert Gezelter]

Fred,

Personally, I would probably set things up so that I could single step the program at a machine code or source code level to determine EXACTLY what call, and what parameter it is having a problem with. As a start, check what is at the address 80C981A4, which would appear to be in system space. On first glance, it seems unlikely that an AIJ issue would produce a synchronous problem, one that occurs at exactly the same place every time.

I would really want to get the debugger on it and see exactly which parameter to which routine is causing the problem. Otherwise, we are working with speculation, not facts.

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

Fred,

A followup thought. A quota related issue could produce a timing invariant problem. In any event, the debugger would allow precise identification of the problem

- Bob Gezelter, http://www.rlgsc.com

[Garry Fruth]

I suggest you add a line number to every statement; this may help to narrow it to a single statement. E.G.


2999 EXTRA_INFO$ = EDIT$(EXTRA_WORK$,8% + 128%) ! STRIP SPACE
29991 LSET MMISER_OUTREC_NONADDRESS = EXTRA_INFO$
29992 IF TRACE$ = 'Y' THEN
29993 PRINT 'LEAVING MMPREP ';MMISER_OUTREC_NONADDRESS;'+'
29994 PRINT ' ADDRESS 1 ';MMISER_INREC_ADDRESS1
29995 PRINT 'ADDRESS 2 ';MMISER_INREC_ADDRESS2
29996 END IF
...

Line numbers do not need to be sequential nor in order; but they do need to be unique. The compiler should let you know about duplicates.

I suspect the accvio occurs in the four lines that "RESET PASSING PARAMETERS". If the calling program passed fixed-length string by descriptor, rather than using dynamic strings (I think my terminology may be a bit off on this), then changing the length of what is passed may not be legal.

[Frederick Hoenisch]

Thank you for the responses thus far. I failed to mention that our attempts to duplicate the problem in our non-production environments all failed.

Because the problem only shows itself when the system is busy, I have to discuss with the Application Manager the possibility of trying to debug the problem during prime time (this is unlikely).

PowerHouse has a debugger, but we're not sure (at this point) if it will also debug the external calls (BASIC code)? Something to try and experiment with over the holidays I guess.

[Volker Halle]

Fred,

consider to issue a SET PROC/DUMP/ID=xxx against a process running the BASIC image, before invoking the problematic user function. Once the ACCVIO happens, you'll get a process dump (SYS$LOGIN:imagename.DMP), which can be analyzed by ANAL/PROC. You can do analysis of the problem in a non-realtime environment first. You can find the failing instruction and the call stack leading to the problem and look at the memory contents being referenced etc.

To make sure, that a complete process dump (including process-pertinent data from system space) can be written, you may want to (temporarily) grant the IMGDMP$READALL right to the user running the application.

Trying to analyse the problem in 'real-time' may be different, especially if the ACCVIO is dependant on system load. You'll never know, whether you'll hit the problem. It may even be impossible to reproduce it with running the debugger.

Volker.

[Volker Halle]

Fred,

could you also try the following, please:

If we believe the PC value reported by BAS-I-USEPC_PSL, you can issue the following SDA command on the system, where this problem had happened:

$ ANAL/SYS
SDA> EXA/INS 80C981A4

As the RM (Reason Mask) has been reported as 04 (=WRITE), the DESTINATION address pointed by the instruction producing the ACCVIO, must point to non-writeable memory.

SDA> MAP 80C981A4

may also tell, which execlet/image/library this code is in.

If the reported instruction, registers etc. make sense, try SDA> EXA/INS 80C981A4-20;20 to find out, where the 'invalid' value (probably 00000000, which is the failing VA reported in the ACCVIO) in the register may have been loaded from.

Volker.

[Volker Halle]

Fred,

from experimenting with a little BASIC test program:

0 FFFFFFFF800A609C FFFFFFFF800A609C <<< SYS$CALL_HANDL_C+0002C
0 FFFFFFFF80C981A4 FFFFFFFF80C981A4 <<< failing instruction
QKDRIVER MMPREP MMPREP 1218 0000000000005834 00000000001122D4

Source code line 1218 in the MMPREP module seems to be the last line executed in this module before calling a routine in system space, which takes the ACCVIO at PC 80C981A4. Finding this line requires a source code listing from the version of the module running at your site.

PC 1122D4 must be a return address (following a JSR instruction).

Taking a process dump from a failing BASIC program works fine. With the process dump, you have all the information you need to figure out, exactly which instruction is failing due to which invalid address in which register.

Volker.

[David B Sneddon]

Fred,

As a longtime user of BASIC, my first instinct on
the info you supplied would be to increase pgflquo
(try doubling it).

Regards
Dave

[Frederick Hoenisch]

Thanks again All:

The next opportunity to test is this weekend and the first 'load' test is Monday morning.

I'll give it a go and let you know. Thanks for the advice. For this next test, I intend to:

1. Double PGFLQUO for all users of the app.
2. Generate PROCESS dump files.
3. Add more line numbers to MMPREP.

Yours truly,
Fred.

[Ian Miller.]

could there be an issue with the location of the AIJ which is badly handled and leads to the ACCVIO. Is the AIJ file placed on a disk on which there is sufficent free space and that the file and the file and directory protection are correct?

[David B Sneddon]

Ian,

I think the AIJ stuff probably requires some memory
that is blowing the pgflquo value. I have seen our
developers make "small" changes like allocating a
new 10000 element string array and suddenly the
application falls over with ACCVIO errors. Increasing
pgflquo makes it go away. The fact that Fred can't
reproduce it in a test environment suggests that
the quotas in the test environment are probably
different to the production environment.

Dave

[Ian Miller.]

Dave, you are correct that quotas are the first thing to suspect. I was just wondering about other possible causes.

[David B Sneddon]

Yes it may be something else but all MEMMANVIO
errors I have seen over the years have been either
pgflquo issues or recursive/circular calls not
terminating (ending up in stack overflows).
Failure to create/rename files due to insufficient
contiguous space for directories usually manifest
as ACP type errors.

Dave

[David B Sneddon]

Having just re-read the original and the comment
about things breaking when the load increases,
is it possible that there is insufficient pagefile
available? Are there now more users than there
used to be?

[Jan van den Ende]

Fred,

Having read and re-read the whole stream, I am still quite in doubt whether this is to the point for you or not, but since I found no definitive arguments to rule it out, I will just relate our expirience (and workaround).

We first stumbled on it when we tried to implement Clusterwide Logical Names.

In ANY area definition in Rdb (and also DBMS), internally the logical names are evaluated up to the level where a physical or a Concealed Device appears. A Concealed device is then tested to be valid - explicitly - in LNM$SYSTEM_TABLE; EXECUTIVE mode.
Should somehow a reference be made to a data area that does NOT conform, then an Access Violation is generated. How that would trickle down trough Powerhouse is unknown to me.
Oracle was able to confirm our findings (back in febrary 2000), and promised support for clusterwide tables 'in the next release'. We have since been finding it not yet repaired in any following release...

As I started this post with, it might have nothing to do with your problem, but it might well be worth checking.


Proost.

Have one on me.

Jan