topic Re: ACCVIO in LIBRTL while calling malloc() in Operating System - OpenVMS

ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Mon, 03 Mar 2008 15:42:55 GMT

Recently I have ported my application from using 32bit pointers to 64bit pointers. The c-application is compiled with /pointer_size=64 on OpenVMS 8.2 for Alpha.

On a test system the application runs without any problems. Today we tried to run the application on our production system. After a couple of minutes we get the following Error and the process dies:

%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=0170495DD7D28823, PC=FFFFFFFF8083824C, PS=0000001B
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
LIBRTL 0 000000000000E24C FFFFFFFF8083824C
DECC$SHR_EV56 0 000000000005C01C FFFFFFFF80ACC01C
Ossl OSSL sizedMalloc 42784 0000000000000108 0000000003D40108

"Ossl" is my module, the "line" points to a call to the crtl-function "malloc()". This should return a 64bit-pointer to an unused memory block.

Does the virtual address in the error message point to S2 space? Is there an error in the LIBRTL.EXE sharable (this file is the same on the test and production system)?

More info: Till the error the application has done about 500 malloc()s and nearly the same count of free()s.

Re: ACCVIO in LIBRTL while calling malloc()

Volker Halle — Mon, 03 Mar 2008 16:58:42 GMT

Hi,

consider to get a process dump. Use SET PROC/DUMP before running the image or run it with /DUMP if a detached process. Then look at the failing instrurtion, the registers and whatever value your routine passed to the malloc call.

The bad virtual address could be interpreted as a P2 space address, but it's most likely just an invalid address.

Volker.

Re: ACCVIO in LIBRTL while calling malloc()

Hein van den Heuvel — Mon, 03 Mar 2008 17:34:00 GMT

This is 99.99% sure to be a usage error.
I'm only saying that you help you focus your trouble shoorting. Do NOT waste time suspecting the sustem function.

This is likely to be a control structure corruption DETECTED (ok, not detected but stumbled into) by malloc, but caused by a bad FREE.

Something de-allocated which was (no more) allocated?
Some structure used after the memory was freed?

You may need to fully trace your mallocs/frees.

Good luck!
Hein.

Re: ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Mon, 03 Mar 2008 20:04:39 GMT

Volker,
thanks to remind me to set the process's dump-flag. I usually use it in all my projects but forgot it here.

Hein,
and what about the other 0.01% ;-) Okay, I agree not to assume a system-function error.

I am definitly sure that each malloc() has its free() and that there are no double free()s. But I cannot be absolutely sure, wether the limits of the allocated memory blocks are respected or not - the blocks are used by a third party api.

During massive tests with several 100 thousand iterations on the test system the limits were respected (validated by using my own implementation of malloc/free). The crash on the production system occurred on the 3rd iteration.

While reading your reply I suspect that the LIBRTL keeps its structures within P2. Is this also the case for the lib$*vm* routines which I suppose you prefere?

And what about thread-safety? Are malloc() and free() (and realloc()) thread-save?

Regards,
Dominik

Re: ACCVIO in LIBRTL while calling malloc()

John Gillings — Mon, 03 Mar 2008 21:36:15 GMT

Dominik,

If you want to easily trace your LIBRTL calls, use FAKE_RTL. Here's a complete transcript, building the fake image and tracing all LIB$GET_VM and LIB$FREE_VM calls in a DIRECTORY command:

$ @fake_rtl
Defining FAKE_RTL environment
$ @fake_rtl librtl
%COPY-S-COPIED, SYS$COMMON:[SYSLIB]LIBRTL.EXE;1 copied to DISK_USER0:[JG.FAKE_RTL]REAL_LIBRTL.EXE;1 (2080 blocks)
Generating symbol vector for LIBRTL on Alpha
%ANALYZE-I-ERRORS, DISK_USER0:[JG.FAKE_RTL]REAL_LIBRTL.EXE;1 0 errors
Generating MACRO code for LIBRTL
Assumed JSB routine LIB$ANALYZE_SDESC_R2
Warning - vector hole before LIB$CVT_DTB. Expecting 272, got 288
Warning - vector hole before LIB$DELETE_FILE. Expecting 336, got 352
Warning - vector hole before LIB$EXTV. Expecting 432, got 448
Warning - vector hole before LIB$GET_COMMAND. Expecting 544, got 576
Warning - vector hole before LIB$INDEX. Expecting 608, got 624
Warning - vector hole before LIB$LOCC. Expecting 656, got 672
Warning - vector hole before LIB$SCANC. Expecting 848, got 864
Assumed JSB routine LIB$SGET1_DD_R6
Warning - vector hole before LIB$TRA_ASC_EBC. Expecting 1168, got 1184
Assumed JSB routine OTS$$CVT_D_T_R8
Assumed JSB routine OTS$$CVT_F_T_R8
Assumed JSB routine OTS$$CVT_G_T_R8
Assumed JSB routine OTS$$CVT_H_T_R8
Assumed JSB routine OTS$MOVE3_R5
Assumed JSB routine OTS$MOVE5_R5
Assumed JSB routine OTS$SGET1_DD_R6
Assumed JSB routine STR$ANALYZE_SDESC_R1
Assumed JSB routine STR$COPY_DX_R8
Assumed JSB routine STR$COPY_R_R8
Assumed JSB routine STR$FREE1_DX_R4
Assumed JSB routine STR$GET1_DX_R4
Assumed JSB routine STR$LEFT_R8
Assumed JSB routine STR$LEN_EXTR_R8
Assumed JSB routine STR$POSITION_R6
Assumed JSB routine STR$POS_EXTR_R8
Assumed JSB routine STR$REPLACE_R8
Assumed JSB routine STR$RIGHT_R8
Assumed JSB routine OTS$$RET_A_CVT_TAB_R1
Warning - vector hole before LIB$EMUL. Expecting 2896, got 2912
Warning - vector hole before LIB$REMQHI. Expecting 3104, got 3120
Warning - vector hole before STR$ADD. Expecting 3392, got 3680
Assumed JSB routine STR$ELEMENT_R8
Warning - vector hole before LIB$TABLE_PARSE. Expecting 4496, got 4576
Warning - vector hole before LIB$FIND_VM_ZONE. Expecting 4608, got 4640
Warning - vector hole before LIB$LOCK_IMAGE. Expecting 6912, got 7248
Warning - vector hole before OTS$CNVOUT_S. Expecting 7328, got 7376
Start of data block 1 0000000000107230 LIB$AB_ASC_EBC
Compiling FAKE_LIBRTL
Linking FAKE_LIBRTL
$

$ define FAKE_DUMPARGS TRUE ! Enable argument dumping
$ fake_librtl ! enable fake_librtl for next image to run
$ dir *librtl*

Directory DISK_USER0:[JG.FAKE_RTL]

FAKE_LIBRTL.EXE;1 271 4-MAR-2008 08:28:41.70
FAKE_LIBRTL.MAR;1 60 4-MAR-2008 08:28:30.62
FAKE_LIBRTL.OBJ;1 426 4-MAR-2008 08:28:32.16
FAKE_LIBRTL.OPT;1 57 4-MAR-2008 08:28:30.62
LIBRTL.DATA;1 1 4-MAR-2008 08:28:30.58
LIBRTL.VEC;1 60 4-MAR-2008 08:28:30.54
REAL_LIBRTL.EXE;1 2080 4-MAR-2008 08:28:29.14

Total of 7 files, 2955 blocks.

$ search fake_argdump.log _VM ! Look for calls in argument dump
LIB$GET_VM at 08:30:30.43 from 00000000000342B4
LIB$GET_VM called with 3 args
LIB$GET_VM returning 3 args
LIB$GET_VM returned: 00000001 at 08:30:30.43
LIB$GET_VM at 08:30:30.43 from 00000000000342B4
LIB$GET_VM called with 3 args
LIB$GET_VM returning 3 args
LIB$GET_VM returned: 00000001 at 08:30:30.43
LIB$GET_VM at 08:30:30.43 from 000000007AFA0D5C
LIB$GET_VM called with 2 args
LIB$GET_VM returning 2 args
LIB$GET_VM returned: 00000001 at 08:30:30.43
LIB$GET_VM at 08:30:30.45 from FFFFFFFF80DD06B8
LIB$GET_VM called with 2 args
LIB$GET_VM returning 2 args
LIB$GET_VM returned: 00000001 at 08:30:30.45
LIB$FREE_VM at 08:30:30.46 from 0000000000034434
LIB$FREE_VM called with 3 args
LIB$FREE_VM returning 3 args
LIB$FREE_VM returned: 00000001 at 08:30:30.46
LIB$FREE_VM at 08:30:30.46 from 0000000000034434
LIB$FREE_VM called with 3 args
LIB$FREE_VM returning 3 args
LIB$FREE_VM returned: 00000001 at 08:30:30.46
LIB$FREE_VM at 08:30:30.46 from FFFFFFFF80DD6004
LIB$FREE_VM called with 2 args
LIB$FREE_VM returning 2 args
LIB$FREE_VM returned: 00000001 at 08:30:30.46

(that said, I'd be interested to see how FAKE_RTL handles 64 bit addressing... it might not work properly)

Re: ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Mon, 03 Mar 2008 21:54:48 GMT

Hi John,

thank you for the hint. I will try this tomorrow. Btw. I've never heard of fake_rtl. Is this link the latest version of it?

http://h71000.www7.hp.com/openvms/journal/v7/fake_rtl_com.txt

Dominik

Re: ACCVIO in LIBRTL while calling malloc()

John Gillings — Tue, 04 Mar 2008 01:13:33 GMT

Dominik,

Come to think of it, that version is a bit old. I've attached the latest version as a text file, just rename to FAKE_RTL.COM

Please let me know what happens with the 64 bit addresses.

Re: ACCVIO in LIBRTL while calling malloc()

Hoff — Tue, 04 Mar 2008 01:45:53 GMT

If you have more than one malloc() call and one free() in a typical application, you'll want to resolve that. Scattered calls are a classic coding issue, because it makes it far harder to find these issues.

Do centralize your memory management calls.

Once you have your memory management calls centralized into your own my_malloc() and my_free(), you can do a couple of things. The first of which is (if platform-specific code is permissible) ditch malloc() and free() in favor of direct lib$get_vm and lib$free_vm calls. You have more control here, and you can particularly establish VM zones for various purposes including for garbage-collection-style memory management. More importantly for this particular situation, you can implement what I call "fenceposts"; structures in the memory management that allow you to detect errors at deallocation (by checking the integrity of the memory), or at run-time using the RTL VM verification call.

Articles on C memory management debugging are here, including details on fenceposts:

http://64.223.189.234/node/401
http://64.223.189.234/node/260

Stephen Hoffman
HoffmanLabs LLC

Re: ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Wed, 05 Mar 2008 11:02:11 GMT

John,

I have written this small C-code:

$ type test64.c
//#include stdlib.h // suppress compile err
void main() {
char *c = malloc(1);
*c = 1; // this is line 4 in traceback
}

and compiled it with 64bit-pointer support:
$ cc test64/noopt/debug/pointer_size=64
$ link test64
$ r test64

This works fine so far. If run after a call to fake_librtl test64 will crash. If the same code is compiled without 64bit-pointer support then test64 does not crash.

In the 64bit-version of test64 the address I get from malloc() is a valid address. But it seems that the related memory page is still protected when used with fake_librtl :-(

$ fake_librtl
$ r test64
%SYSTEM-F-ACCVIO, access violation, reason mask=04, virtual address=FFFFFFFF80000010, PC=0000000000020108, PS=0000001B
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
test64 TEST64 main 4 0000000000000108 0000000000020108
test64 TEST64 __main 0 0000000000000094 0000000000020094
0 FFFFFFFF8033BF94 FFFFFFFF8033BF94

I have also tried to use fake_librtl to trace lib$ calls within my module. But because my module is itself a sharable image it is very difficult to get it running (the main image crashed while using fake_librtl). I have given it up to get this running.

Next I will check if any code tries to reuse invalid memory addresses.

Dominik

Re: ACCVIO in LIBRTL while calling malloc()

John Gillings — Wed, 05 Mar 2008 20:28:10 GMT

Dominik,

Thanks for the feedback, I suspected it might not work, as there has been no consideration for 64 bit addresses in FAKE_RTL. FAKE_RTL assumes a VAX style calling standard with homed 32 bit arguments. I'll consider adding 64 bit support when I have long term access to an I64 system to test on.

Interesting that the VA of the fault is 8000010 - that's the address of the AST dispatcher, and you're trying to WRITE to it!

The implication is your pointer "c" has been corrupted with the system address.

Re: ACCVIO in LIBRTL while calling malloc()

Jur van der Burg — Thu, 06 Mar 2008 08:44:25 GMT

>Interesting that the VA of the fault is 8000010 - that's the address of the AST dispatcher, and you're trying to WRITE to it!

John,

That's not the case. If you're compiling with 64 bit pointers you will get a 64 bit pointer to a piece of memory, in this case 00000000.80000010 which is a valid P2 space address.

Fwiw,

Jur.

Re: ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Thu, 06 Mar 2008 09:47:09 GMT

Jur,

fake_librtl returns a signed(?) 32bit pointer. It is implicit converted to 64bit to the value FFFFFFFF.80000010. This means this is not a valid P2 address. I should have looked at the address given by the traceback.
I have verified this with the debugger. Or append:

printf("%Lx\n", c);

after the call to malloc(). Btw. malloc() has called lib$vm_malloc_64 which itself allocated P2 space at 00000000.80000000:00000000.80001FFF (see SDA). At least this has worked fine.

Dominik

Re: ACCVIO in LIBRTL while calling malloc()

Volker Halle — Fri, 07 Mar 2008 18:59:12 GMT

Dominik,

FAKE_RTL uses PUSHR/POPR to save and restore R0 and R1 around various calls. This seems to cause the high-order longword of the quadword register values to be clobbered.

$PUSH64 and $POP64 might be correct replacements for this.

Volker.

Re: ACCVIO in LIBRTL while calling malloc()

Volker Halle — Sun, 09 Mar 2008 07:26:57 GMT

Dominik,

you can make FAKE_RTL.COM work for routines returning 64-bit status values in R0/R1 like this:

In FAKE_RTL.COM replace

PUSHR #^M ; save R0 & R1

with

$PUSH64 R0 ; save full quadword values
$PUSH64 R1

and replace

POPR #^M ; restore registers

with

$POP64 R1 ; restore full quadword values
$POP64 R0

After this modification, your TEST64 program does work and the malloc returns the correct address in P2 sapce.

Making FAKE_DUMPARGS print the full 64-bit register values takes a little bit more work.

Volker.

Re: ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Tue, 11 Mar 2008 09:12:18 GMT

Volker,

thanks for your patience. I got a dump file from my process. The accvio of this dumpfile is not in malloc() but in some module for which I have sourcecode. The module was compiled with /noopt and /debug enabled. This means I should have all symbols.
When invoking "ana/proc " the debugger always claims that the source listing for my module is not available. This is also the case if I force using specific sources by typing "set source/latest ".
Normally this works fine for me. But in this case the problem may be because my module is linked as a shared image and the symbols of the shared image are resolved at runtime (here: by Java's JNI).
Do you see any chance to make the source code visible in the debugger in this case?

Dominik

Re: ACCVIO in LIBRTL while calling malloc()

Volker Halle — Tue, 11 Mar 2008 13:42:49 GMT

Dominik,

did you try DBG> SET IMAGE xxx

Otherwise, you might have to compile your source code with /LIST/MACHINE and find the offending instruction stream by comparing the instructions around the failing PC in the dump and the machine code listing.

Volker.

Re: ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Tue, 11 Mar 2008 14:01:32 GMT

Volker,

I tried to use "set image xxx" but it claims that the specified image cannot be found??? I also tried "set image/all" which loaded my module into the process set. At this time I could see the correct symbols in the call frames. But I am still not able to view the source code within the debugger.
I do not need to compile with /list/machine because the traceback (at the time the dump occurs) shows me the linenumbers. But I would like to examine the used variables and the contents of specific memory-addresses. I can do this only in the debugger, but only if the debugger finds the source-listings.

Okay, this is one thing. But let's go back to the origin for this thread:

Sure malloc() is not the origin of the failure. But I wonder that a module that is run in USER-MODE can corrupt memory areas which is used by system functions. (At least I call routines in LIBRTL as system function.)
I have expected that LIBRTL functions keep their internal structures within memory pages which are not accessable by USER-MODE modules. That's why I thought of an error within malloc().

Dominik

Re: ACCVIO in LIBRTL while calling malloc()

Volker Halle — Tue, 11 Mar 2008 14:09:50 GMT

Dominik,

the LIB$ routines are NOT system services, so they run in user mode and therefore can't protect their data from other user mode code.

To make progress with your process dump, you would probably need to resort to the machine code listings - tedious, but works...

Volker.

Re: ACCVIO in LIBRTL while calling malloc()

John Gillings — Tue, 11 Mar 2008 22:13:08 GMT

Dominik,

You may want to try using the HeapAnalyzer to see who's doing what with your heap memory. It's part of the debugger. See Chapter 12 in the "HP OpenVMS Debugger Manual".

http://h71000.www7.hp.com/doc/82final/4538/4538pro_024.html#index_x_1206

Re: ACCVIO in LIBRTL while calling malloc()

Dominik Andelshauser — Tue, 25 Mar 2008 09:34:48 GMT

Thanks to all of you, who tried to help me. I have found the error!

Because my module and the 3rd party api are cross-platform I built the same code on Windows. With a good portion of luck the application then crashed near the point with the coding error. The problem was: I not only ported the code to 64bit-pointers but also tried to reduce the heap-usage by changing the sequence of some api-calls - in the wrong order...

Side-effect of the code review: On OpenVMS I now use lib$get/free_vm_64() directly. The code is about 25% faster than before, because the used api makes heavy use of memory allocation and deallocation.

Again, thanks to all of you!
Dominik