Receiving error "SEGV_MAPERR - Address not mapped to object" consistently

[Thiagu_1]

We are building our application (C++ code) in HP-UX 11.23 IPF 64-bit platform and the compiler version we use is A.06.10.

We get the following error quite often and we are not sure of its root cause.

Program received signal SIGSEGV, Segmentation fault
si_code: 1 - SEGV_MAPERR - Address not mapped to object.
0x4000000002b439c0:1 in AutoPtr<:defectfilters>::AutoPtr<:defectfilters>(DataSet::DefectFilters*)+0x11 ()

Earlier we got the same exception in other place where we were creating 10000 array of class objects. We changed the value from 10000 to 1000 and it worked fine.

We thought there could be some deficiency in stack size and increased the size of the kernel parameter "maxssiz_64bit" to a maximum value and still we persist with the same problem.

We use certain libraries in our application like SourcePro Edition 8, Orbix 6.3, Oracle 10G, etc.

Also please find the compiler and linker options that we use to build our code and can you please tell us if there are any incompatibility with the options?

Compiler options
-----------------
aCC +DD64 -g -D_DEBUG -DUSE_AA_OPTION +W641 -Wl,+s -DPERFORMANCE -DKRWUNIX -DUSE_NAMESPACE_KTSTDKRW -DUSE_ROUGEWAVE_INTERFACE -DKRW_EXTENDED_CLASS_SET -DKRW_KLARITY_ENUM_ORTN_LOC -D_RWCONFIG__RogueWaveStdLib_NoThrLib_Static_Debug -I./include -I/opt/SourcePro64/Ed8/orasrc -I/opt/SourcePro64/Ed8 -I/opt/Orbix6.3/asp/6.3/include -D_CMA_NOWRAPPERS_ -AA -c test.cpp -o test.o

Linker Options
--------------
aCC +DD64 -g -D_DEBUG -DUSE_AA_OPTION +W641 -Wl,+s -DPERFORMANCE -DKRWUNIX -DUSE_NAMESPACE_KTSTDKRW -DUSE_ROUGEWAVE_INTERFACE -DKRW_EXTENDED_CLASS_SET -DKRW_KLARITY_ENUM_ORTN_LOC -D_RWCONFIG__RogueWaveStdLib_NoThrLib_Static_Debug -I./include -I/opt/Orbix6.3/asp/6.3/include -I/opt/SourcePro64/Ed8/include -I/opt/SourcePro64/Ed8 -D_CMA_NOWRAPPERS_ -AA -o test.exe test.o /opt/SourcePro64/Ed8/lib/rwoci_RogueWaveStdLib_NoThrLib_Static_Debug.o -L/opt/SourcePro64/Ed8/lib -loci62-_RogueWaveStdLib_NoThrLib_Static_Debug -ldbt62-_RogueWaveStdLib_NoThrLib_Static_Debug -ltls79-_RogueWaveStdLib_NoThrLib_Static_Debug -lstd41-_RogueWaveStdLib_NoThrLib_Static_Debug -L/opt/Orbix6.3/asp/6.3/lib/ipf_64 -L/opt/Orbix6.3/shlib/ipf_64 -L/opt/Orbix6.3/shlib/default/ipf_64 -lit_naming -lit_poa -lit_art -lit_ifc -lpthread -L/opt/Oracle/Oracle10.2/oracle/lib -lclntsh -lsql10 -lcommon10 -lnls10 -lcore10 -lnls10 -lclient10 -lX11

Thanks
Thiagu

[Dennis Handly]

>A.06.10

The latest patch is A.06.13, PHSS_35752, with A.06.14 coming out next week or so.

>We thought there could be some deficiency in stack size and increased the size of the kernel parameter "maxssiz_64bit" to a maximum value and still we persist with the same problem.

Setting maxssiz_64bit will have no affect on thread stacks. The default is 256 Kb.

Do you have a stack trace of your abort?
Can you do the following to get an estimate of your thread stack size?
(gdb) frame 0
(gdb) p /x $save_sp=$sp
(gdb) frame X where X is the largest frame from bt
(gdb) p $sp - $save_sp

If this doesn't make it obvious, in frame 0, do:
(gdb) disas
(gdb) info reg

>-D_CMA_NOWRAPPERS_

This is ignored for kernel threads.

>-lpthread

You need to compile with -mt if you are using HP's aC++ runtime with threads. If you are replacing it with SourcePro, you should be linking with +nostl.

[Thiagu_1]

>Do you have a stack trace of your abort?
>Can you do the following to get an estimate of your thread stack size?
>(gdb) frame 0
frame 0 is the point where it is crashing

>(gdb) p /x $save_sp=$sp
Saved the sp of frame 0

>(gdb) frame X where X is the largest frame from bt
frame 20 is the last frame of bt

>(gdb) p $sp - $save_sp
The value is 69504

>If this doesn't make it obvious, in frame 0, do:
>(gdb) disas
>(gdb) info reg
We could not figure out the issue on looking into the assembler code. We find the same memory related issue in other places as well?

For example,
MyFile.h
-------------
const int MAX_VALUE = 10000;
class MyClass
{
public:
MyClass();
virtual ~MyClass();
class InnerClass
{
public:
InnerClass() { }
~InnerClass() { }
};
private:
InnerClass arrObj[MAX_VALUE];
InnerClass *ptrObj;
};


MyFile.cpp
----------------
#include "MyFile.h"
MyClass::MyClass() // Same error is thrown at this line
{
ptrObj = arrObj;
....

}

We got the same error "SEGV_MAPERR - Address not mapped to object" for the above code.
On changing the array value from 10000 to 1000, the problem got resolved. So we suppose the problem could be with the internal memory stack settings.

Could you please tell us the root cause for the same?

[Dennis Handly · Accepted solution]

>The value is 69504

This should fit since the default is 256Kb/2.

>(gdb) disas
>(gdb) info reg
>We could not figure out the issue on looking into the assembler code. We find the same memory related issue in other places as well?

You are suppose to attach the output so I can look at it. The "make it obvious" was the thread stack size.

>const int MAX_VALUE = 10000;
>InnerClass arrObj[MAX_VALUE];

You have at least 10000 bytes for this, unless you left out the data members.

>On changing the array value from 10000 to 1000, the problem got resolved. So we suppose the problem could be with the internal memory stack settings. Could you please tell us the root cause for the same?

Yes, it seems you have a thread stack overflow. You need to increase your thread stack with either:
pthread_attr_setstacksize
pthread_default_stacksize_np

Or with a recent libpthread patch, PHCO_34718:
export PTHREAD_DEFAULT_STACK_SIZE= ...

[Thiagu_1]

After installing PHCO_34944, we still persist with the same problem.

We increased the stack size to 2MB (setenv PTHREAD_DEFAULT_STACK_SIZE 2048), still find the same problem and still increased the size to 4MB for which I got the following error.

Pid 11782 in trap loop, signal 11
Segmentation fault

Thanks,
Thiagu

[Dennis Handly]

>After installing PHCO_34944, we still persist with the same problem.
>We increased the stack size to 2MB (setenv PTHREAD_DEFAULT_STACK_SIZE 2048),

This ENV VAR only affects threads created with the default stack size. If they use an explicit size, then they will still abort. You need to see which thread is getting the abort.

>still increased the size to 4MB for which I got the following error.
Pid 11782 in trap loop, signal 11
Segmentation fault

Hmm, I don't think I've seen that recently and I'm not sure why increasing it would cause that issue. I would need to see a stack trace. The few hits in google show extensive memory corruption.

[Thiagu_1]

>This ENV VAR only affects threads created with the default stack size.
>If they use an explicit size, then they will still abort. You need to see which thread is getting the abort.

We don't explicitly create thread in our codebase. We use Orbix 6.3 as one of our third-party tool which uses threads internally. So we stopped all the orbix daemons and set the ENV VAR to 2MB(2048) and started the daemons again but was unable to resolve this issue.

Do you have any other suggestions?

[Dennis Handly]

>Do you have any other suggestions?

I need another stack trace. With an estimate of the thread size.

[Dennis Handly]

>We increased the stack size to 2MB (setenv PTHREAD_DEFAULT_STACK_SIZE 2048),

I'm not sure where you got the idea that this was 1Kb units? It doesn't mention it. It seemed reasonable but I wanted to double check.

The units are in bytes, so you need to increase the value by a thousand fold, or maybe just to 400000.

I still don't understand why the 256 Kb default won't let you have 69504?? Perhaps they are setting the thread stack size?

You can use tusc to check, look for these lines where it allocates the stack and adds a guard page:
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|0x40000, -1,
0) = 0x6788c000
mprotect(0x678ac000, 4096, PROT_NONE) .. = 0

[Dennis Handly]

From your latest tusc.out it seems your signal 11 is right on the guard page.

It seems that Orbix 6.3 is setting the thread stack to 128 Kb for most threads and taking the default for others. (The thread stack of 128 Kb would be divided by 2 on IPF.)