About memory fragmentation and MallocNextGen

[blackwater]

Hi,

I have been developing a multithread application on C++ on HP-UX ia64 11.23 and 11.31. Usually it works as a daemon for at least a few days. The problem is that its virtual size measured by ps keeps growing while the application process requests.

I read carefully the thread discussing the similar question http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1351858 and I would like to ask a few more questions:

1) One of recommendation is to start using MallocNextGen. My question is whether this library is available for 64 applications or only for 32 applications.

2) Does MallocNextGen require setting any variable like _M_SBA_OPTS or not. I have not fine anything about that in http://www.docs.hp.com/en/5992-4174/5992-4174.pdf.

3) Then, how to get “Detailed Report” about memory? There was a long detailed report in the thread. I use info heap in gdb in order to get information about memory allocations but it only reports heap usage and list of allocations but I can’t find the “Detailed Report”. What am I doing wrong? The reason why I want to get a detailed report is to use this information for setting appropriately _M_SBA_OPTS.

P.S. this thread has been moved from HP-UX > General to HP-UX > languages - HP Forums Moderator

[Don Morris_1]

Yes, it has 32-bit and 64-bit libraries. Just link the appropriate one.

From the documentation, _M_SBA_OPTS has no influence with MallocNextGen.
(http://docs.hp.com/en/5992-4174/ch10s09.html -- the Compatibility section).

Don't know about the Detailed Report, sorry. I'm sure someone else will (likely Dennis).

[Dennis Handly]

>I read carefully the thread discussing the similar question

Have you determined whether you have a leak or fragmentation?

>how to get "Detailed Report" about memory?

Hmm, other than doing one by one, the documentation isn't obvious.
http://www.hp.com/go/wdb
Perhaps you have to write it to a file?
info heap filename
info heap idnumber
info heap arena
info heap arena [0 |1|2|..] blocks stacks
info heap process

You should also look at "Debugging dynamic memory usage errors using HP WDB".

[blackwater]

Dennis,

> Have you determined whether you have a leak or fragmentation?
I can't answer to your question. The problem is when I ran the command info leaks in order to get information about leaks I got:
(gdb) info leaks
Scanning for memory leaks...


Program received signal SIGTRAP, Trace/breakpoint trap
si_code: 1073741952 - .

The program being debugged was signaled while in a function called from GDB.
GDB has restored the context to what it was before the call.
To change this behavior use "set unwindonsignal off"
Evaluation of the expression containing the function (__rtc_leaks_info) will be abandoned.


OK, I did: set unwindonsignal off
Then I ran info leaks again and got:

Scanning for memory leaks...


Program received signal SIGTRAP, Trace/breakpoint trap
si_code: -2147151679 - .

The program being debugged was signaled while in a function called from GDB.
GDB remains in the frame where the signal was received.
To change this behavior use "set unwindonsignal on"
Evaluation of the expression containing the function (__rtc_leaks_info) will be abandoned.

[blackwater]

Dennis, you are right.
When I run: info leaks there is no detailed report.
When I run: info leaks filename there is a detailed report.

[Dennis Handly · Accepted solution]

>I can't answer to your question.

By calling mallinfo(3) you can. What about "info heap arena" and "info heap process"?

>The problem is when I ran the command info leaks in order to get info

What wdb version are you using? Have you downloaded 6.0?

>When I run: info leaks filename there is a detailed report.

Do two of them separated by time and see if it grows.

[blackwater]

Don, thanks for your answer.
Just want to clarify. If I use MallocNextGen all my statements like these:

ClassName1 *p1 = new ClassName1;
ClassName2 *p2 = new ClassName2[5];
int *p3 = new int[1000];
std::vector arr;
arr.push_back(ClassName3());

will use MallocNextGen?

[blackwater]

Dennis,
1)
My application is 64-bit. I read for example this thread: http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=943026. As far as I understand mallinfo doesn't return correct data for 64 applications. Am I right?

2)
As for gdb version:
srv2-rx8 B.11.31 ia64 bash-3.2$ gdb -v
HP gdb 6.0 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.0 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.

And before running my application I did these:
set unwindonsignal on
set heap-check leaks on
set heap-check frame-count 16

[blackwater]

As for "info leaks". It doesn't always fail. When I run it while the application starts up it say:
(gdb) info leaks
Scanning for memory leaks...


2150464 bytes leaked in 3 blocks

No. Total bytes Blocks Address Function
0 2146304 1 0x600000000255c740 dtl::obj_allocator_base::allocate_new_block(unsigned long,unsigned long,char const*,char const*,unsigned long,bool)
1 4096 1 0xc098b000 _hp_ldap_client_connect_server()
2 64 1 0x600000000292fba0 snlui()



But after a thread processing requests has been started the "info leaks" gives me:
[Switching to thread 9 (system thread 7138095)]

Breakpoint 6, rating_server::tariffer::rate_current_record (
this=0x600000002fae9a50, id_rec=3) at rating_server.cpp:1626
1626 ++statjob.recs_total;
(gdb) info leaks
Scanning for memory leaks...


Program received signal SIGSEGV, Segmentation fault
si_code: 2 - SEGV_ACCERR - Invalid Permissions for object.
mark (first=0x600000000a1074b0, last=0x600000000a107510, leak_detection=true,
internal_gc=false, location=HEAP) at ../../../Src/gnu/gdb/infrtc.c:6825
6825 ../../../Src/gnu/gdb/infrtc.c: No such file or directory.
in ../../../Src/gnu/gdb/infrtc.c

arg sulong failed. 0, 0x9fffffffe97442a8

Setup args failed.

Pid 14862 was killed due to failure in writing to user register stack - possible stack overflow.

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.
No stack.

[Dennis Handly]

>will use MallocNextGen?

You tell us about your C++ application. If you haven't replaced operator new and operator new[] or added your own allocators, it will.

>As far as I understand mallinfo doesn't return correct data for 64 applications. Am I right?

It should return something but I recall they messed up and didn't expand to truly 64 bit values. But this may be enough to check for leaks.

>Pid 14862 was killed due to failure in writing to user register stack - possible stack overflow.

It appears you need to increase your thread stack size.

Instead of info leaks, you may want to try the info heap commands.

[blackwater]

Dennis, thanks for your response.

I have already used info heap filename and then compared the reports and found a memory leak. I think formally speaking iIt's not a memory leak, but anyway, the result was as if I had a memoty leak.

The problem was with a global list variable and a global boolean flag. They were defined and used in this way:

std::list global_list_variable;
bool global_list_variable_initialized = false;

void f()
{
if(!global_list_variable_initialized) {
global_list_variable.push_back(MyClass());
// global_list_variable_initialized
// by mistake was not set to true
}
}

As for your suggestion "It appears you need to increase your thread stack size.". What do you mean? My application doesn't cause this error when it runs. It only happens when I run info leaks under gdb. So where should I increase thread stack size and how? Do you mean setting maxssiz_64bit? Could you explain, please?

[Dennis Handly]

>The problem was with a global list variable and a global boolean flag.

Why have two data structures to track one thing? Just use:
if (global_list_variable.begin() == global_list_variable.end()) {
global_list_variable.push_back(MyClass())

(Unless f() is called after there are entries on the list.)

My application doesn't cause this error when it runs. It only happens when I run info leaks under gdb.

Right. IF you want to use gdb, you would need to increase your thread stacksize.

>So where should I increase thread stack size and how?

When you do pthread_create, you need to set the thread stacksize attribute with:
pthread_attr_setstacksize(3)

Or export PTHREAD_DEFAULT_STACK_SIZE with a larger size, in bytes.

[blackwater]

Dennis, thanks for your answer.

I made a search and found in one of libraries that I use a call to pthread_attr_setstacksize. I think I will comment it and recompile my application in order to experiment with PTHREAD_DEFAULT_STACK_SIZE. If I still have a problem with info leaks causing a coredump I will start another thread later.

[blackwater]

1) there is 64-bit MallocNextGen library

2) Seems like there is no need to set any enviroment variable in order to use it and existing enviroment variable like _M_SBA_OPTS doen't affect it.

3) The detailed memory report is generated
if I run a command "info heap filename"

4) "info heap" provides lots of information in order to find memory leaks and situations which look like a memory leak.

5) In order to run "info leak" for my application I need to increase thread stack size but I can't do it right now just by setting PTHREAD_DEFAULT_STACK_SIZE because in one of my library there is a call to pthread_attr_setstacksize(). I need to comment the call and recompile the application in order to see if "info leaks" does not result in a coredump.

[blackwater]

One more point.

The detailed report showed that the majority of memory blocks are less than 1024 bytes and there are few which are quite big.
So it is appropriate to set
_M_SBA_OPTS=1024:100:16

Seems like after assigning _M_SBA_OPTS this value I can expect that there might be no memory fragmentation.

[Jini Susan George]

Hi,

Here it looks like your main thread has exited when 'info leaks' fails with stack overflow. Could you please check if this is the case? If so, could you check if the main thread can be retained till whenever you need to issue the command to detect leaks?

- Jini.

[blackwater]

One more point. I have made a version of the program that didn't use pthread_attr_setstacksize(). Then I created a shell script that exported PTHREAD_DEFAULT_STACK_SIZE and ran the program under gdb. I tried setting PTHREAD_DEFAULT_STACK_SIZE with a very big number and with not so big number. In both cases when I used "info leaks" I got the same error (a signal and then a coredump).

At this point I decided to change the way I test the program. Instead of loading all the records from a database I changed a configuration of the program and started loading only records which were necessary for running my tests. That resulted in a significant decrease in number of memory allocations. And as a result of it executing "info leaks" didn't result in a coredump.