crash in mallinfo

[Nitin kashyap]

Hello,

I am observing the following crash:

Stack trace:
===================
#0 0x6ff36588 in mallinfo+0xec8 () from /usr/lib/libc.2
#1 0x6ff33d64 in __thread_callback_np+0x624 () from /usr/lib/libc.2 #2 0x6ff398b0 in malloc+0x198 () from /usr/lib/libc.2 #3 0x6fd6a5a0 in operator new []+0x40 () from /usr/lib/libCsup_v2.2 #4 0x96630 in jfXSLTranslator::process (this=0x6e5dc850, oXMLInStream=@0x6e5dc8c0, oOutTextStream=@0x6e5dcd28) at xsltrans.cpp:181 #5 0xa67d0 in jfElementImpl::cloneNodeViaXSL (this=0x401bd6b4, oTranslator=@0x6e5dc850, poXSLResult=0x6e5dcd28) at elementimpl.cpp:743 #6 0xa6474 in jfElementImpl::cloneNodeViaXSL (this=0x401bd6b4, oInStream=@0x6e5dc360, poXSLResult=0x0) at elementimpl.cpp:725 #7 0xa63e0 in jfElementImpl::cloneNodeViaXSL (this=0x401bd6b4, uri=@0x6e5dc240, poXSLResult=0x0) at elementimpl.cpp:716 #8 0x8a7b0 in jfDomElement::cloneNodeViaXSL (this=0x6e5dc23c, #aggretxform#137=@0x6e5dc250, uri=@0x6e5dc240) at domelement.cpp:100 #9 0x62bc0 in jfElementXslTester::ClassTest (this=0x40061618) at elementxsltest.cpp:83 #10 0x6e704 in ExecuteTest (p=0x40061618) at ../utility/tester.cpp:555 #11 0x1aff18 in jfThread::ExecThread (this=0x40063850) at threads.cpp:187 #12 0x1afdf8 in jfThread::ThreadFunc (pThread=0x40063850) at threads.cpp:159 #13 0x6fb2d0b0 in __pthread_create_system+0x430 () from /usr/lib/libpthread.1 =================== the call to new is made with the argument new[3453]

the version of libC I have installed is: ========================================
# /usr/sbin/swlist -l patch | grep -i libc
# PHCO_18102 1.0 cumulative 10.20 libc compatibility support
# PHCO_18227 1.0 libc cumulative patch
# PHCO_18228 1.0 libc man page cumulative patch
# PHCO_18229 1.0 libc cumulative header file patch
# PHCO_23963 1.0 libc cumulative header file patch
# PHCO_24148 1.0 libc cumulative patch
# PHCO_24904 1.0 libc cumulative man page patch
# PHCO_25681 1.0 libcrash cumulative patch
# PHCO_25883 1.0 cumulative 10.20 libc compatibility support
# PHCO_26111 1.0 libc cumulative header file patch
# PHCO_28425 1.0 libc cumulative patch
# PHCO_28480 1.0 cumulative 10.20 libc compatibility support
# PHCO_29043 1.0 libcrash cumulative patch
# PHCO_29286 1.0 libc cumulative man page patch
# PHCO_29330 1.0 libc configuration file patch.
# PHCO_29633 1.0 libc cumulative patch
# PHSS_16849 1.0 LIBCL patch
# PHSS_28302 1.0 LIBCL patch
========================================

and the output of the program when run under tusc just before the crash is: ===========================================================================
mmap(NULL, 1069056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, NULL) .................................................................. = 0x6fcbe000 mprotect(0x6fdc2000, 4096, PROT_NONE) .......................................................................................................... = 0 _lwp_create(0x6fff14e0, LWP_DETACHED|LWP_INHERIT_SIGMASK|LWP_USER_TID, 0x40058c5c, 0x6fff19b8) ................................................. = 0 (17849) access("elementxsltest.in", F_OK) .............................................................................................................. = 0 gettimeofday(0x6fff139c, NULL) ................................................................................................................. = 0 clock_gettime(CLOCK_REALTIME, 0x6fff1420) ...................................................................................................... = 0 open("elementxsltest.in", O_RDONLY, 0666) ...................................................................................................... = 13 open("elementxsltest.out", O_WRONLY|O_CREAT|O_TRUNC, 0666) ..................................................................................... = 14 ioctl(13, TCGETA, 0x6fcbe960) .................................................................................................................. ERR#25 ENOTTY read(13, "e l e m e n t t e s t . x m l \n".., 8192) ........................................................................................... = 57 access("elementtest.xml", F_OK) ................................................................................................................ = 0 open("elementtest.xml", O_RDONLY, 0666) ........................................................................................................ = 15
time(0x6fcbf598) ............................................................................................................................... = 1090363400 lstat(".", 0x6fcc00e8) ......................................................................................................................... = 0 lstat("/home/nitin/mail-india-extra/gretzkyxtghpux/xtg/src/impl/test/domtest", 0x6fcc0ce8) ..................................................... = 0 readlink("/home/nitin/mail-india-extra/gretzkyxtghpux/xtg/src/impl/test/domtest/elementtest.xml", "", 1024) .................................... ERR#22 EINVAL ioctl(15, TCGETA, 0x6fcbf920) .................................................................................................................. ERR#25 ENOTTY
read(15, "< ? x m l v e r s i o n = " 1 ".., 8192) ........................................................................................... = 7018
read(15, 0x401354c0, 8192) ..................................................................................................................... = 0
close(15) ...................................................................................................................................... = 0 access("xcttoxfa.xsl", F_OK) ................................................................................................................... = 0 open("xcttoxfa.xsl", O_RDONLY, 0666) ........................................................................................................... = 15 lstat(".", 0x6fcbf6e8) ......................................................................................................................... = 0 lstat("/home/nitin/mail-india-extra/gretzkyxtghpux/xtg/src/impl/test/domtest", 0x6fcc02e8) ..................................................... = 0 readlink("/home/nitin/mail-india-extra/gretzkyxtghpux/xtg/src/impl/test/domtest/xcttoxfa.xsl", "", 1024) ....................................... ERR#22 EINVAL lseek(15, 0, SEEK_SET) ......................................................................................................................... = 0 ioctl(15, TCGETA, 0x6fcbef60) .................................................................................................................. ERR#25 ENOTTY
read(15, "< x s l : s t y l e s h e e t ".., 8192) ........................................................................................... = 8192
read(15, " n a m e = " t h i c k n e s s ".., 8192) ........................................................................................... = 4836 read(15, 0x40106bf0, 8192) ..................................................................................................................... = 0
time(0x6fcbea58) ............................................................................................................................... = 1090363400 gettimeofday(0x6fcbeb98, NULL) ................................................................................................................. = 0 gettimeofday(0x6fcbec18, NULL) ................................................................................................................. = 0
Received signal 11, SIGSEGV, in user mode, [SIG_DFL], partial siginfo
Siginfo: si_code: I_NONEXIST, faulting address: 0x948ce574, si_errno: 0
PC: 0xc018558b, instruction: 0x48413ff9
exit(11) [implicit] ............................................................................................................................ WIFSIGNALED(SIGSEGV)|WCOREDUMP ksleep(PTH_CONDVAR_OBJECT, 0x40065b38, 0x40065b40, 0x6fff13a4) ................................................................................. [running] munmap(0x6fcbe000, 1069056) .................................................................................................................... [running] ===========================================================================


any ideas how do i debug it further?

[Nitin kashyap]

Further after reading some similar mails i added some diagnostic code (which also uses mallinfo) just before the new statement.

Just before teh crash the output is as follows:
==============================
page size is 4096 bytes
physical memory size is 524288 pages, 2147483648 bytes

total virtual memory allocated is 137181 pages, 561893376 bytes
active virtual memory is 78407 pages, 321155072 bytes
total real memory is 282684 pages, 1157873664 bytes
active real memory is 177057 pages, 725225472 bytes
free memory is 7163 pages, 29339648 bytes

malloc arena memory is 17589640 bytes
free malloc ordinary buffer memory is 14 482 776 bytes
free malloc small block memory is 1 908 824 bytes

in-use malloc ordinary buffer memory is 17 986 336 bytes
in-use malloc small block memory is 115 176 bytes

This process is using 23732224 bytes of RAM.
This process is using 48701440 bytes of VM.
This process is using 17821696 bytes of data.
Segmentation fault (core dumped)

=================================

as i see it the in use ordinary buffer memory (17986336 bytes) is greater than the total memory in the malloc arena ()17589640 bytes).

Is this the reason?
Does this mean i have some meory leaks in my code or simply the memory requirement is higher?
what is the way to increase the memory availabel to malloc arena?
if i am right chatr +q3pemable increases the data space only. is it so?? (because i tried that and it did not work :( )

[Mike Stroyan]

I think the strange mallinfo output that you got is just a symptom of corrupted malloc data structures. (The mallinfo in the crash stack trace is really a static function that is at an address following the mallinfo function.)

The '+q3p enable' option is not needed until a program is near 2GB of data. Your program is not even using 1% of that.

You should use the wdb memory check tool to look for code that is misbehaving with malloc memory.

[Nitin kashyap]

Hi Mike,
Well the mystery deepens:

When I run it under wdb with Memory Checks enabled, the program does not crash. It does give two warnings like:
-----------
Continuing.
[Switching to thread 4 (system thread 8234)]
warning: The block being freed (0x40642760) appears to have a corrupted footer __rtc_event () at ../../../Src/gnu/gdb/infrtc.c:571
571 ../../../Src/gnu/gdb/infrtc.c: No such file or directory.
Current language: auto; currently c
#0 __rtc_event () at ../../../Src/gnu/gdb/infrtc.c:571
571 in ../../../Src/gnu/gdb/infrtc.c
Continuing.
-----------
But they are in gdb code and via a third party library that we have.

Could this be the culprit?

[Mike Stroyan]

That definitely could be the culprit. You should enable the wdb check box for "Stop when block is freed if bad writes occured before or after block bounds". Then look very carefully at how that corrupted block is allocated and used. It is very likely that some writes are going past the length that was requested.