topic Re: cor.pid.leaks generated but empty using GDB. Errors reported. in Operating System - HP-UX

cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Wed, 15 Sep 2010 23:10:02 GMT

Hi,
The system used is PA-RISC HP-11.11
We are trying to generate the files as cor.pid.leaks and cor.pid.heap.

- These are the details of the case:

cat /fusion/opt4.1.7/OT/bin/rtcconfig
check_leaks=on
check_heap=on
frame_count=60
output_dir=/fusion/home

- Environmental variables set:
export DSD_DEBUG=1
export BATCH_RTC=on
export RTC_NO_ABORT=1
export LD_PRELOAD=/opt/langtools/lib/librtc.sl
export GDBRTC_CONFIG=/fusion/opt4.1.7/OPENET/bin/rtcconfig
export GDB_SERVER=/opt/langtools/bin/gdb

- The leaking process is started and firstly this message is received:
Warning: Entry/Exit code is optimized. Cannot restore context (UNWIND 22)

- When the process is killed, using 'kill pid' to generate the files required, this error is reported:
/fusion/home/.rtcinit_29801:12: Error in sourced command file:
To run Batch RTC,
Debug enable the executable using
"chatr +dbg enable " or
"/opt/langtools/bin/pxdb -s on ".
whether we do or not "chatr +dbg enable cor" (being cor the name of the process)
A file as cor.pid.leaks is generated, but empty.

- Also:
cat /fusion/home/.rtcinit_29801
set height 0
set {int} &shlib_info_invalid=0
set {int} &__gdb_synchronizer=1
set heap-check frame-count 4
set heap-check min-leak-size 0
set heap-check min-heap-size 0
tb __sync_up_with_gdb
commands
silent
end
c
info leaks /fusion/home/cor.29801.leaks
info heap /fusion/home/cor.29801.heap
q

- When the process reaches a max of memory permitted, it's automatically killed (not manually this time) and a file as cor.pid.mem is generated.

Please feel free to ask for further information as I pressume that my description might be incomplete.

Thanks in advance,
David

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

Dennis Handly — Thu, 16 Sep 2010 05:19:59 GMT

>When the process is killed, using 'kill pid' to generate the files required, this error is reported:

Does the WDB documentation indicate that you can kill the process, AND still get any output?

It may expect to run to completion.

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Thu, 16 Sep 2010 08:05:50 GMT

>Does the WDB documentation indicate that you can kill the process, AND still get any output?
It may expect to run to completion.

Thanks, will increase the limit so that it won't be automatically killed.
The errors obtained might be related to that problem? Are they irrelevant and the output will be generated anyway?

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Thu, 16 Sep 2010 08:10:38 GMT

>Does the WDB documentation indicate that you can kill the process, AND still get any output?
It may expect to run to completion.

What indicates that the file has been finally generated? Do we need to be checking until it stops increasing its size?

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

Dennis Handly — Thu, 16 Sep 2010 08:31:46 GMT

>The errors obtained might be related to that problem? Are they irrelevant and the output will be generated anyway?

What errors and what problems?

Because it was killed, there were no leaks or memory corruption reported.

>What indicates that the file has been finally generated? Do we need to be checking until it stops increasing its size?

When the process exits normally, otherwise no output is produced until the end.

>- When the process reaches a max of memory permitted, it's automatically killed and a file as cor.pid.mem is generated.

Anything in this file?

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Thu, 16 Sep 2010 08:51:31 GMT

>What errors and what problems?
- Warning: Entry/Exit code is optimized. Cannot restore context (UNWIND 22)

- Warning: Load module /fusion/opt4.1.7/OT/bin/cor has been stripped
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
warning: Load module /fusion/opt4.1.7/OT/lib/libcomponent.sl has been stripped
(no debugging symbols found)...
warning: Load module /fusion/opt4.1.7/OT/lib/libdfc.sl has been stripped
...
/fusion/home/.rtcinit_29801:12: Error in sourced command file:
To run Batch RTC,
Debug enable the executable using
"chatr +dbg enable " or
"/opt/langtools/bin/pxdb -s on ".
whether we do or not "chatr +dbg enable cor" (being cor the name of the process)

>When the process exits normally, otherwise no output is produced until the end.
The process should be always running. We haven't checked how long it would take to exit itself because the system runs out of memory (up to now we were putting the 1GB limit).

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Thu, 16 Sep 2010 08:52:55 GMT

> Anything in this file?

-----------------------------------------
Process ran out of memory when attempting to allocate 268 bytes
Out of memory while reading in symbol table of /orabin/product/9.2.0/lib32/libclntsh.sl.9.0
( 0) 0x783dd818 redirect_stack_trace + 0x64 [/opt/langtools/lib/librtc.sl]
( 1) 0x783dd910 print_stack_trace_to_log_file + 0xc0 [/opt/langtools/lib/librtc.sl]
( 2) 0x783de160 __rtc_event + 0x70 [/opt/langtools/lib/librtc.sl]
( 3) 0x783e10b4 malloc + 0x3c8 [/opt/langtools/lib/librtc.sl]
( 4) 0x7aaca074 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
( 5) 0x7a595610 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
( 6) 0x7a57cc7c [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
( 7) 0x7a5f6028 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
( 8) 0x7a76a5b0 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
( 9) 0x7a628380 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
(10) 0x7a3692b0 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
(11) 0x7a3d9a10 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
(12) 0x7a395840 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
(13) 0x7a39725c [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
(14) 0x7a3462e8 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]
(15) 0x7ad16f80 [/fusion/opt4.1.7/OT/lib/libdbapi.sl]
(16) 0x7ad1f3cc [/fusion/opt4.1.7/OT/lib/libdbapi.sl]
(17) 0x7826dff4 [/fusion/opt4.1.7/OT/share/dsd/modules/db.sl]
(18) 0x78725054 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(19) 0x787266b0 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(20) 0x78729c80 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(21) 0x7872fe1c [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(22) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(23) 0x7872e9cc [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(24) 0x787268b4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(25) 0x787266b0 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(26) 0x78729c80 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(27) 0x78726d00 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(28) 0x78736810 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(29) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(30) 0x7872e9cc [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(31) 0x787268b4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(32) 0x787266b0 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(33) 0x78729c80 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(34) 0x78726d00 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(35) 0x7872fef4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(36) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(37) 0x7872fef4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(38) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(39) 0x7872e9cc [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(40) 0x787268b4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(41) 0x787266b0 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(42) 0x78729c80 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(43) 0x78726d00 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(44) 0x7872e9cc [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(45) 0x787268b4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(46) 0x787266b0 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(47) 0x78729c80 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(48) 0x78726d00 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(49) 0x78732440 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(50) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(51) 0x7872e9cc [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(52) 0x787268b4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(53) 0x787266b0 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(54) 0x78729c80 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(55) 0x78726d00 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(56) 0x7872fef4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(57) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(58) 0x7872fef4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(59) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(60) 0x78736810 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(61) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(62) 0x7872e9cc [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(63) 0x787268b4 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(64) 0x787266b0 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(65) 0x78729c80 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(66) 0x78726d00 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(67) 0x78736810 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(68) 0x78726d30 [/fusion/opt4.1.7/OT/lib/libdsd.sl]
(69) 0x780cf8d4 [/fusion/opt4.1.7/OT/share/dsd/modules/qs/qs.sl]
(70) 0x780ca9f4 [/fusion/opt4.1.7/OT/share/dsd/modules/qs/qs.sl]
(71) 0x78b69e24 [/fusion/opt4.1.7/OT/lib/libon.sl]
(72) 0x79fd02e4 __pthread_body + 0x44 [/usr/lib/libpthread.1]
(73) 0x79fda574 __pthread_start + 0x14 [/usr/lib/libpthread.1]

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Thu, 16 Sep 2010 12:05:12 GMT

>Does the WDB documentation indicate that you can kill the process, AND still get any output?
- Just found through documentation:
"Batch mode memory-debugging stops the application at the end of the program when exit()
is called or when all the statically linked libraries (including librtc.[sl|so]) are unloaded.
After the application is stopped, it invokes the debugger to print the leak or heap data."

- We were using this value:
maxdsizis (a kernel parameter wich have a default value of 1G)
From http://docs.hp.com/en/B3921-90010/maxdsiz.5.html

- If the program doesn't exit by itself, how can we unloaded all the statically linked libraries?

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Thu, 16 Sep 2010 16:52:13 GMT

Misconfiguration

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

Dennis Handly — Fri, 17 Sep 2010 03:25:01 GMT

>Process ran out of memory when attempting to allocate 268 bytes
>Out of memory while reading in symbol table of
( 4) 0x7aaca074 [/orabin/product/9.2.0/lib32/libclntsh.sl.9.0]

Ideally you would want your test run to not run out of memory and still give you leaks. I.e. try to have your application have a smaller input set.

>Just found through documentation:
"Batch mode memory-debugging stops the application at the end of the program when exit() is called or when all the statically linked libraries (including librtc.[sl|so]) are unloaded.

>- We were using this value:
maxdsiz (which have a default value of 1G)

>- If the program doesn't exit by itself, how can we unloaded all the statically linked libraries?

You have to have it exit, in practical terms, it would be impossible to unload the shlibs yourself.

One way to do that is to run gdb interactively.
Or add a signal handler for SIGUSR1 or SIGUSR2 and have that call exit.

>Thread closed by author: Misconfiguration

It would be helpful to others to mention what was misconfigured.
If it was specific to your application, you don't need to go into details.

Re: cor.pid.leaks generated but empty using GDB. Errors reported.

David R A — Thu, 23 Sep 2010 09:13:38 GMT

My apologies as the customer won't give me details on what they have done, for privacy reasons, but they in the end they decided to upgrade to the last GDB version.