Weird memory problem

[Dennis Handly]

>The problem is that I'll need to rebuild everything and unfortunately on the test machine that can take considerable time.

If you are lucky, all you need to do is rebuild the lib. If not, you'll have to rebuild every module that uses those allocate/deallocate member functions. I suppose you could use nm -pxNA to scan the objects to see how pervasive the functions are. (If inlined, that won't work.)

[Peter Hug]

We have done all of what you suggested. But now have another problem: as soon as we start the app, it starts writing massive amounts of messages like:

1164805853:243022000:9987394:13105:1:2:33:4:0x680c80d8:0x0:0x1:0x20:0x0:0x0:

to a file called tr.x.00001 (where x=ProcID)

Eventually the app crashes and the last messages in the above file read:

Pthread internal error: message: write() failed, file: /ux/core/libs/threadslibs/src/common/pthreads/trace.c, line: 774
Error msg: No space left on device

From these messages I take it that pthread is printing these messages. Does this sound familiar to you? How could I suppress these?

[Dennis Handly]

>as soon as we start the app, it starts writing massive amounts of messages

I can't see how those messages would come from U_STACK_TRACE and the fprintfs I suggested.

Pthread internal error: message: write() failed, file: .../pthreads/trace.c, line: 774
>From these messages I take it that pthread is printing these messages. Does this sound familiar to you? How could I suppress these?

Yes, libpthread*.
This is completely unexpected. Are you using libpthread_tr?

[Peter Hug]

We are using boost::threads.

[Dennis Handly]

>We are using boost::threads.

Are you linking with -lpthread_tr?

[Lalit Seth]

Hi Dennis,

I reply on behalf of Peter.

As Pete mentioned we are using Boost::threads.

We compile and link files using -pthread flag and link using the Boost thread libraries.

Here is the complete linking command -
g++ -g -I/opt/apal3/lib/boost_1_30_0 -I/opt/apal3/lib/XML-02_01_07/include -I/opt/odbc/include -I/usr/local/include -I/opt/apal3/lib/OTL_4_1_18/include -I/opt/oracle/rtcis/product/9.2.0/rdbms/demo -I/opt/oracle/rtcis/product/9.2.0/rdbms/public -Wunused-variable -Wno-deprecated -pthread -DTRACE_CORRUPTION -I../D3 -I../SystemParameters -I../P3T3Lib -I../P3Alg -I../T3Alg -o Debug/apal3_devel3 Debug/APAL3.o Debug/APAL3Daemon.o Debug/AP3Socket.o Debug/APAL3Thread.o Debug/forcelink.o Debug/SessionThread.o Debug/TaskManagerThread.o Debug/Thread.o Debug/types.o Debug/XMLDoc.o Debug/APAL3HouseKeeper.o Debug/convenience_ext.o -L../T3Alg/Debug -lT3Alg -L../P3Alg/Debug -lP3Alg -L../D3/Debug -lD3 -L../P3T3Lib/Debug -lP3T3Lib -L../SystemParameters/Debug -lSystemParameters -L/opt/apal3/lib/boost_1_30_0/libs/thread/build/bin/libboost_thread.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi -lboost_thread -lboost_threadd -L/opt/apal3/lib/boost_1_30_0/libs/date_time/build/bin/libboost_date_time.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi -lboost_date_time -L/opt/apal3/lib/boost_1_30_0/libs/filesystem/build/bin/libboost_filesystem.a/gcc/debug/runtime-link-dynamic/threading-multi -lboost_filesystem -L/opt/apal3/lib/XML-02_01_07/lib -lcslxmld -L/usr/local/lib -lodbc++-mt -L/opt/odbc/lib -L/opt/apal3/lib -L/opt/oracle/rtcis/product/9.2.0/lib32 -lclntsh -lCsup -Wl,+b: -Wl,+s


When is do ldd on binary I see -

/usr/lib/libc.2 => /usr/lib/libc.2
/usr/lib/libdld.2 => /usr/lib/libdld.2
/usr/lib/libc.2 => /usr/lib/libc.2
/opt/langtools/lib/libpthread_tr.1 => /opt/langtools/lib/libpthread_tr.1
/usr/local/lib/libgcc_s.sl => /usr/local/lib/libgcc_s.sl
/usr/lib/libc.2 => /usr/lib/libc.2
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/local/lib/libstdc++.sl.5 => /usr/local/lib/libstdc++.sl.5
/usr/lib/libc.2 => /usr/lib/libc.2
/opt/apal3/GCC/gcc-bin-3.3.2/gcc/libgcc_s.sl => /usr/local/lib/libgcc_s.sl
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/lib/libCsup.2 => /usr/lib/libCsup.2
/opt/oracle/rtcis/product/9.2.0/lib32/libclntsh.sl.9.0 => /opt/oracle/rtcis/product/9.2.0/lib32/libclntsh.sl.9.0
/usr/lib/libcl.2 => /usr/lib/libcl.2
/usr/lib/libisamstub.1 => /usr/lib/libisamstub.1
/usr/lib/libdld.2 => /usr/lib/libdld.2
/usr/lib/libc.2 => /usr/lib/libc.2
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/lib/libdld.2 => /usr/lib/libdld.2
/usr/lib/libnss_dns.1 => /usr/lib/libnss_dns.1
/usr/lib/libdld.2 => /usr/lib/libdld.2
/usr/lib/libpthread.1 => /opt/langtools/lib/libpthread.1
/usr/lib/librt.2 => /usr/lib/librt.2
/opt/oracle/rtcis/product/9.2.0/lib32/libwtc9.sl => /opt/oracle/rtcis/product/9.2.0/lib32/libwtc9.sl
/usr/local/lib/libodbc++-mt.sl.4 => /usr/local/lib/libodbc++-mt.sl.4
/usr/lib/libc.2 => /usr/lib/libc.2
/opt/odbc/lib/libodbc.sl => /opt/odbc/lib/libodbc.sl
../../ivicu/exe/libivicu19.sl => ../../ivicu/exe/libivicu19.sl
/usr/lib/librt.2 => /usr/lib/librt.2
/usr/lib/libpthread.1 => /opt/langtools/lib/libpthread.1
../exe/libodbcinst.sl => ../exe/libodbcinst.sl
/usr/lib/librt.2 => /usr/lib/librt.2
/usr/lib/libpthread.1 => /opt/langtools/lib/libpthread.1
/usr/lib/librt.2 => /usr/lib/librt.2
/usr/lib/libpthread.1 => /opt/langtools/lib/libpthread.1
/usr/lib/libpthread.1 => /opt/langtools/lib/libpthread.1
/usr/lib/libCsup.2 => /usr/lib/libCsup.2
/opt/apal3/lib/boost_1_30_0/libs/date_time/build/bin/libboost_date_time.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_date_time.sl => /opt/apal3/lib/boost_1_30_0/libs/date_time/build/bin/libboost_date_time.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_date_time.sl
/usr/local/lib/libgcc_s.sl => /usr/local/lib/libgcc_s.sl
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/local/lib/libstdc++.sl.5 => /usr/local/lib/libstdc++.sl.5
/usr/lib/librt.2 => /usr/lib/librt.2
/opt/apal3/lib/boost_1_30_0/libs/thread/build/bin/libboost_thread.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_threadd.sl => /opt/apal3/lib/boost_1_30_0/libs/thread/build/bin/libboost_thread.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_threadd.sl
/usr/local/lib/libgcc_s.sl => /usr/local/lib/libgcc_s.sl
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/local/lib/libstdc++.sl.5 => /usr/local/lib/libstdc++.sl.5
/usr/lib/librt.2 => /usr/lib/librt.2
/opt/apal3/lib/boost_1_30_0/libs/thread/build/bin/libboost_thread.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_thread.sl => /opt/apal3/lib/boost_1_30_0/libs/thread/build/bin/libboost_thread.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_thread.sl
/usr/local/lib/libgcc_s.sl => /usr/local/lib/libgcc_s.sl
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/local/lib/libstdc++.sl.5 => /usr/local/lib/libstdc++.sl.5
/usr/lib/librt.2 => /usr/lib/librt.2
libjvm.sl => libjvm.sl
/usr/lib/libCsup.2 => /usr/lib/libCsup.2
/usr/lib/libcl.2 => /usr/lib/libcl.2
/usr/lib/librt.2 => /usr/lib/librt.2
/usr/lib/libm.2 => /usr/lib/libm.2
/usr/lib/libdld.2 => /usr/lib/libdld.2


Here u can see we have both libpthread_tr and libpthread reffered.

Is this correct? Please suggest.

Thanks
Lalit Seth

[Dennis Handly]

>We compile and link files using -pthread flag and link using the Boost thread libraries.
>Here you can see we have both libpthread_tr and libpthread referred.

You need to remove libpthread_tr.1. This is what is causing the output bloat. It must be a dependent lib of some other shlib? You use chatr on each shlib until you find it. You might be able to find it with "odump -sllibload" on your executable.

Also, you are illegally mixing g++ and aC++. You have to remove -lCsup from your link line.

[Lalit Seth]

Hi Dennis,


Thanks for this :)

I have removed reference of libCsup. Here is the output -

1 ^ /opt/apal3/lib/boost_1_30_0/libs/thread/build/bin/libboost_thread.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_thread.sl
2 ^ ^ /usr/lib/librt.2
3 ^ ^ /usr/local/lib/libstdc++.sl.5
4 ^ ^ ^ /usr/lib/libm.2
5 ^ ^ ^ /usr/local/lib/libgcc_s.sl
6 ^ ^ ^ ^ /usr/lib/libc.2
7 ^ ^ ^ ^ ^ /usr/lib/libdld.2
8 ^ ^ ^ /usr/lib/libc.2
9 ^ ^ ^ ^ /usr/lib/libdld.2
10 ^ ^ /usr/lib/libm.2
11 ^ ^ /usr/local/lib/libgcc_s.sl
12 ^ ^ ^ /usr/lib/libc.2
13 ^ ^ ^ ^ /usr/lib/libdld.2
14 ^ /opt/apal3/lib/boost_1_30_0/libs/thread/build/bin/libboost_thread.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_threadd.sl
15 ^ ^ /usr/lib/librt.2
16 ^ ^ /usr/local/lib/libstdc++.sl.5
17 ^ ^ ^ /usr/lib/libm.2
18 ^ ^ ^ /usr/local/lib/libgcc_s.sl
19 ^ ^ ^ ^ /usr/lib/libc.2
20 ^ ^ ^ ^ ^ /usr/lib/libdld.2
21 ^ ^ ^ /usr/lib/libc.2
22 ^ ^ ^ ^ /usr/lib/libdld.2
23 ^ ^ /usr/lib/libm.2
24 ^ ^ /usr/local/lib/libgcc_s.sl
25 ^ ^ ^ /usr/lib/libc.2
26 ^ ^ ^ ^ /usr/lib/libdld.2
27 ^ /opt/apal3/lib/boost_1_30_0/libs/date_time/build/bin/libboost_date_time.so/gcc/debug/runtime-link-dynamic/shared-linkable-true/threading-multi/libboost_date_time.sl
28 ^ ^ /usr/lib/librt.2
29 ^ ^ /usr/local/lib/libstdc++.sl.5
30 ^ ^ ^ /usr/lib/libm.2
31 ^ ^ ^ /usr/local/lib/libgcc_s.sl
32 ^ ^ ^ ^ /usr/lib/libc.2
33 ^ ^ ^ ^ ^ /usr/lib/libdld.2
34 ^ ^ ^ /usr/lib/libc.2
35 ^ ^ ^ ^ /usr/lib/libdld.2
36 ^ ^ /usr/lib/libm.2
37 ^ ^ /usr/local/lib/libgcc_s.sl
38 ^ ^ ^ /usr/lib/libc.2
39 ^ ^ ^ ^ /usr/lib/libdld.2
40 ^ /usr/local/lib/libodbc++-mt.sl.4
41 ^ ^ /opt/langtools/lib/libpthread.1
42 ^ ^ /opt/odbc/lib/libodbc.sl
43 ^ ^ ^ /opt/langtools/lib/libpthread.1
44 ^ ^ ^ /usr/lib/librt.2
45 ^ ^ ^ /opt/odbc/lib/libodbcinst.sl
46 ^ ^ ^ ^ /opt/langtools/lib/libpthread.1
47 ^ ^ ^ ^ /usr/lib/librt.2
48 ^ ^ ^ /opt/odbc/lib/libivicu19.sl
49 ^ ^ ^ ^ /opt/langtools/lib/libpthread.1
50 ^ ^ ^ ^ /usr/lib/librt.2
51 ^ ^ /usr/lib/libc.2
52 ^ ^ ^ /usr/lib/libdld.2
53 ^ /opt/oracle/rtcis/product/9.2.0/lib32/libclntsh.sl.9.0
54 ^ ^ /opt/oracle/rtcis/product/9.2.0/lib32/libwtc9.sl
55 ^ ^ /usr/lib/librt.2
56 ^ ^ /opt/langtools/lib/libpthread.1
57 ^ ^ /usr/lib/libnss_dns.1
58 ^ ^ ^ /usr/lib/libdld.2
59 ^ ^ /usr/lib/libdld.2
60 ^ ^ /usr/lib/libm.2
61 ^ ^ /usr/lib/libc.2
62 ^ ^ ^ /usr/lib/libdld.2
63 ^ ^ /usr/lib/libcl.2
64 ^ ^ ^ /usr/lib/libdld.2
65 ^ ^ ^ /usr/lib/libisamstub.1
66 ^ /usr/local/lib/libstdc++.sl.5
67 ^ ^ /usr/lib/libm.2
68 ^ ^ /usr/local/lib/libgcc_s.sl
69 ^ ^ ^ /usr/lib/libc.2
70 ^ ^ ^ ^ /usr/lib/libdld.2
71 ^ ^ /usr/lib/libc.2
72 ^ ^ ^ /usr/lib/libdld.2
73 ^ /usr/lib/libm.2
74 ^ /usr/local/lib/libgcc_s.sl
75 ^ ^ /usr/lib/libc.2
76 ^ ^ ^ /usr/lib/libdld.2
77 ^ /usr/lib/libpthread_tr.1
78 ^ /usr/lib/libc.2
79 ^ ^ /usr/lib/libdld.2

Also libpthread is a soft link to libpthread_tr. So we are actually using libpthread_tr and there is nothing like libpthread.

lrwxr-xr-x 1 bin bin 24 Nov 16 11:36 /opt/langtools/lib/libpthread.1 -> /usr/lib/libpthread_tr.1

How can we overcome the tr.*.* logs?

Thanks
Lalit Seth

[Dennis Handly]

>I have removed reference of libCsup.

It looks like that was the only place.

>Also libpthread is a soft link to libpthread_tr.
/opt/langtools/lib/libpthread.1 -> /usr/lib/libpthread_tr.1

>How can we overcome the tr.*.* logs?

Who put that link there? Remove it, it is not part of HP-UX. You could use the following to make sure:
swlist -l file | fgrep opt/langtools/lib/libpthread.1

[Lalit Seth]

Hi Dennis,

I removed the bad link for libpthread and now it linking to right lib ie /usr/lib/linpthread.

I think now we are in position to test as per urs suggestions for stl_alloc.h.

Will post the results soon.

Thanks a lot Dennis.

Lalit Seth