topic Re: library difference between compile env. and production env. in Operating System - HP-UX

library difference between compile env. and production env.

tash S — Tue, 03 Apr 2007 03:41:18 GMT

Hi All,

If we use running on HP-UX 11i of compile env, we sometimes got the core when apache start.

#0 0xc7ee43b8 in ()
warning: Attempting to unwind past bad PC 0xc7ee43b8
#1 0xc80cfbe4 in __rw::__rb_tree

But when we use the library on HP-UX 11i of the other machine, which is different from compile env., we did not get the error.

If you know the reason, please let me know.

The both of OS is the same,
----------------
bash-2.04# uname -r
B.11.11
----------------

Re: library difference between compile env. and production env.

Piergiacomo Perini — Tue, 03 Apr 2007 04:14:50 GMT

Hi Tash S,
i don't know the reason but, if you know the main library of Apache, try to do a "cksum" and make comparison between two file .

hth
regards
pg

Re: library difference between compile env. and production env.

tash S — Tue, 03 Apr 2007 05:06:31 GMT

The two binary is the exact same, because I copied from one machine (the problem is happening at the machiine) to the other.

Re: library difference between compile env. and production env.

Piergiacomo Perini — Tue, 03 Apr 2007 07:03:45 GMT

Hi again tash,
not same as your issue but maybe this

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1107787

can help you a little bit (try debugging with gdb if you know this tool).

kind regards
pg

Re: library difference between compile env. and production env.

A. Clay Stephenson — Tue, 03 Apr 2007 09:20:51 GMT

The differences are almost certainly in the shared libraries and there may be 10 or so shared libraries associated with your executable. Do an "ldd" on the executable to list the shared libraries and then compare them (using cksum) on both machines. You can also use swlist on both machine and look for differecnces in patches. When you find differences DO NOT SIMPLY COPY the files unless they are your own (or a 3rd party's) libraries because you will get the installed software database in an inconsistant state. When you identify the differences then install patches to bring the two machines in sync. Swlist will be very helpful in spotting the differences.

One other note: Reading between the lines and noting your ${PS1}, it appears that bash is root's shell. Make absolutely certain that root's shell (as defined in /etc/passwd_ is /sbin/sh because root must have a statically linked shell. It's okay for root to explicitly invoke bash from /sbin/sh and I trust that this is what you are acrtually doing. You won't know anything is wrong until you try to reboot.

Re: library difference between compile env. and production env.

tash S — Tue, 03 Apr 2007 18:57:53 GMT

Clay,

>Reading between the lines and noting
>your ${PS1},

Can you tell me the meaning of ${PS1}?

regards,

Re: library difference between compile env. and production env.

A. Clay Stephenson — Tue, 03 Apr 2007 19:06:04 GMT

The shell's prompt string 1.

Try this and answer your own question:

echo "${PS1}"
echo "${PS2}"

Since '#' is, by convention, part of root's ${PS1}, it appears that bash is root's shell. If so, you are an accident waiting to happen.

Re: library difference between compile env. and production env.

tash S — Tue, 03 Apr 2007 19:17:41 GMT

Clay,

The both is the same result.

bash-2.04# echo "${PS2}"
>
bash-2.04# echo "${PS1}"
\s-\v\$

Re: library difference between compile env. and production env.

A. Clay Stephenson — Tue, 03 Apr 2007 19:26:21 GMT

They are NOT the same result. Do a man bash and look for PS1, it will explain what you are seeing but the important point is that you are running bash (or any other dynamically linked shell) as root's shell. You should fully understand the consequences of this under HP-UX.

Re: library difference between compile env. and production env.

tash S — Tue, 03 Apr 2007 20:44:56 GMT

Hi All,

I notice that the problem machine in the below.

====================
bash-2.04# ls -lt /usr/lib/libCsup_v2.2
-r-xr-xr-x 1 bin bin 221184 May 23 2006 /usr/lib/libCsup_v2.2*
====================

But no problem machine is the below.
====================
bash-2.04# ls -lt /usr/lib/libCsup_v2.2
-r-xr-xr-x 1 bin bin 217088 Sep 25 2002 /usr/lib/libCsup_v2.2*
====================

The question is that how can I upgrade libCsup_v2.2 library?

Re: library difference between compile env. and production env.

tash S — Tue, 03 Apr 2007 22:09:46 GMT

In order to be the same library (/usr/lib/libCsup_v2.2) I will try to install the patch. usually do we need to restart box?

And what is the good command for reboot on HP-UX 11i, because the box is at remote.

Re: library difference between compile env. and production env.

A. Clay Stephenson — Wed, 04 Apr 2007 00:09:48 GMT

The patch itself will reboot the machine if necessary. The preferred command to reboot a box is "shutdown -r -y 0". Man shutdown for details.

Re: library difference between compile env. and production env.

Dennis Handly — Wed, 04 Apr 2007 02:02:02 GMT

This seems a valid shlib address. But you could have unloaded that shlib though?

pg's link is pretty useless, that deals with missing libs.

>I notice that the problem machine in the below.

You are only imagining this. ;-)
This lib doesn't contain __rb_tree.

221184 May 23 2006 /usr/lib/libCsup_v2.2*
>But no problem machine is the below.
217088 Sep 25 2002 /usr/lib/libCsup_v2.2*

>The question is that how can I upgrade libCsup_v2.2 library?

You're not looking closely. It says that the newer is the bad one. You don't want to upgrade?

>In order to be the same library (/usr/lib/libCsup_v2.2) I will try to install the patch. usually do we need to restart box?

No, you don't need to reboot to install the latest aC++ runtime, PHSS_33944.

It helps if you have a longer stack trace so we can determine your problem.
__rb_tree is only used for map and set containers and is typically in user code or shlibs.

Re: library difference between compile env. and production env.

tash S — Wed, 04 Apr 2007 19:20:13 GMT

Hi Dennis,

> pg's link is pretty useless,
> that deals with missing libs.

What is the meaning of pg's ?

> It says that the newer is the bad one.
> You don't want to upgrade?

Our customer might use new one, so I would like to know the root of causes.

> It helps if you have a longer stack trace > so we can determine your problem.
> __rb_tree is only used for map and set
> containers and is typically in user code > or shlibs.

Here is the trace of gdb, and please note that we omits some function due to confidential issue.

------------------
#1 0xc80cfbe4 in __rw::__rb_tree<..>,std::allocator<:pair>onst,char *>>>::_C_get_link (this=0x79f26f1c)
at /opt/aCC/include_std/rw/tree:409
#2 0xc7766ac0 in std::__sinit_aaa_cpp+0xe0 ()
from /opt/hpws/apache/a1.sl
#3 0xc123605c in __shlInit+0x88 () from /usr/lib/libCsup_v2.2
#4 0xc7718b80 in _shlInit+0x20 ()
from /opt/hpws/apache/a1.sl
#5 0xc1235988 in __shlinit+0xac () from /usr/lib/libCsup_v2.2
#6 0xc0032458 in _shl_load_invoke_initializers+0xb0 () from /usr/lib/dld.sl
#7 0xc0032df0 in _shl_load+0x5cc () from /usr/lib/dld.sl
#8 0xc0005e44 in __shl_load+0xf0 () from /usr/lib/libdld.2
#9 0xc3af4674 in apr_dso_load (res_handle=0x7a001574,
path=0x400ac600 "/opt/hpws/apache/a1.sl",
pool=0x4003d990) at b1o.c:83
#10 0x40bf8 in load_module (cmd=0x7a001278, dummy=0x7a0013b4,
modname=0x400ac5b0 "test_module",
filename=0x400ac5c8 "/opt/hpws/apache/a1.sl")
at mod_so.c:239
#11 0x4660c in invoke_cmd (cmd=0x40003e00, parms=0x7a001278,
---Type to continue, or q to quit---
mconfig=0x7a0013b4, args=0x4009ab26 "") at config.c:701
#12 0x483a0 in execute_now (cmd_line=0x400ac538 "LoadModule",
args=0x4009aae3 "a1 /opt/hpws/apache/a1.sl", parms=0x7a001278, p=0x4003d990, ptemp=0x40084a90,
sub_tree=0x7a0013b4, parent=0x0) at config.c:1334
#13 0x47418 in ap_build_config_sub (p=0x4003d990, temp_pool=0x40084a90,
l=0x4009aad8 "LoadModule rsawebagent_module /opt/hpws/apache/a1.sl", parms=0x7a001278, current=0x7a001330,
curr_parent=0x7a001334, conftree=0x7a000f78) at config.c:913
#14 0x47c04 in ap_build_config (parms=0x7a001278, p=0x4003d990,
temp_pool=0x40084a90, conftree=0x7a000f78) at config.c:1126
#15 0x48b2c in process_resource_config_nofnmatch (s=0x4003f7b8,
fname=0x400a8438 "/opt/hpws/apache/a1.conf",
conftree=0x7a000f78, p=0x4003d990, ptemp=0x40084a90, depth=0)
at config.c:1511
#16 0x48d34 in ap_process_resource_config (s=0x4003f7b8,
fname=0x400a8438 "/opt/hpws/apache/b2.conf",
conftree=0x7a000f78, p=0x4003d990, ptemp=0x40084a90) at config.c:1547
#17 0x666b8 in include_config (cmd=0x7a000cb8, dummy=0x7a000df4,
name=0x400a8388 "/opt/hpws/apache/a1.conf")
at core.c:2298
#18 0x46508 in invoke_cmd (cmd=0x4000a540, parms=0x7a000cb8,
mconfig=0x7a000df4, args=0x40072a95 "") at config.c:691
#19 0x483a0 in execute_now (cmd_line=0x400a8328 "include",
args=0x40072a68 "/opt/hpws/apache/a1.conf",
parms=0x7a000cb8, p=0x4003d990, ptemp=0x40084a90, sub_tree=0x7a000df4,
parent=0x0) at config.c:1334
#20 0x47418 in ap_build_config_sub (p=0x4003d990, temp_pool=0x40084a90,
l=0x40072a60 "include /opt/hpws/apache/a1.conf",
parms=0x7a000cb8, current=0x7a000d70, curr_parent=0x7a000d74,
conftree=0x4000cea0) at config.c:913
#21 0x47c04 in ap_build_config (parms=0x7a000cb8, p=0x4003d990,
temp_pool=0x40084a90, conftree=0x4000cea0) at config.c:1126
#22 0x48b2c in process_resource_config_nofnmatch (s=0x4003f7b8,
fname=0x4008a248 "/opt/hpws/apache/conf/httpd.conf", conftree=0x4000cea0,
p=0x4003d990, ptemp=0x40084a90, depth=0) at config.c:1511
#23 0x48d34 in ap_process_resource_config (s=0x4003f7b8,
fname=0x4008a248 "/opt/hpws/apache/conf/httpd.conf", conftree=0x4000cea0,
p=0x4003d990, ptemp=0x40084a90) at config.c:1547
#24 0x49ddc in ap_read_config (process=0x4003ba08, ptemp=0x40084a90,
filename=0x4000649c "conf/httpd.conf", conftree=0x4000cea0)
at config.c:1890
#25 0x4e0a0 in main (argc=3, argv=0x7a00076c) at main.c:588

----------------------

I would like to compare the patch level in different machine.

swlist -l patch
is the only command ?
This is too much information and it would be difficult to compare.

Re: library difference between compile env. and production env.

tash S — Wed, 04 Apr 2007 19:34:09 GMT

In above problem, is it for link problem or source code problem?

Our source code use Roque Wave C++ library.

Re: library difference between compile env. and production env.

Dennis Handly — Wed, 04 Apr 2007 21:32:17 GMT

>What is the meaning of pg's?

pg as in Piergiacomo's sig.

>Our customer might use new one, so I would like to know the root of causes.

Right. We need to know why the latest fails.

>Here is the trace of gdb, and please note that we omits some function due to confidential issue.

What's missing except for #0? (You meant you changed some names/strings/libs?)

Basically you are loading a shared lib and the static initializers are running for the file aaa.cpp.
You are creating a map.

I'm not sure I can find out why #0 is missing and you got a signal, unless I see the machine code and registers. Please do:
(gdb) frame 1
(gdb) info reg
(gdb) disas

Re: library difference between compile env. and production env.

tash S — Wed, 04 Apr 2007 23:14:56 GMT

Hi Dennis,

I missed just 'cut & past' in previous thread.

(gdb) bt
#0 0xc88243b8 in ()
warning: Attempting to unwind past bad PC 0xc88243b8

The below is the result.
>(gdb) frame 1
>(gdb) info reg
>(gdb) disas

====================

(gdb) frame 1
#1 0xc89cfbe4 in __rw::__rb_tree<...>::_C_get_link (this=0x79f26f1c)
at /opt/aCC/include_std/rw/tree:409
409 : (_C_next_avail == _C_last ? (_C_add_new_buffer(), _C_next_
avail++)
(gdb) info reg
flags: 6000041
r1: 79f26f1c rp: c89cfbe7
r3: c82a69e0 r4: 79f292f8
r5: 79f26f1c r6: 79f292e8
r7: 79f29338 r8: 1
r9: 0 r10: 0
r11: 79fff1f4 r12: 79fff1f0
r13: 0 r14: ffffffff
r15: 8000084 r16: 40013c30
r17: 40014950 r18: 0
r19: 79f1cb34 r20: 0
r21: c88243b8 r22: 79f26f24
r23: 79feb210 r24: 79f291a0
r25: 79fdedd8 r26: 79f26f1c
dp: 40010648 ret0: 79f29ef1
ret1: 79fb8dc0 sp: 7a002070
r31: 79f26f28 sar: 8
pcoqh: c88243b8 pcsqh: 8f24c00
pcoqt: c88243bc pcsqt: 8f24c00
eiem: ffffffff iir: 43ffff80
isr: 1824002d ior: 6d405058
ipsw: 4fe1f goto: 2
sr4: c89c800 sr0: 8f24c00
sr1: 8f24c00 sr2: 0
sr3: 0 sr5: 8c98000
sr6: 8f24c00 sr7: 8f24c00
cr0: 2e00 cr8: 2e40000
cr9: 10000 ccr: 1
cr12: 2e40 cr13: 2e50000
cr24: 1000 cr25: 3
cr26: 2e50 mpsfu_high: 79f4d070
941c38 mpsfu_ovfl: 0
pad: 912050 fpsr: 8200000
fpe1: 0 fpe2: 0
fpe3: 0 fpe4: 0
fpe5: 0 fpe6: 0
fpe7: 0
(gdb)
(gdb)
(gdb) disas
Dump of assembler code for function _C_get_link__Q2_4__rw9__rb_treeXTP10RSACStri
ngTQ2_3std4pairXTCP10RSACStringTPc_TQ2_4__rw11__select1stXTQ2_3std4pairXTCP10RSA
CStringTPc_TP10RSACString_T16rw_deref_compareXTQ2_3std4lessXT10RSACString_T10RSA
CString_TQ2_3std9allocatorXTQ2_3std4pairXTCP10RSACStringTPc___Fv:
;;; File: /opt/aCC/include_std/rw/tree
;;; 405 _C_link_type _C_get_link() {
0xc89cfb68 <__rw::__rb_tree<...>::_C_get_link(void)>: stw %rp,-0x14(%sp)

..
..

Re: library difference between compile env. and production env.

Dennis Handly — Wed, 04 Apr 2007 23:24:45 GMT

>In above problem, is it for link problem or source code problem?

It could be a link order problem, but it wouldn't work before.

>The below is the result.

I don't see every byte of the disassembly. (".." isn't helpful.) You need to attach a file with the output.

And while you're at it, do:
(gdb) info shared

Re: library difference between compile env. and production env.

tash S — Wed, 04 Apr 2007 23:32:26 GMT

Hi,

(gdb) info shared
Shared Object Libraries
flags tstart tend dstart dend dlt
/usr/lib/dld.sl
0x02000200 0xc0010000 0xc0042fbc 0x79ffb000 0x79fff558 0x79ffb0a0
/opt/hpws/apache/lib/libaprutil-0.sl.9
0x03000301 0xc1490000 0xc14b6000 0x79f52000 0x79f54000 0x79f520f0
/opt/hpws/apache/lib/libexpat.sl.1
0x03000301 0xc3ba0000 0xc3bd4000 0x79f56000 0x79f59000 0x79f560f0
/opt/hpws/apache/lib/libapr-0.sl.9
0x03000301 0xc3ac0000 0xc3af8000 0x79f59000 0x79f5c000 0x79f59120
/usr/lib/libm.2
0x03000301 0xc00b0000 0xc00dc000 0x79f5e000 0x79f5f000 0x79f5e28c
/usr/lib/librwtool_v2.2
0x03000301 0xc9600000 0xc97c7000 0x79f79000 0x79fb9000 0x79f7c374
/usr/lib/libnsl.1
0x03000301 0xc0280000 0xc030e000 0x79f64000 0x79f71000 0x79f64430
/usr/lib/libxti.2
0x03000301 0xc0070000 0xc0088000 0x79f5f000 0x79f62000 0x79f5f0e0
/usr/lib/libstd_v2.2
0x03000301 0xc9400000 0xc95f1000 0x79fbd000 0x79fdc000 0x79fbeadc
/usr/lib/libCsup_v2.2
0x03000301 0xc1220000 0xc123c000 0x79fdc000 0x79fdf000 0x79fdc400
/usr/lib/libc.2
0x03000301 0xc0100000 0xc0247000 0x79fe4000 0x79ff7000 0x79fe4794
/usr/lib/libdld.2
0x03000301 0xc0005000 0xc0008000 0x79fdf000 0x79fe0000 0x79fdf00c
/opt/hpws/apache/rsawebagent/Plugins/libaceauth.sl
0x03ff0348 0xc8080000 0xc80e3000 0x79efe000 0x79f06000 0x79efe4f4
/usr/lib/libc.1
0x03000348 0xc0600000 0xc0768000 0x79ecf000 0x79efc000 0x79ecf85c
/usr/lib/libdld.1
0x03000348 0xc00dc000 0xc00dd000 0x79ec8000 0x79ec9000 0x79ec800c
/opt/hpws/apache/modules/mod_access.so
0x03ff0340 0xc0536000 0xc0538000 0x79f4c000 0x79f4d000 0x79f4c00c
/opt/hpws/apache/modules/mod_include.so
0x03ff0340 0xc0fb0000 0xc0fbe000 0x79f4a000 0x79f4c000 0x79f4a028
/opt/hpws/apache/modules/mod_log_config.so
0x03ff0340 0xc0768000 0xc076f000 0x79f49000 0x79f4a000 0x79f49018
/opt/hpws/apache/modules/mod_env.so
0x03ff0340 0xc04af000 0xc04b0000 0x79f48000 0x79f49000 0x79f4800c
/opt/hpws/apache/modules/mod_setenvif.so
0x03ff0340 0xc07dd000 0xc07df000 0x79f47000 0x79f48000 0x79f47010
/opt/hpws/apache/modules/mod_proxy.so
0x03ff0340 0xc1214000 0xc121f000 0x79f45000 0x79f47000 0x79f4502c
/opt/hpws/apache/modules/mod_proxy_connect.so
0x03ff0340 0xc07ea000 0xc07ec000 0x79f44000 0x79f45000 0x79f4400c
/opt/hpws/apache/modules/mod_proxy_ftp.so
0x03ff0340 0xc07f8000 0xc0800000 0x79f42000 0x79f44000 0x79f42010
/opt/hpws/apache/modules/mod_proxy_http.so
0x03ff0340 0xc0e44000 0xc0e4a000 0x79f41000 0x79f42000 0x79f41010
/opt/hpws/apache/modules/mod_mime.so
0x03ff0340 0xc0dec000 0xc0df0000 0x79f40000 0x79f41000 0x79f40010
/opt/hpws/apache/modules/mod_status.so
0x03ff0340 0xc12ac000 0xc12b1000 0x79f3e000 0x79f40000 0x79f3e024
/opt/hpws/apache/modules/mod_autoindex.so
0x03ff0340 0xc12b4000 0xc12bc000 0x79f3c000 0x79f3e000 0x79f3c010
/opt/hpws/apache/modules/mod_asis.so
0x03ff0340 0xc04ef000 0xc04f0000 0x79f3b000 0x79f3c000 0x79f3b008
/opt/hpws/apache/modules/mod_suexec.so
0x03ff0340 0xc053f000 0xc0540000 0x79f3a000 0x79f3b000 0x79f3a00c
/opt/hpws/apache/modules/mod_cgid.so
0x03ff0340 0xc1304000 0xc130c000 0x79f39000 0x79f3a000 0x79f39024
/opt/hpws/apache/modules/mod_cgi.so
0x03ff0340 0xc1384000 0xc138a000 0x79f38000 0x79f39000 0x79f38018
/opt/hpws/apache/modules/mod_negotiation.so
0x03ff0340 0xc13c8000 0xc13d1000 0x79f35000 0x79f36000 0x79f35024
/opt/hpws/apache/modules/mod_dir.so
0x03ff0340 0xc0ced000 0xc0cef000 0x79f34000 0x79f35000 0x79f3400c
/opt/hpws/apache/modules/mod_actions.so
0x03ff0340 0xc076f000 0xc0770000 0x79f33000 0x79f34000 0x79f3300c
/opt/hpws/apache/modules/mod_userdir.so
0x03ff0340 0xc0db9000 0xc0dbb000 0x79f32000 0x79f33000 0x79f3200c
/opt/hpws/apache/modules/mod_alias.so
0x03ff0340 0xc0de9000 0xc0dec000 0x79f31000 0x79f32000 0x79f3100c
/opt/hpws/apache/modules/mod_rewrite.so
0x03ff0340 0xc13f0000 0xc1401000 0x79f2e000 0x79f31000 0x79f2e01c
/opt/hpws/apache/rsawebagent/mod_rsawa_apache.sl
0x03ff0340 0xc76c0000 0xc778c000 0x79f1c000 0x79f2c000 0x79f1cb34
(gdb)
======

(gdb) frame 1
#1 0xc80cfbe4 in __rw::__rb_tree*>,__rw::__select1st<:pair>,RSACString *>,rw_deref_
compare<:less>,RSACString>,std::allocator<:pair>onst,char *>>>::_C_get_link (this=0x79f26f1c)
at /opt/aCC/include_std/rw/tree:409
409 : (_C_next_avail == _C_last ? (_C_add_new_buffer(), _C_next_
avail++)
Current language: auto; currently c++

Re: library difference between compile env. and production env.

tash S — Wed, 04 Apr 2007 23:40:21 GMT

Hi Dennis,

f you think the link order problem, can you refer to documents on docs.hp.com

Thanks,