Tuxedo Error: masked synchronous signal 10.

[ShivS]

Hello Experts,

I am not able to start tuxedo on HP unix machine. Following are some details:

uname -a
HP-UX opqa1 B.11.23 U ia64 0409362437 unlimited-user license


$ tmadmin
[HP ARIES64]: ***** Log time: Mon Sep 13 17:29:48 EDT 2010 *****
[HP ARIES64]:
[HP ARIES64]: Internal Error (code : 021) [PID : 16920]
[HP ARIES64]: Got masked synchronous signal 10.
[HP ARIES64]: Aborting process!!
[HP ARIES64]:
[HP ARIES64]: ARIES Version :
[HP ARIES64]: 11.23 (PHSS_39897) Ver: 2.0.00027 [Oct 28, 2009 15:12:11]
[HP ARIES64]: Invoked command :
[HP ARIES64]: /opt/bea/tuxedo8.1/bin/tmadmin


Can somebody please help me in finding out the cause of this error?

Thanks
ShivS

[Dennis Handly]

Unfortunately you may need to change your application.

You must never block signal 11 and signal 10 (SIGBUS).

The abort occurs because the kernel notices that there is only one thread and if you block a signal, then no forward progress will be made, if you get that signal, while blocked.

Did you get a core file that you can analyze to see where you are getting the signal 10?

[ShivS]

Hello Dennis,

No I am not getting any core file.

Also please note that I am simply running some command files from Tuxedo like tmadmin or tmboot -B . I am not even trying to start my application tuxedo services.

Can you please let me know how to find out where the signal 10 or 11 is getting blocked as you mentioned?

Also, I forgot to mention in my earlier post that the tuxedo software and my tuxedo application were running fine for almost one and a half year. It is only since last couple of days or so that I am seeing this issue.

Thanks in advance.
ShivS

[Dennis Handly]

You might try installing the recommended Aries patch: PHSS_41098

>Can you please let me know how to find out where the signal 10 or 11 is getting blocked as you mentioned?

If you don't get a core file, you would have to use a debugger.

>It is only since last couple of days or so that I am seeing this issue.

Anything change? I assume you have been running this on Integrity for longer than that?

[ShivS]

Hello Dennis,

There were no changes to the HP machine before we started seeing this Tuxedo error condition.

As far as I remember, Tuxedo software and my tuxedo application have been running fine on this machine for atleast one and half year.

How do I find out if the patch PHSS_41098 you recommended is already present on this HP Unix machine?

Thanks
ShivS

[James R. Ferguson]

Hi ShivS:

> How do I find out if the patch PHSS_41098 you recommended is already present on this HP Unix machine?

A way to easily find out whether or not you have the patch in question or a superseding one with:

# show_patches -a -s -l product|grep PHSS_41098

Regards!

...JRF...

[ShivS]

Hi James,

Thanks for the show_patches command line. I used it on my machine and nothing showed up in the result. So I will have to install this patch PHSS_41098.

Can you please tell me where can I find this patch? Also how do I install (or apply) this patch on this machine?

Thanks in advance.
ShivS.

[Jim Walls]

You can get it from the ITRC site.

http://www11.itrc.hp.com/service/patch/mainPage.do

However; if you do not have a current HP support agreement, you had better be quick off the mark because the service becomes restricted from September 18!

[James R. Ferguson]

Hi ShivS:

Since you appear to be unfamiliar with HP-UX patches and the HP-UX patch mechanism, I would urge you to look at:

http://www.docs.hp.com/en/5992-6582/5992-6582.pdf

Pages 18-22 discuss how to install a single patch. As noted, patches are obtained from the HP Patch Database:

http://www.itrc.hp.com/service/patch/mainPage.do

Regards!

...JRF...

[ShivS]

Hi James,

The Unix admin department at my workplace with be performing the patch application. I simply told them which patch to apply.

Thanks for your help.

ShivS.

[Dennis Handly]

>How do I find out if the patch PHSS_41098 you recommended is already present on this HP-UX machine?

Since your Aries abort message says PHSS_39897, it is unlikely you have PHSS_41098.

[ShivS]

Hello Dennis,

My admin folks have installed patch PHSS_41098 on this machine.


$ swlist | grep PHSS_41098
PHSS_41098 1.0 Aries cumulative patch


But when I run the tuxedo admin command tmadmin, I am again getting similar error:

$ tmadmin
[HP ARIES64]: ***** Log time: Thu Sep 16 10:35:27 EDT 2010 *****
[HP ARIES64]:
[HP ARIES64]: Internal Error (code : 021) [PID : 15437]
[HP ARIES64]: Got masked synchronous signal 10.
[HP ARIES64]: Aborting process!!
[HP ARIES64]:
[HP ARIES64]: ARIES Version :
[HP ARIES64]: 11.23 (PHSS_41098) Ver: 2.0.00029 [May 15, 2010 2:57:0]
[HP ARIES64]: Invoked command :
[HP ARIES64]: /opt/bea/tuxedo8.1/bin/tmadmin

Any ideas on what could be going wrong now?

Thanks in advance.

ShivS

[Dennis Handly]

>when I run the tuxedo admin command tmadmin, I am again getting similar error:

Either your application is broken or if there is an Aries bug, it's still there.

>Internal Error (code : 021) [PID : 15437]
>Got masked synchronous signal 10.
>Aborting process!!
>Any ideas on what could be going wrong now?

You could try tusc and see there the signals are blocked.
And use /usr/ccs/bin/gdb64 to try to debug it and get a stack trace where the signal is occurring.

If you could get a core file that would also be helpful.

You may need to contact the Response Center because recently a customer had a similar error and couldn't get a core file.

Ok, I can duplicate the missing corefile under Aries. For an IPF executable, I get a core and it points to the exact instruction.

Invoking your application in gdb and continuing, points right to the error.

SHELL=~/ksh.pa PA_DEBUG=1 /usr/ccs/bin/gdb64 \
/opt/bea/tuxedo8.1/bin/tmadmin

Where ~/ksh.pa is a copy of ksh from a PA system.

[ShivS]

Hello Dennis,

Here's the gdb output:

opqa1:tuxdev>gdb /opt/bea/tuxedo8.1/bin/tmboot
Detected 64-bit executable.
Invoking /usr/ccs/bin/gdb64.
HP gdb 6.0 for PA-RISC 2.0 (wide), HP-UX 11i
and target hppa2.0w-hp-hpux11.00.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.0 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.
..(no debugging symbols found)...
(gdb) set args -B OPQA1
(gdb) run
Starting program: /opt/bea/tuxedo8.1/bin/tmboot -B OPQA1
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...
Program received signal SIGBUS, Bus error
si_code: 0 - BUS_UNKNOWN - Unknown Error.
0xc000000002a27118 in _dlopen+0x390 () from /usr/lib/pa20_64/dld.sl
(gdb) where
#0 0xc000000002a27118 in _dlopen+0x390 () from /usr/lib/pa20_64/dld.sl
#1 0x9fffffffdfa42400 in __dlopen+0xb0 () from /opt/oracle/product/9.2.0/lib32/libdl.1
#2 0x9fffffffdfa443fc in dlopen+0x34 () from /opt/oracle/product/9.2.0/lib32/libdl.1
#3 0x9fffffffdfa1f02c in real_load+0x2c4 () from /usr/lib/pa20_64/libc.2
#4 0x9fffffffdfa1ed40 in find_locale+0x570 () from /usr/lib/pa20_64/libc.2
#5 0x9fffffffdfa1da38 in setlocale+0x680 () from /usr/lib/pa20_64/libc.2
#6 0x40000000000178ac in main+0xfc ()
(gdb)


The stack trace refers to lib32 oracle libraries. Is that the cause of the problem?

Please advice.

Thanks
ShivS

[Dennis Handly · Accepted solution]

#0 0xc000000002a27118 in _dlopen+0x390 pa20_64/dld.sl
#1 0x9fffffffdfa42400 in __dlopen+0xb0 /opt/oracle/product/9.2.0/lib32/libdl.1
#2 0x9fffffffdfa443fc in dlopen+0x34 libdl.1
#3 0x9fffffffdfa1f02c in real_load+0x2c4 libc.2
#4 0x9fffffffdfa1ed40 in find_locale+0x570 libc.2
#5 0x9fffffffdfa1da38 in setlocale+0x680 libc.2
#6 0x40000000000178ac in main+0xfc

That explains the blocked signals, dld does that.

>The stack trace refers to lib32 oracle libraries. Is that the cause of the problem?

Very likely. It is illegal to use any other version of libdl other than the one in /usr/lib/.

Please remove or rename the one in /opt/oracle/product/9.2.0/lib32/.

(Also, for a 64 bit process, why is there a 64 bit libdl in a directory called lib32? ;-)

Also, have you changed your locale (LANG) lately?

[ShivS]

Hello Dennis,

After removing the path /opt/oracle/product/9.2.0/lib32/ from SHLIB_PATH, now the tuxedo software is working fine on this machine.

Thanks a ton for all you help.

ShivS.

[ShivS]

I found the solution to my problem from one of the user's response.