Error in unwind information

[Naina U]

A thread crashes consistently in the HP-UX 11.23 itanium IP64 machine

#0 0x60000000fac17570:1 in Raise_Exception::operator() ()
at /vobs/build/mwsrc/kernel32/syncobjs/exception.C:1581
#1 0x60000000fac173b0:0 in Signal_Handler::The_Handler ()
at /vobs/build/mwsrc/kernel32/syncobjs/exception.C:1566
#2
#3 0x60000000fa2d7510:0 in kill+0x30 () from /usr/lib/hpux32/libc.so.1
#4 0x60000000fa1cdd50:0 in raise+0x30 () from /usr/lib/hpux32/libc.so.1
#5 0x60000000fe29f190:0 in _SigBusSegvIotHandler (nSigNo=11,
nCode=-219458032, pSigContext=0xf2eb5800) at osdsun.cpp:886
#6
#7 0x60000000e35ff250:1 in IlcActivity::requires ()
at ../../../src/ilc/schedule.cpp:8642
warning: No unwind information found.
Skipping this library /usr/lib/hpux32/libcl.so.1.

When i do ldd -r /usr/lib/hpux32/libcl.so.1


libIO77.so.1 => /usr/lib/hpux32/libIO77.so.1
libunwind.so.1 => /usr/lib/hpux32/libunwind.so.1
libuca.so.1 => /usr/lib/hpux32/libuca.so.1
symbol not found: funlockfile (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvxf_dbl_to_quad (/usr/lib/hpux32/libIO77.so.1)
symbol not found: fileno (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __filbuf (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvff_dbl_to_quad (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _calloc (/usr/lib/hpux32/libIO77.so.1)
symbol not found: malloc (/usr/lib/hpux32/libIO77.so.1)
symbol not found: realloc (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _memcpy (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _memcmp (/usr/lib/hpux32/libIO77.so.1)
symbol not found: memcmp (/usr/lib/hpux32/libIO77.so.1)
symbol not found: ctime_r (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _memset (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _ioctl (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _tempnam (/usr/lib/hpux32/libIO77.so.1)
symbol not found: toupper (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __ldecvt_r (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __errno (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strcoll (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _getenv (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _putenv (/usr/lib/hpux32/libIO77.so.1)
symbol not found: system (/usr/lib/hpux32/libIO77.so.1)
symbol not found: getenv (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _getpid (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __thread_once (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _clearerr (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strerror (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _sprintf (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __ftruncate64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _fprintf (/usr/lib/hpux32/libIO77.so.1)
symbol not found: fprintf (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _unlink (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _fflush (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strncpy (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strncmp (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __iob (/usr/lib/hpux32/libIO77.so.1)
symbol not found: localtime_r (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __thread_mutex_unlock (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _gettimeofday (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _fputc (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _isatty (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _fread (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __ftello64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _access (/usr/lib/hpux32/libIO77.so.1)
symbol not found: flockfile (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __signal_magic_cookie (/usr/lib/hpux32/libIO77.so.1)
symbol not found: sigprocmask (/usr/lib/hpux32/libIO77.so.1)
symbol not found: ___exit (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvff_quad_to_sgl (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvfxt_quad_to_i8 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: shmdt (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _shmat (/usr/lib/hpux32/libIO77.so.1)
symbol not found: setlocale (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvff_quad_to_f80 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvff_quad_to_dbl (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __ismt (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __xpg4_extended_mask (/usr/lib/hpux32/libIO77.so.1)
symbol not found: clock (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _fclose (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _close (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __fseeko64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _qfns (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __stat64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: atoi (/usr/lib/hpux32/libIO77.so.1)
symbol not found: atol (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _read (/usr/lib/hpux32/libIO77.so.1)
symbol not found: pause (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvfxt_quad_to_i16 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _fwrite (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _write (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvfxt_quad_to_dbl (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvfxt_quad_to_sgl (/usr/lib/hpux32/libIO77.so.1)
symbol not found: getchar (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _time (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __thread_mutex_lock (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __open64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: sigaddset (/usr/lib/hpux32/libIO77.so.1)
symbol not found: free (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __thread_rec_mutex_init (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _catgets (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __ecvt_r (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __lstat64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _sigreturn (/usr/lib/hpux32/libIO77.so.1)
symbol not found: sigaction (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __fstat64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __fopen64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __tls_get_addr (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strncasecmp (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcnvff_f80_to_quad (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _shmctl (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _shmget (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _sigsetreturn (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _catopen (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _fdopen (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __libc_mutex_trylock (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strcat (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strcpy (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strcmp (/usr/lib/hpux32/libIO77.so.1)
symbol not found: mblen (/usr/lib/hpux32/libIO77.so.1)
symbol not found: abort (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _strlen (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfadd (/usr/lib/hpux32/libIO77.so.1)
symbol not found: setvbuf (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfdiv (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfcmp (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfmpy (/usr/lib/hpux32/libIO77.so.1)
symbol not found: __lseek64 (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfneg (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _U_Qfsub (/usr/lib/hpux32/libIO77.so.1)
symbol not found: _signal (/usr/lib/hpux32/libIO77.so.1)
symbol not found: signal (/usr/lib/hpux32/libIO77.so.1)
symbol not found: sigemptyset (/usr/lib/hpux32/libIO77.so.1)
symbol not found: malloc (/usr/lib/hpux32/libunwind.so.1)
symbol not found: _memcpy (/usr/lib/hpux32/libunwind.so.1)
symbol not found: _memset (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __load_info (/usr/lib/hpux32/libunwind.so.1)
symbol not found: getenv (/usr/lib/hpux32/libunwind.so.1)
symbol not found: printf (/usr/lib/hpux32/libunwind.so.1)
symbol not found: sprintf (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fprintf (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __iob (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fopen (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __thread_mutex_unlock (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fread (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fseek (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __ismt (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fclose (/usr/lib/hpux32/libunwind.so.1)
symbol not found: atoi (/usr/lib/hpux32/libunwind.so.1)
symbol not found: dlmodinfo (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __thread_mutex_lock (/usr/lib/hpux32/libunwind.so.1)
symbol not found: exit (/usr/lib/hpux32/libunwind.so.1)
symbol not found: free (/usr/lib/hpux32/libunwind.so.1)
symbol not found: dlgetname (/usr/lib/hpux32/libunwind.so.1)
symbol not found: strcmp (/usr/lib/hpux32/libunwind.so.1)
symbol not found: strcpy (/usr/lib/hpux32/libunwind.so.1)
symbol not found: abort (/usr/lib/hpux32/libunwind.so.1)
symbol not found: strlen (/usr/lib/hpux32/libunwind.so.1)


the same here

libuca.so.1 => /usr/lib/hpux32/libuca.so.1
symbol not found: malloc (/usr/lib/hpux32/libunwind.so.1)
symbol not found: _memcpy (/usr/lib/hpux32/libunwind.so.1)
symbol not found: _memset (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __load_info (/usr/lib/hpux32/libunwind.so.1)
symbol not found: getenv (/usr/lib/hpux32/libunwind.so.1)
symbol not found: printf (/usr/lib/hpux32/libunwind.so.1)
symbol not found: sprintf (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fprintf (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __iob (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fopen (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __thread_mutex_unlock (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fread (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fseek (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __ismt (/usr/lib/hpux32/libunwind.so.1)
symbol not found: fclose (/usr/lib/hpux32/libunwind.so.1)
symbol not found: atoi (/usr/lib/hpux32/libunwind.so.1)
symbol not found: dlmodinfo (/usr/lib/hpux32/libunwind.so.1)
symbol not found: __thread_mutex_lock (/usr/lib/hpux32/libunwind.so.1)
symbol not found: exit


(/usr/lib/hpux32/libunwind.so.1)
symbol not found: free (/usr/lib/hpux32/libunwind.so.1)
symbol not found: dlgetname (/usr/lib/hpux32/libunwind.so.1)
symbol not found: strcmp (/usr/lib/hpux32/libunwind.so.1)
symbol not found: strcpy (/usr/lib/hpux32/libunwind.so.1)
symbol not found: abort (/usr/lib/hpux32/libunwind.so.1)
symbol not found: strlen (/usr/lib/hpux32/libunwind.so.1)


Here again

nm -AC /usr/lib/hpux32/libunwind.so.1 |grep malloc
/usr/lib/hpux32/libunwind.so.1: U malloc


What could be possibel issue in the machine

I also found a patch for PHSS 38423 but that was for clusters which is not in the machine. Could anyone tell me how to resolve this issue

[Suprateeka R Hegde]

if you want unwind information, just use the libunwind library. you dont have to use the bigger libcl (which contains fortran IO, pascal try recover, etc) just for unwind information.

the latest unwind library patch ID is PHSS_39101 for 11.23. try that and just for unwind information, link only libunwind (-lunwind) and not libcl. the usage of linking libcl for unwind is seen on PA machines.

the "symbol not found" message is not a problem. it just says that there is a reference to that symbol and that symbol is not found in the library. all those are usually found in libc which will be linked by default by the compiler driver and are available at runtime.

[Dennis Handly]

I see no error in unwind, just a warning and two recursive signals and handlers.

#7 IlcActivity::requires is getting signal 11 and this is what you need to debug.
#3 kill, this is being called from your first signal handler.

>What could be possible issue in the machine?

You'll need to debug the first signal. It could be a thread stack overflow? But only if you have an alternate signal stack to catch it.

>Suprateeka: pascal try recover, etc)

There is no Pascal on Integrity. And if there was, TRY/RECOVER IS the job for libunwind.

[Naina U]

Thanks Suprateeka and Dennis for your insights on the issue. It is indeed a thread stack overflow issue and as Suprateeka mentioned its just a warning and not a real issue in hand. Will debug the issue and get insights from third party. Thanks a lot for your inputs it heldped a lot.

[Dennis Handly]

>Thanks a lot for your inputs it helped a lot.

If you are happy with the answers you were given, please read the following on how to assign points:
http://forums.itrc.hp.com/service/forums/helptips.do?#33

[Naina U]

Hey that was useful information too :) will keep in mind to point the answers from now on.. Thanks Dennis

[Dennis Handly]

>will keep in mind to point the answers from now on

The idea is to do it for ALL replies:
http://forums.itrc.hp.com/service/forums/pageList.do?userId=WW352727&listType=unassigned