SIGBUS error

[Jaideep Krishnan]

Hi,

What is the "exact" reason for a SIGBUS error if an application that works on one 64-bit HP-UX system, fails on another 64-bit HP-UX system?

* Are there any memory settings that can be upgarded?
* Are there any OS patches that could be different on the two? How to know that?

Please suggest.

Thanks,
jaideep

[A. Clay Stephenson]

There is no way to give you an "exact" answer. That is really a state-of-the-art dumb question especially when you don't even bother to identify the application much less the system call that generated the error.

POSSIBLE reasons include 1) different kernel tunable settings (maxssiz, maxdsiz, et al) 2) different patches 3) different shared libraries

Start by comparing kmtune outputs to look for kernel differences and then compare swlist outputs for differences in patches and libraries.

The real key to finding your problem is to do a stack trace on the core file with a debugger. That will zero in on "exactly" what caused the program crash.

[Jaideep Krishnan]

SIGNAL "10" is the problem (SIGBUS error).

I wanted to included some observations we made in retrospect of the 2 systems:

* system on which application FAILS
* system on which application WORKS


====================================
SYSTEM ON WHICH APPLICATION "FAILS"
====================================

$ ulimit -Sa shows :

core file size (blocks, -c) 2097151
data seg size (kbytes, -d) 1048576
file size (blocks, -f) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 2048
pipe size (512 bytes, -p) 16
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 76
virtual memory (kbytes, -v) unlimited


$ ulimit -Ha shows :

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 262144
file size (blocks, -f) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 16
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 76
virtual memory (kbytes, -v) unlimited


# /usr/sbin/swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 0 4194304 0% 0 - 1 /dev/vg00/lvol2
reserve - 180936 -180936
memory 1568112 1160704 407408 74%



====================================
SYSTEM ON WHICH APPLICATION "WORKS"
====================================

$ ulimit -Sa shows:

core file size (blocks, -c) 2097151
data seg size (kbytes, -d) 262144
file size (blocks, -f) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 60
pipe size (512 bytes, -p) 16
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 76
virtual memory (kbytes, -v) unlimited


$ ulimit -Ha shows the below:

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) 1048576
file size (blocks, -f) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 2048
pipe size (512 bytes, -p) 16
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 76
virtual memory (kbytes, -v) unlimited


$ /usr/sbin/swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4194304 125316 4068988 3% 0 - 1 /dev/vg00/lvol2
reserve - 680380 -680380
memory 1565848 1195652 370196 76%


===========================================
ALSO THE BELOW PARAMENTERS ARE MISSING IN /stand/system FILE WHERE THE APPLCIATION FAILS:
===========================================
maxusers 512
nproc 2048
max_thread_proc 3000
nkthread 6000
nfile 4097
maxfiles 2048
maxfiles_lim 2048
ncallout 6000
maxdsiz 2063835136

===========================================

Can someone please gather and please let us know on these observations?

Thanks,
jaideep

[A. Clay Stephenson]

You have still missed the bus. SIGBUS can be generated because of a error detected in a number of system calls. The SIGBUS is but a symptom of the problem; it is not the cause.
Rather than ulimit outputs and system file listings, you should have posted the data I first indicated which would include all of what you posted and more.

The first significant thing I notice is the difference in maxdsiz.

[Jaideep Krishnan]

BTW what exactly does "data seg size" and "open files" mean on the HP system?

Going by the difference between the 2 HP systems:

====================================
SYSTEM ON WHICH APPLICATION "WORKS"
====================================
data seg size (kbytes, -d) 1048576
open files (-n) 2048

====================================
SYSTEM ON WHICH APPLICATION "FAILS"
====================================
data seg size (kbytes, -d) 262144
open files (-n) 1024 & 60

ALSO, FOLLOWING ENTRIES ARE MISSING ON THE SYSTEM ON WHICH APPLICATION "FAILS":

maxusers
nproc
max_thread_proc
nkthread
nfile
maxfiles
maxfiles_lim
ncallout
maxdsiz

[Jaideep Krishnan]

And HOW TO INCREASE the "data seg size" and "open files" ? Also, should they be increased....? Also, should the below parameters be hardcoded in /stand/system:

maxusers 512
nproc 2048
max_thread_proc 3000
nkthread 6000
nfile 4097
maxfiles 2048
maxfiles_lim 2048
ncallout 6000
maxdsiz 2063835136

Please let us know.

Appreciate your help.

[A. Clay Stephenson]

Maxdsiz and open files are kernel tunables. Maxdsiz is a limit on the amouint of memory that a process is able to allocate for data. Maxfiles limits the number of files a process is allowed to have open. You have to go into SAM and adjust these values and then generate a new kernel. Before you do that, you must learn what these values (and others do). It's time to study some kernel/UNIX fundamentals.

[Jaideep Krishnan]

Hi C.Stephenson,

Here is the information you needed:

********************************************
kmtune on MACHINE ON WHICH SYSTEM "FAILS"
********************************************

Parameter Current Dyn Planned Module Ver
===========================================================================
NSTRBLKSCHED - - 2
NSTREVENT 50 - 50
NSTRPUSH 16 - 16
NSTRSCHED 0 - 0
STRCTLSZ 1024 - 1024
STRMSGSZ 65535 - 65535
acctresume 4 - 4
acctsuspend 2 - 2
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
allocate_fs_swapmap 0 - 0
alwaysdump 1 - 1
bcvmap_size_factor 2 - 2
bootspinlocks - - 256
bufcache_hash_locks 128 - 128
bufpages 0 - (NBUF*2)
chanq_hash_locks 256 - 256
core_addshmem_read 0 Y 0
core_addshmem_write 0 Y 0
create_fastlinks 0 - 0
dbc_max_pct 50 - 50
dbc_min_pct 5 - 5
default_disk_ir 0 - 0
desfree - - 0
disksort_seconds 0 - 0
dmp_rootdev_is_vol 0 - 0
dmp_swapdev_is_vol 0 - 0
dnlc_hash_locks 512 - 4*128
dontdump 0 - 0
dskless_node - - 0
dst 1 - 1
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eisa_io_estimate - - 0x300
enable_idds 0 - 0
eqmemsize 15 - 15
executable_stack 1 - 1
fcp_large_config 0 - 0
file_pad - - 10
fs_async 0 - 0
ftable_hash_locks 64 - 64
hdlpreg_hash_locks 128 - 128
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
hfs_ra_per_disk 64 - 64
hfs_revra_per_disk 64 - 64
hp_hfs_mtra_enabled 1 - 1
hpux_aes_override - - 0
initmodmax 50 - 50
io_ports_hash_locks 64 - 64
iomemsize - - 40000
ksi_alloc_max 2208 - (NPROC*8)
ksi_send_max 32 - 32
lotsfree - - 0
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 64 - 64
maxdsiz 0x10000000 - 0x10000000
maxdsiz_64bit 0x40000000 - 0x0000000040000000
maxfiles 60 - 60
maxfiles_lim 1024 Y 1024
maxqueuetime - - 0
maxssiz 0x800000 - 0x00800000
maxssiz_64bit 0x800000 - 0x00800000
maxswapchunks 2048 - 2048
maxtsiz 0x4000000 Y 0x04000000
maxtsiz_64bit 0x40000000 Y 0x0000000040000000
maxuprc 75 Y 75
maxusers 32 - 32
maxvgs 10 - 10
mesg 1 - 1
minfree - - 0
modstrmax 500 - 500
msgmap 42 - (2+MSGTQL)
msgmax 8192 Y 8192
msgmnb 16384 Y 16384
msgmni 50 - 50
msgseg 2048 - 2048
msgssz 8 - 8
msgtql 40 - 40
nbuf 0 - 0
ncallout 515 - (16+NKTHREAD)
ncdnode 150 - 150
nclist 612 - (100+16*MAXUSERS)
ncsize 5596 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
ndilbuffers 30 - 30
netisr_priority - - -1
netmemmax - - 0
nfile 910 - (16*(NPROC+16+MAXUSERS)/10+32+2*(NPTY+NST
NSTRTEL))
nflocks 200 - 200
nhtbl_scale 0 - 0
ninode 476 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 499 - (((NPROC*7)/4)+16)
nni - - 2
no_lvm_disks 0 - 0
nproc 276 - (20+8*MAXUSERS)
npty 60 - 60
nstrpty 60 - 60
nstrtel 60 - 60
nswapdev 10 - 10
nswapfs 10 - 10
nsysmap 800 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 800 - ((NPROC)>800?2*(NPROC):800)
o_sync_is_o_dsync 0 - 0
page_text_to_local - - 0
pfdat_hash_locks 128 - 128
public_shlibs 1 - 1
region_hash_locks 128 - 128
remote_nfs_swap 0 - 0
rtsched_numpri 32 - 32
scroll_lines 100 - 100
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
select_enh 0 Y 0
sema 1 - 1
semaem 16384 - 16384
semmap 66 - (SEMMNI+2)
semmni 64 - 64
semmns 128 - 128
semmnu 30 - 30
semmsl 2048 Y 2048
semume 10 - 10
semvmx 32767 - 32767
sendfile_max 0 - 0
shmem 1 - 1
shmmax 0x4000000 Y 0X4000000
shmmni 200 - 200
shmseg 120 Y 120
st_ats_enabled 0 - 0
st_fail_overruns 0 - 0
st_large_recs 0 - 0
st_san_safe 0 - 0
streampipes 0 - 0
swapmem_on 1 - 1
swchunk 2048 - 2048
sysv_hash_locks 128 - 128
tcphashsz 0 - 0
timeslice 10 - (100/10)
timezone 420 - 420
unlockable_mem 0 - 0
vas_hash_locks 128 - 128
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vol_checkpt_default 10240 - 10240
vol_dcm_replay_size 262144 - (256*1024)
vol_default_iodelay 50 - 50
vol_fmr_logsz 4 - 4
vol_max_bchain 32 - 32
vol_max_nconfigs 20 - 20
vol_max_nlogs 20 - 20
vol_max_nmpool_sz 4194304 - (4*1024*1024)
vol_max_prm_dgs 1024 - 1024
vol_max_rdback_sz 4194304 - (4*1024*1024)
vol_max_vol 8388608 - (8*1024*1024)
vol_maxio 256 - 256
vol_maxioctl 32768 - 32768
vol_maxkiocount 2048 - 2048
vol_maxparallelio 256 - 256
vol_maxspecialio 256 - 256
vol_maxstablebufsize 256 - 256
vol_min_lowmem_sz 524288 - (512*1024)
vol_mvr_maxround 256 - 256
vol_nm_hb_timeout 10 - (10)
vol_subdisk_num 4096 - 4096
vol_vvr_transport 1 - (1)
vol_vvr_use_nat 0 - (0)
volcvm_cluster_size 16 - 16
volcvm_smartsync 1 - 1
voldrl_max_drtregs 2048 - 2048
voldrl_min_regionsz 512 - 512
voliomem_chunk_size 65536 - (64*1024)
voliomem_maxpool_sz 4194304 - (4*1024*1024)
voliot_errbuf_dflt 16384 - 16384
voliot_iobuf_default 8192 - 8192
voliot_iobuf_limit 131072 - 131072
voliot_iobuf_max 65536 - 65536
voliot_max_open 32 - 32
volraid_rsrtransmax 1 - 1
vps_ceiling 16 - 16
vps_chatr_ceiling 1048576 - 1048576
vps_pagesize 4 - 4
vx_fancyra_enable 0 - 0
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vx_ninode 0 - 0
vxfs_max_ra_kbytes 1024 - 1024
vxfs_ra_per_disk 1024 - 1024
vxtask_max_monitors 32 - 32





********************************************
kmtune on MACHINE ON WHICH SYSTEM "WORKS"
********************************************

Parameter Current Dyn Planned Module Version
===============================================================================
NSTRBLKSCHED - - 2
NSTREVENT 50 - 50
NSTRPUSH 16 - 16
NSTRSCHED 0 - 0
STRCTLSZ 1024 - 1024
STRMSGSZ 65535 - 65535
acctresume 4 - 4
acctsuspend 2 - 2
aio_listio_max 256 - 256
aio_max_ops 2048 - 2048
aio_physmem_pct 10 - 10
aio_prio_delta_max 20 - 20
allocate_fs_swapmap 0 - 0
alwaysdump 1 - 1
bcvmap_size_factor 2 - 2
bootspinlocks - - 256
bufcache_hash_locks 128 - 128
bufpages 0 - (NBUF*2)
chanq_hash_locks 256 - 256
core_addshmem_read 0 Y 0
core_addshmem_write 0 Y 0
create_fastlinks 0 - 0
dbc_max_pct 50 - 50
dbc_min_pct 5 - 5
default_disk_ir 0 - 0
desfree - - 0
disksort_seconds 0 - 0
dmp_rootdev_is_vol 0 - 0
dmp_swapdev_is_vol 0 - 0
dnlc_hash_locks 512 - 4*128
dontdump 0 - 0
dskless_node - - 0
dst 1 - 1
effective_maxpid - - ((NPROC<=30000)?30000:(NPROC*5/4))
eisa_io_estimate - - 0x300
enable_idds 0 - 0
eqmemsize 15 - 15
executable_stack 1 - 1
fcp_large_config 0 - 0
file_pad - - 10
fs_async 0 - 0
ftable_hash_locks 64 - 64
hdlpreg_hash_locks 128 - 128
hfs_max_ra_blocks 8 - 8
hfs_max_revra_blocks 8 - 8
hfs_ra_per_disk 64 - 64
hfs_revra_per_disk 64 - 64
hp_hfs_mtra_enabled 1 - 1
hpux_aes_override - - 0
initmodmax 50 - 50
io_ports_hash_locks 64 - 64
iomemsize - - 40000
ksi_alloc_max 16384 - (NPROC*8)
ksi_send_max 32 - 32
lotsfree - - 0
max_async_ports 50 - 50
max_fcp_reqs 512 - 512
max_mem_window 0 - 0
max_thread_proc 3000 - 3000
maxdsiz 2063835136 - 2063835136
maxdsiz_64bit 0x40000000 - 0x0000000040000000
maxfiles 2048 - 2048
maxfiles_lim 2048 Y 2048
maxqueuetime - - 0
maxssiz 0x800000 - 0x00800000
maxssiz_64bit 0x800000 - 0x00800000
maxswapchunks 2048 - 2048
maxtsiz 0x4000000 Y 0x04000000
maxtsiz_64bit 0x40000000 Y 0x0000000040000000
maxuprc 75 Y 75
maxusers 512 - 512
maxvgs 10 - 10
mesg 1 - 1
minfree - - 0
modstrmax 500 - 500
msgmap 42 - (2+MSGTQL)
msgmax 8192 Y 8192
msgmnb 16384 Y 16384
msgmni 50 - 50
msgseg 2048 - 2048
msgssz 8 - 8
msgtql 40 - 40
nbuf 0 - 0
ncallout 6000 - 6000
ncdnode 150 - 150
nclist 8292 - (100+16*MAXUSERS)
ncsize 7848 - (NINODE+VX_NCSIZE)+(8*DNLC_HASH_LOCKS)
ndilbuffers 30 - 30
netisr_priority - - -1
netmemmax - - 0
nfile 4097 - 4097
nflocks 200 - 200
nhtbl_scale 0 - 0
ninode 2728 - ((NPROC+16+MAXUSERS)+32+(2*NPTY))
nkthread 6000 - 6000
nni - - 2
no_lvm_disks 0 - 0
nproc 2048 - 2048
npty 60 - 60
nstrpty 60 - 60
nstrtel 60 - 60
nswapdev 10 - 10
nswapfs 10 - 10
nsysmap 4096 - ((NPROC)>800?2*(NPROC):800)
nsysmap64 4096 - ((NPROC)>800?2*(NPROC):800)
o_sync_is_o_dsync 0 - 0
page_text_to_local - - 0
pfdat_hash_locks 128 - 128
public_shlibs 1 - 1
region_hash_locks 128 - 128
remote_nfs_swap 0 - 0
rng_bitvals - - 654 rng -
rng_log_deskew - - 0 rng -
rng_log_raw - - 0 rng -
rng_sleeptime - - 2 rng -
rtsched_numpri 32 - 32
scroll_lines 100 - 100
scsi_max_qdepth 8 Y 8
scsi_maxphys 1048576 - 1048576
sema 1 - 1
semaem 16384 - 16384
semmap 66 - (SEMMNI+2)
semmni 64 - 64
semmns 128 - 128
semmnu 30 - 30
semmsl 2048 Y 2048
semume 10 - 10
semvmx 32767 - 32767
sendfile_max 0 - 0
shmem 1 - 1
shmmax 0x4000000 Y 0X4000000
shmmni 200 - 200
shmseg 120 Y 120
st_ats_enabled 0 - 0
st_fail_overruns 0 - 0
st_large_recs 0 - 0
st_san_safe 0 - 0
streampipes 0 - 0
swapmem_on 1 - 1
swchunk 2048 - 2048
sysv_hash_locks 128 - 128
tcphashsz 0 - 0
timeslice 10 - (100/10)
timezone 420 - 420
unlockable_mem 0 - 0
vas_hash_locks 128 - 128
vnode_cd_hash_locks 128 - 128
vnode_hash_locks 128 - 128
vol_checkpt_default 10240 - 10240
vol_dcm_replay_size 262144 - (256*1024)
vol_default_iodelay 50 - 50
vol_fmr_logsz 4 - 4
vol_max_bchain 32 - 32
vol_max_nconfigs 20 - 20
vol_max_nlogs 20 - 20
vol_max_nmpool_sz 4194304 - (4*1024*1024)
vol_max_prm_dgs 1024 - 1024
vol_max_rdback_sz 4194304 - (4*1024*1024)
vol_max_vol 8388608 - (8*1024*1024)
vol_maxio 256 - 256
vol_maxioctl 32768 - 32768
vol_maxkiocount 2048 - 2048
vol_maxparallelio 256 - 256
vol_maxspecialio 256 - 256
vol_maxstablebufsize 256 - 256
vol_min_lowmem_sz 524288 - (512*1024)
vol_mvr_maxround 256 - 256
vol_nm_hb_timeout 10 - (10)
vol_subdisk_num 4096 - 4096
vol_vvr_transport 1 - (1)
vol_vvr_use_nat 0 - (0)
volcvm_cluster_size 16 - 16
volcvm_smartsync 1 - 1
voldrl_max_drtregs 2048 - 2048
voldrl_min_regionsz 512 - 512
voliomem_chunk_size 65536 - (64*1024)
voliomem_maxpool_sz 4194304 - (4*1024*1024)
voliot_errbuf_dflt 16384 - 16384
voliot_iobuf_default 8192 - 8192
voliot_iobuf_limit 131072 - 131072
voliot_iobuf_max 65536 - 65536
voliot_max_open 32 - 32
volraid_rsrtransmax 1 - 1
vps_ceiling 16 - 16
vps_chatr_ceiling 1048576 - 1048576
vps_pagesize 4 - 4
vx_fancyra_enable 0 - 0
vx_maxlink 32767 - 32767
vx_ncsize 1024 - 1024
vx_ninode 0 - 0
vxfs_max_ra_kbytes 1024 - 1024
vxfs_ra_per_disk 1024 - 1024
vxtask_max_monitors 32 - 32

********************************************

Any suggestions???

[A. Clay Stephenson]

Start by making maxdsiz, maxdsiz_64bit, maxssiz, maxssiz_64bit, maxtsiz, maxtsiz_64bit, shmmax, maxfiles, maxfile_lim the same on the good and bad boxes. I would also reduce dbc_max_pct on both boxes to around 20% for better performance. You should also compare msg*, sem*, and shm* values on the two boxes.

[Jaideep Krishnan]

HI,

WE RAN "GDB" AGAINST THE CORE AND CAME TO KNOW THAT FOLLOWING WAS THE OUTCOME:

=======================================
(no debugging symbols found)...#0 0xc019b0e0 in free+0x130 ()
from /usr/lib/libc.2
=======================================

ANY POINTERS TO WHY IS GDB POINTING TO /usr/lib/libc.2 ?
BELIEVE IT IS A SYSTEM LIBRARY AND NOT RELATED TO THE APPLICATION.
WHAT COULD BE THE POSSIBLE REASONS ?

YOUR QUICK HELP WILL BE GREATLY APPRECIATED !!!

THANKS A LOT.

[Jaideep Krishnan]

HI, FOR YOUR HANDY REFERENCE, HERE IS THE COMPLETE CONTENTS FOR THE GDB OUTPUT.

===========================================================================

warning: Load module /opt/sun/mps/serverroot/admin-serv/config/../../bin/https/b
in/ns-httpd has been stripped

(no debugging symbols found)...
Core was generated by `ns-httpd'.
Program terminated with signal 10, Bus error.

warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libns-httpd40.sl has
been stripped

(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libnsfc.sl has been s
tripped

(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libnstp.sl has been s
tripped


(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libdirmon4.sl has bee
n stripped

(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libnstime.sl has been
stripped


(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libsupport.sl has bee
n stripped

(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libares3.sl has been
stripped


(no debugging symbols found)...(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/liblibsi18n.sl has be
en stripped


(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/liblibdbm.sl has been
stripped

(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libgetprop.sl has bee
n stripped


(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libnsprwrap.sl has be
en stripped

(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
warning: Load module /opt/sun/mps/serverroot/bin/https/lib/libxerces-c.sl has be
en stripped


(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
---Type to continue, or q to quit---
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...(no debugging symbols found)...
(no debugging symbols found)...#0 0xc019b0e0 in free+0x130 ()
from /usr/lib/libc.2

===========================================================================

Please let us know.

[Gregory Fruth]

1) IT IS CONSIDERED RUDE WHEN YOU TYPE IN ALL
CAPS.

2) You are 0 for 32 in assigning points to
responses. Please consider assigning points.

3) Please provide OS and machine type info.
Of these two machines, is one PA-RISC and
the other Itanium, by chance?

[Jaideep Krishnan]

1) IT IS CONSIDERED RUDE WHEN YOU TYPE IN ALL
CAPS.

- Thank you so much for this handly knowledge.

2) You are 0 for 32 in assigning points to
responses. Please consider assigning points.

- I haven't. I will start with this. Point 1 already deserves points!

3) Please provide OS and machine type info.
Of these two machines, is one PA-RISC and
the other Itanium, by chance?

- No both are PA-RISC. Uname output is the following:

m/c where it crashed: HP-UX B.11.11 U 9000/800 870743066 unlimited-user license
FRU info: PA 8800 CPU Module 3.1

m/c where it works: HP-UX B.11.11 U 9000/800 2640250083 unlimited-user license
FRU info: PA 8800 CPU Module 3.1


Anymore information you need, please let us know.

Thanks a million!!!

[Jaideep Krishnan]

Hi,

One quick question:

From the below, can you please let me know what could this mean??

Can the problem be in the last line??

==========================================================================
(gdb) bt
#0 0xc019b0e0 in free+0x130 () from /usr/lib/libc.2
#1 0xc0162050 in __getipnodebyname_r+0x9b8 () from /usr/lib/libc.2
#2 0xc01615f4 in getipnodebyname+0x94 () from /usr/lib/libc.2
#3 0xc2fab004 in PR_GetIPNodeByName+0x88 ()
from //opt/sun/private/lib/libnspr4.sl
#4 0xc14babc0 in prldap_connect+0x5c ()
from //opt/sun/private/lib/ldk_dsas/libprldap50.sl
#5 0xc643bf3c in nsldapi_connect_to_host+0x68 ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#6 0xc64417b0 in nsldapi_new_connection+0x108 ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#7 0xc643b4d4 in nsldapi_open_ldap_defconn+0x8c ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#8 0xc6440bec in nsldapi_send_server_request+0x32c ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#9 0xc6440868 in nsldapi_send_initial_request+0x3c ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#10 0xc6447df4 in simple_bind_nolock+0x3dc ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#11 0xc6447a0c in ldap_simple_bind+0x28 ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#12 0xc6447e3c in ldap_simple_bind_s+0x24 ()
from //opt/sun/private/lib/ldk_dsas/libldap50.sl
#13 0xc2b6e2c8 in admldap_simple_bind_s+0x4c ()
from //opt/sun/private/lib/libadminutil53.sl
#14 0xc2b6d524 in admldapBuildInfoCbk+0x248 ()
from //opt/sun/private/lib/libadminutil53.sl
#15 0xc2b6d9e0 in admldapBuildInfo+0x20 ()
from //opt/sun/private/lib/libadminutil53.sl
#16 0xc272c08c in _admin40_check_ds_availability_init+0xb0 ()
from /opt/sun/mps/serverroot/bin/https/lib/libAdmservPlugin.sl
#17 0xc272c550 in admin40_check_ds_availability_init+0x4c ()
from /opt/sun/mps/serverroot/bin/https/lib/libAdmservPlugin.sl
#18 0xc3fac440 in ()
from /opt/sun/mps/serverroot/bin/https/lib/libns-httpd40.sl
#19 0xc3fab2bc in ()
from /opt/sun/mps/serverroot/bin/https/lib/libns-httpd40.sl
#20 0xc3fab8e0 in ()
from /opt/sun/mps/serverroot/bin/https/lib/libns-httpd40.sl
#21 0xc3fa68c4 in ()
from /opt/sun/mps/serverroot/bin/https/lib/libns-httpd40.sl
#22 0xc4045660 in ()
from /opt/sun/mps/serverroot/bin/https/lib/libns-httpd40.sl
#23 0x372c in ()
(gdb)
==========================================================================

Thanks,
jaideep

[A. Clay Stephenson]

Your crash is a result of free() probably as a result of receiving an invalid pointer. For example, being asked to free a memory location that was never dynamically allocated. Free() was in turn called by __getipnodebyname() which was called by getipnodebyname() and so on. The first thing to check is that the versions of libc.* are identical on the "good" and "bad" boxes. Man getipnodebyname for clues -- the first thing that comes to mind is comparing your /etc/nsswitch.conf, /etc/resolv.conf, and /etc/hosts entries on the two boxes.

If you have the source code, compile it with -g and no optimization and don't strip the code. Let it crash and debug again.

[Jaideep Krishnan]

Hi,

I resolved this issue by just copying a library (unstripped libldap50.sl) from my workspace to the runtime location over existing libldap50.sl, and I was was able to start the service, as the error went away.

However, now when I am trying to do a different operation (start the associated console), the authentication fails with similar error.

Now which library should I copy??

I have a "core" gererated for a binary that crashed in the process, known as "userauth".

I followed these 2 steps:

gdb userauth core
bt

However it did not give any meaningful information, unlike the earlier case where it directed me to libldap50.sl, one way or the other, that helped me resolve the issue.

Is there any more specific way to read this core and get some information from the bottom of it? Please indicate.

Thanks,
Jaideep

[Jaideep Krishnan]

Hi,

Also, attaching the section of the TRUSS outout here :

[Jaideep Krishnan]

We compiled our sources using -c flag.

But unfortunately we are not in a position to compile and build the libraries (that are used in the linkage process with our source) also with -c flag.

Hence it thows same messages while running GDB.
"... (no debugging symbols found) ..."

Can someone please suggest on how to reach the exact library that is causing this?

Thanks a lot,
Jaideep

[A. Clay Stephenson]

You need to use the "-g" flag in addition to the "-c". I must say, copying shared libraries from one HP-UX box to another is an excellent method of getting the software database hopelessly confused so that subsequent software updates and patches become easily corrupted and impossible to return to a known and stable state.

You need to develop a systematic approach using the software management tools like swlist to compare the systems. The bad news is that your copying has made those comparisons meaningless.

[Jaideep Krishnan]

Sorry, what I really meant was "-g" option, not "-c".

We compiled our sources using "-g" flag.

But we are not in a position to compile and build the libraries (that are used in the linkage process with our source) also with "-g" flag.

Hence it thows same messages while running GDB.
"... (no debugging symbols found) ..."

Can you please indicate on how to reach the exact library that is causing this fault?

[Jaideep Krishnan]

Are there any workarounds possible in doing this?

I even copied all the libraries that I used with the crashing binary during the compilation time, to the runtime location - the error still stays.

Presently we would be interested in going for a work-around solution, (E.g., copying a different version of the same library to the runtime location, etc.)

Is there any way of knowing through GDB which libray could be replaced (mind you it is a CGI that has crashed due to SIGBUS error).

Please let us know. It would be highly appreciated.

Thanks a lot,
Jaideep

[Jaideep Krishnan]

Hi,

Are there any tools for comparing the settings of the two machines: I have executed the following commands on both the machines for comparisons, and found that parametes relating to both these machines were matching:

*********************************
ulimit -Sa
ulimit -Ha
/usr/sbin/swapinfo
cat /stand/system
kmtune
cat /etc/nsswitch.conf
cat /etc/resolv.conf
*********************************

Can you please comment?

Thanks,
Jaideep