Help with Symbols in Debug Core File

[Tharish_1]

Hi,
I am from the HP Identity Management Group and we use a software called SiteMinder from Computer associates which is installed on the HP-UX 11.23 Servers. The SiteMinder Application is crashing regularly producing core files. We had contacted the vendor with all the logs and core files to CA support and after analyzing the core file, they think that they will need some help from HP-UX Operating systems group to explain some of the symbols in the core file since the call which causes the crash is a HP-UX System Call.
The stack trace is attached to the case.
Any help would be greatly appreicated.

[Dennis Handly]

I don't see anything in your stack trace that says what's wrong. What was the signal?

Most threads are in __ksleep, _poll_sys, __sigtimedwait_sys, _recv_sys, __pause_sys or _nanosleep_sys.

There is a thread in _write_sys, _open_sys
_lwp_kill and clock_gettime.

The stack for thread 106 (_lwp_kill) isn't correct. What version of gdb do you have? 5.5 is the latest.

What were CA's specific questions about symbols? You might want to ask them why they are calling pthread_kill.

[Tharish_1]

Hi Dennis,

The pthreadkill is called by the vendor when the applications fails to create a new thread.

We use GDB version 5.2.

There is not much info about the signal which cuased abort

Core was generated by `smpolicysrv'.
Program terminated with signal 6, Aborted.
SI_UNKNOWN - signal of unknown origin

[Dennis Handly]

>We use GDB version 5.2.

You might try the latest, 5.5.

>The pthread_kill is called by the vendor when the applications fails to create a new thread.

Then you need to solve that. Either you are out of memory or your kernel config needs to allow more threads. It would have been helpful if vendor printed the error return from pthread_create. If you are lucky, it may still be in your core file.

[Tharish_1]

Thank you Dennis!
Yes,I will try GDB 5.5
Also from our recent investigations it does point towards an out of memory problem. We see Some problems in the heapstat.Also CA promised that they can give debug lib which can log more info on why the creating a new thread failed.We are also using a utility as adviced by a GTEC engineer called aps which can run against the running CA
process to see where the bulk of the heap expansions are coming from



***** Sorted by Count *****

Size Count bytesAll Type %ofAlloc %ofTotal
0x00000018 1035604 24854496 S 3.2468 2.8962
0x00000078 665501 79860120 S 10.4323 9.3059
0x00000068 617351 64204504 S 8.3872 7.4816
0x00000028 356846 14273840 S 1.8646 1.6633
0x00000038 349562 19575472 S 2.5572 2.2811
0x00000050 311921 24953680 S 3.2598 2.9078
0x00000030 237780 11413440 S 1.4910 1.3300
0x00000741 231590 430062630 L 56.1800 50.1139
0x00000040 114777 7345728 S 0.9596 0.8560
0x00000048 75302 5421744 S 0.7083 0.6318
0x00000088 10727 1458872 S 0.1906 0.1700
0x00000411 9415 9801015 L 1.2803 1.1421
0x00000058 9288 817344 S 0.1068 0.0952
0x00000020 9241 295712 S 0.0386 0.0345
0x000005a9 8201 11883249 L 1.5523 1.3847
0x000000f8 6273 1555704 S 0.2032 0.1813
0x00000421 4096 4329472 L 0.5656 0.5045
0x000000b8 2058 378672 S 0.0495 0.0441

[Dennis Handly]

>our recent investigations it does point towards an out of memory problem.

The size of the corefile's data area can be a good estimate of being out of memory. Just compare to maxdsiz. If close, that's it. If not, it could be out of swap space.