Hanging process. C/C++ function to get process information.

[Miguel Sanchez_2]

We have a C++ program running on Open VMS 7.1.
We start several instances of the program (>20). The program is connected to a message bus to get requests and control commands like "exit".
Some times, when we send an "exit" command some of the instances hang.
I have added additional logs to the program to get a clue where the process is hanging.
The logs show that the main() function runs until the last line in code ("return").
What could be the cause for that.
I would like to log the process information just before "return" in main() hopping to get an idea on what the process is doing.
How can I get the process information?

Thanks,
Miguel

[Antoniov.]

Miguel,
welcome to vms forum!

You posted too little informazions for helping you. I hint you checking for return code status.
I guess you have one or both troubles:
a) Your software writes using bad pointer but inside its allocation space; in this case your application doesn't get Invalid Allocation Error when which's using pointer but when exits;
b) On exit your application executes some function that's use a bad pointer.

About return, AFAIK on main function it's better use exit() but application can work with return too.

Antonio Vigliotti

[Ian Miller.]

what state is the process in when it is not responding?

[Miguel Sanchez_2]

Thanks Antoniov and Ian for your reply,

I should say that I am not an VMS expert. I do the development on UNIX and than port the program to VMS.
Please, let me know what kind of information you need to help me.

Is there a possibility to get more proccess information before exiting the main function?
For example, I think it could be useful to write a proccess dump (stack dump) like after a crash. How can I do that?.
Do I have a chance to know if the application ist using bad pointers?

Ian:
How can I get the proccess state you are asking for?
Below is the process information provided by "show proccess/all/id=xxxx":

24-JUN-2005 09:13:06.37 User: AA_XX Process ID: 49C32B1C
Node: AAA Process name: "WSM_BGH0LW_P_1"

Terminal:
User Identifier: [BBBUSR,AA_XX]
Base priority: 4
Default file spec: Not available
Number of Kthreads: 2

Devices allocated: BG411:

Process Quotas:
Account name:
CPU limit: Infinite Direct I/O limit: 960
Buffered I/O byte count quota: 97440 Buffered I/O limit: 2688
Timer queue entry quota: 199 Open file quota: 294
Paging file quota: 966096 Subprocess quota: 50
Default page fault cluster: 64 AST quota: 499
Enqueue quota: 2999 Shared file limit: 0
Max detached processes: 0 Max active jobs: 0

Accounting information:
Buffered I/O count: 3756229 Peak working set size: 31792
Direct I/O count: 889650 Peak virtual size: 202976
Page faults: 2711 Mounted volumes: 0
Images activated: 17
Elapsed CPU time: 0 00:53:46.39
Connect time: 3 23:51:21.52

Authorized privileges:
GROUP NETMBX OPER SYSLCK TMPMBX

Process privileges:
GROUP may affect other processes in same group
NETMBX may create network device
SYSLCK may lock system wide resources
TMPMBX may create temporary mailbox

Process rights:
AA_XX resource

System rights:
SYS$NODE_AAA

Auto-unshelve: on

Image Dump: off

Soft CPU Affinity: off

Parse Style: Traditional

Home RAD: 0

There is 1 process in this job:

WSM_BGH0LW_P_1 (*)

Thanks,
Miguel

[Volker Halle]

Miguel,

you can simply obtain the state of your process with SHOW SYSTEM/PROC=

As you are running an old version of OpenVMS, you cannot use the SET PROC/DUMP=NOW command to force a process dump, when your process is hung.

But you can look at the hung process with SDA

$ ANAL/SYS
SDA> SET PROC/IND=
SDA> SHOW PROC
SDA> SHOW PROC/LOCK
SDA> SHOW PROC/CHAN
SDA> CLUE CALL
SDA> SHOW STACK

The information obtained by SDA needs to be reviewed by someone experienced with OpenVMS internals.

Volker.

[Robert Gezelter]

Miguel,

Which log are you looking at?

- Bob Gezelter, http://www.rlgsc.com

[Miguel Sanchez_2]

Thanks Volker for the commands,

Attached is the information provided by SDA for two hanging processes.

Thanks,
Miguel

[Volker Halle]

Miguel,

o.k. - so the program 'hangs' in PTRHEADs (indicated by the process state of HIBernate and PTRHEAD$RTL as the caller of SYS$HIBER_C). You now need to use the SDA PTHREAD extension to look at the PTHREAD status of the process:

SDA> SET PROC/IND=
SDA> pthread help ! gives some help about options

SDA> pthread t -1a ! show thread state

You are running OpenVMS Alpha V7.2-2. Make sure you have all available patches installed. There is a VMS722_PTRHEAD-V0100 patch (among many others).

But before installing patches, start with looking at the thread states of your application first, maybe you can find the problem in the application...

Volker.

[Miguel Sanchez_2]

Hello Volker,

Attached is the threads information provided by SDA.

Thanks,
Miguel

[Volker Halle]

Miguel,

thanks for the info. The situation is the same in both hanging processes:

The exit-handling thread is blocked on a mutex

SDA> pthread mutexes

will show the mutexes. To see which code is running in the exit-handling thread, one could try:

SDA> pthread stack

and then examine the current stack of thread -4

SDA> SHOW STACK ;60

If you can locate a frame pointer, try SDA> SHOW CALL , otherwise just provide the output from the commands above.

Please be aware, that this might become a lengthy troubleshooting effort. Did you check, whether you have the most recent PTHREAD patches installed ?

Volker.

[Miguel Sanchez_2]

Hell Volker,

Attached is the output of SDA with the thread stack information.

Thanks,
Miguel

[Anthony Magana]

I had some "fun" debugging a C++ program that was threaded a few months back. A few points that might help:

- You can use debug/keep on the process thats hanging. Then you can actually do a show calls to see where in your app its hanging. The sequence you would follow is:
1. debug/keep
2. At debug prompt : connect
3. type cmd: show task/all to see all
the threads running in your app and their
state.
4. set task to set context to
a particular thread.
5. show calls to see the call stack for
the thread.

Even better if you have the code built debug, you can then set break to a function and see where in the code its sitting.

- Other common problem that is encountered with threaded apps is the the pthread stack size being too small. Sometimes the app doesn't fail in obvious ways. The default stack size is oftentimes too small. You can explicitly set it with a call to pthread_attr_setstacksize. If some thread that is is supposed to sync up with some other threads dies before that sync point the other threads could be waiting there forever waiting for that sync point which will never happen because of the dead thread.

[Volker Halle]

Miguel,

unfortunately, the SDA> SHOW STACK 277990;60 (of the blocked exit-handling thread) did not provide enough information. Next time, try SDA> SHOW STACK ...;100

Or - even better - try the kept debugger as suggested by Anthony.

Volker.

[Miguel Sanchez_2]

Hi Volker,

I have attached the output from SDA, this time with more pthread stack info.

I have checked with the system administrator and they have the latest pthread patches installed.

Anthony:
We have tried to debug the process but we get the error:
DBG> connect WSM_BGH1SW_P_1
%DEBUG-E-NOCONNECT, CONNECT command failed
-SYSTEM-W-NONEXPR, nonexistent process

But that is actually the process name. What are we doing wrong?.
Another question: how can I get the pthread stack size?

Thanks for your help,
Miguel

[Volker Halle]

Miguel,

to use DBG> CONNECT process-name, both processes must be running under the same group UIC. Alternatively you can use:

DBG> CONNECT %PROCESS_PID pid-of-process

If it hangs the next time and the thread -4 stack pointer is again 277990, you can look at the call frames with

SDA> SHOW CALL 277A60
SDA> SHOW CALL/NEXT
and so on, or

SDA> CLUE CALL 277A60

277A60 seems to be a valid FP pointing to the stack where the PDSC address (Procedure Descriptor) of PTHREAD_MUTEX_BLOCK is stored.

When looking at the call frames, try to find the first Return address in P0 space, i.e. in your WMSRV image. The look up the map and source listing and find out, what code is running at that address and what it's trying to do.

Volker.

[Anthony Magana]

In the app I worked with there was a mix of C and C++ code. The threads were explicitly created and started from C code with pthread_create calls. Before the call to the create we would set up the thread attribute with the desired stack size by calling pthread_attr_setstacksize. If you don't explicitly set the stack size there is some default that pthreads uses (don't remember offhand what it is). As I'm not that familiar with C++, I don't if there is some language native construct for creating threads in C++. I'd guess that you would be using the pthreads lib if the app explicitly codes to use threads.