topic Re: Analyzing hanging process in Operating System - OpenVMS

Analyzing hanging process

Kirsten Knüttel — Wed, 17 Mar 2004 08:14:56 GMT

Hello,

I need a bit help with analyzing a hanging process. This process is hanging sporadically.
So, I've got a process with a subprocess. The subprocess seems to hang in a mailbox
SDA>show proc/chann

0020 7FF30020 00000000 Busy MBA5953:

The mailbox is empty. So we think, that this process waits for something that the main process should send him. Can I see exactly where the main process hangs at the moment? A show proc/chann didn't show something busy. But he must wait for something and we want to know what he is waiting for.
We have the problem that we know not enough about the product that we use (SAS) to analyze this deeper.
Perhaps I've explained it in a wrong way, but it is really difficult do explain it.

In the attachment I wrote down a show sys of the main and the child process. Perhaps this helps you.

Kind regards,

Kirsten

Re: Analyzing hanging process

Ian Miller. — Wed, 17 Mar 2004 09:47:48 GMT

the EFWM in SDA will show you which event flag the process is waiting for.
SHOW STACK for the processes may show you PC addresses that can be matched to a MAP for the image.

Re: Analyzing hanging process

Wim Van den Wyngaert — Wed, 17 Mar 2004 10:23:23 GMT

Also try SDA show proc to check the quotas of both processes.

Re: Analyzing hanging process

Martin P.J. Zinser — Wed, 17 Mar 2004 22:44:23 GMT

Hello Kirsten,

if this is SAS as in SAS Institute software, then what exactly are you doing with mailboxes?

IIRC these transports were deprecated years ago. Also if you do have a reprodcuer do make sure this gets escalated to the HQ in Cary, SAS Germany in Heidelberg does not have the resources to investigate a problem like this.

Greetings, Martin

P.S. Who used to have business cards from SI in a former life ;-)

Re: Analyzing hanging process

Brian Reiter — Thu, 18 Mar 2004 03:15:21 GMT

Hi there,

In my experience an outstanding QIO on a mailbox is shown as busy using SHO PROC/CHAN.

Its a pity really that its third party software. My approach to software hangs/tight loops (mutex and processes in COM) is to extract a number of PC values. You can SHO PROC/CONT and from ANAL/SYS use SHO CALL and SHO CALL/NEXT (Look at the return address on the stack).

With a good number of values run the image up under debug (doesn't need to start running or even be a debug image). Then from the debug prompt just

SET RADIX HEX
SET MODU/ALL
EX

Depending on how its been compiled you may get useful information (modules and approximate line numbers) which can be used to escalate the problem.

cheers

Brian

Re: Analyzing hanging process

Hein van den Heuvel — Thu, 18 Mar 2004 22:00:22 GMT

Ideally, but unlikely, the application has an interface to activate the debugger (lib$signal ss$_debug).

SHOW PROC/CONT is a crude, but OK, first step to get an impression on what the process might be doing. Watch the 'state' and PC, the snapshot you posted shows an x80xxxxxx : system space! So in that instance it is nor running user code. But watch it for a while and you'll catch user space adddresses which can be translated back by using "run/debu ", set mode hex, exa/inst . By watching for a while,
with an eye on the IO counters, state and PC your soon get an impression what the process might be doing (or no doing).

Next good tool is ANAL/SYST. As replied before, set proc, show proc/stack and so on.
System address, like you captured, are valies in all contexts, so you can just do for example:

SDA> exa/inst 8014DC54
F11BXQP_NPRO+01C54: LDQ R27,#X0058(R2)

This was done on my box, running 7.1, unlikey to match yours.
To get a more exact picture try:

SDA> read/exec
SDA> exa/inst 8014DC54
MAKE_DIRINDX_C+00360: LDQ R27,#X0058(R2)

There is a series of SDA extentions 'clue' and 'trc' that open up a wolrd of debugging help, but that's too much for here now.

Any chance you can join the OpenVMS bootcamp?
http://h71000.www7.hp.com/symposium/index.html
There is a whole session and tehn soem dedicated to this subject! "Troubleshooting Hung and Looping Processes, course M405".

http://h71000.www7.hp.com/symposium/may_2004/M405.html

See you there?!

Hein.

Re: Analyzing hanging process

Willem Grooters — Fri, 19 Mar 2004 08:00:14 GMT

(off topic)

Any chance you can join the OpenVMS bootcamp?

I'm still looking for a sponsor....

Willem

Re: Analyzing hanging process

Martin P.J. Zinser — Fri, 19 Mar 2004 22:31:04 GMT

Hello Hein,

while generally speaking your analysis is very valuable (as usual ;-), in this particular case I do not think it should be the customers problem to perform this analysis. SAS is a very very complex application and comes with a considerable yearly license fee to cover new releases and support. So all there should be to do really is to get the pertinent information to the right people and let the engeneers in Cary do their work.

Greetings, Martin

Re: Analyzing hanging process

Hein van den Heuvel — Sat, 20 Mar 2004 01:17:59 GMT

I fully agree Martin. I fact, that is usually my line! I don't know why I did not stick that one in this time :-).

Actually, I looking back at the topic I do know. The customer was already going in that direction, proven by the screenshots that were attched. I was lining out possible futher steps loosing sight of the fact that this is a major, supported app, not homegrown.

An of course there is nothing like 'being there', and a little pre-work by the customer with active overvations while the problem is happening will allow to bring a much more problem clear case to report.

Instead of 'it's stuck', they can hopefully indicate it is stuck in module xyx calling service abc with filesuch and so open.

It makes the suggestion for that particular tringin session more relevant. It'll be too late for an immediate problem but it would be a god skill in the back pocket should this happen again.

Kirsten, you did call in the SAS troops for support right?!
(hmmm, I know an HP person or two (Susan, Carl) who work on site at SAS in North Carolina. In fact, up untill half a year they were in the same group I am in! :-)

Sorry, I'm rambling, Have a great weekend you all !
(already well under way for half the world :-)

Regards,
Hein.

Re: Analyzing hanging process

Kirsten Knüttel — Mon, 22 Mar 2004 01:56:46 GMT

Hello,

hopy you had a nice weekend and many thanks for all of your help.

Normally we contact SAS for their help. But the problem at this time ist that the error is not reproducable. The procedure runs every 2 days and normally everything is O.K. But from time to time it hangs. And we don't know the circumstances that makes the procedure hanging. So it is very difficult to make a call for it.
So we wanted to find these circumstances. But I think now that we've got collected enough to call SAS (resp. a collegue has the contacts).

@Martin:
Yes, it is this SAS. And we really don't know what it is doing with these mailboxes. It was the first time that we've seen that SAS uses mailboxes

@Hein:
An exa/inst brings:
EXE_STD$SYNCH_LOOP_C+00084: BLBC R0,#X000050

Perhaps it helps SAS a bit more.
And I really have no chance to join the OpenVMS bootcamp. I have no sponsor for it. I even don't get UNIX training courses although we want to migrate to UNIX (sorry, we must migrate to UNIX).

Kind regards,

Kirsten

Re: Analyzing hanging process

Wim Van den Wyngaert — Mon, 22 Mar 2004 03:08:08 GMT

Long time ago, I had SAS with VMS 6.2.

It used the whole pagefilequota is was assigned. If you increased the pagefilequota, it used the increased value also completely. You had to specify somewhere in a config file how much it should reserve for other operations. May be, the value reserved for other operations is simply to low.

Of course, if you checked the quotas already, you can ignore this posting.

Re: Analyzing hanging process

Martin P.J. Zinser — Mon, 22 Mar 2004 14:59:29 GMT

Hello Kirsten,

assuming the code does not contain any confidential information, would you mind posting an example of the failing code? Do you use Access/Share/Connect in there?

Greetings, Martin