topic Re: Infinite loop problem in Operating System - OpenVMS

Infinite loop problem

Kai.Zou — Wed, 05 Aug 2009 04:57:29 GMT

Hi, I am newbie here. And I met an Infinite loop problem.

The process paging file quota became critical after activated for 52 days and gave the following messages:
[Error creating space for command interpreter symbol table
Process quota exceeded]
And One more day later, the process was trapped into infinite loop.

I wondered that the reason might be the abuse of lib$find_file ( lack of lib$find_file_end ).
Here is my questions:
1. I knew that lack of lib$find_file_end would cause a memory leak, but would it cause lib$find_file trapped into dead loop?
2. When infinite dead loop occured, could I use SDA show stack command to determine the dead loop position ( function )? How is the detail way?

thanks very much.

Re: Infinite loop problem

John Gillings — Wed, 05 Aug 2009 05:12:49 GMT

Kai,

>reason might be the abuse of lib$find_file
>( lack of lib$find_file_end ).

That's certainly possible (and a rather advanced diagnosis for a "newbie"! :-). If you don't call lib$find_file_end, you can leak heap space and eventually run out of virtual memory (pagefile quota, frep0va, etc... what you hit depends on quotas).

>[Error creating space for command
>interpreter symbol table Process quota
>exceeded]

This sounds more like a DCL error? Are you running a program, or executing a procedure? Or, maybe SPAWNing a subprocess from a program?

>could I use SDA show stack command to
>determine the dead loop position

Yes, basic principle is to sample the PC and correlate it against your image map. See SDA> SHOW IMAGE to get the addresses of various images. Then use your link map to work out which routine it's in from the image offset. Then use your compilation listing to work out the line from the routine offset.

That said, if this is the result of hitting a memory limit, whatever code is looping is most likely a victim, not culprit, so it's not really helpful to find it. If you know you have cases where LIB$FIND_FILE isn't properly closed off, fix that issue first. When you restart your process, track it's VM usage to make sure it isn't increasing. If it is, try debugging using the heap analyzer.

Re: Infinite loop problem

Pramod Kumar M — Thu, 06 Aug 2009 04:42:37 GMT

1. I knew that lack of lib$find_file_end would cause a memory leak,but would it cause lib$find_file trapped into dead loop?

You can use SDA and look at the call stack using SHOW CALL/SUMMARY command to figure out whether you are stuck traversing the same calls again and again.

Are you sure you are stuck in a loop ?
Did you check the status of the process in SDA and figured out?
Is that process entered some kind of wait state from which it never returns?

I think if you are out of pgflquota or memory lib$ call should have returned an error status. If you think it's a bug please report to HP support.

Some of the commands I generally use to figure out VM usage is
SDA>sh proc/phd
$sh proc/cont

2. When infinite dead loop occured, could I use SDA show stack command to determine the dead loop position ( function )? How is the detail way?

I think you should be able to figure using SDA>sh call/summ, it's bit tricky and would require looking at the output multiple time over a period of time.

You can also put some print statements around your code while calling lib$find_file to figure how long it takes to return from that call.
If it never returns it could be a infinite loop.

You can also use PCS SDA extension ( SDA>PCS gives you the help). SDA>PCS SH TRACE would show you the pc's in a timely manner.

Regards,
Pramod

Re: Infinite loop problem

Kai.Zou — Thu, 06 Aug 2009 06:01:40 GMT

Hi, Dear Pramod,
The logic of the program I was tracking is like that:
while (1) {
call lib$find_file ( "*.csv", result buff,&context);
if search result buff contains a "*" then
return SS$_NORMAL;
else
rename the search result buff to "*.log"
}
call lib$find_file_end(&context);

This logic has the following problems:
1. The while(1) loop only has one existance, when result buff contains a "*" char.
2. when breaking from the loop, faile to call lib$find_file_end.

Usually, when no files, lib$find_file will return no file return value and the result buff is as same as input searching string(like "*.csv"), the logic seems to be no problem.

So I imagined the situation should be as the following:
1) lack of lib$find_file_end will cause a vm leak.
2) the vm leak increased every time when the above logic was called.
3) vm leak hit the limit of system vm avaliable.
4) lib$find_file failed, but the result contains no "*", the program was stuck in the while(1) loop.

Re: Infinite loop problem

Kai.Zou — Thu, 06 Aug 2009 06:24:41 GMT

Hi, Dear Pramod,
Thanks very much for your hints.

I had input some printf statement, and wrote a test program, it seemed that the printf statement had affected on the behavir of the program, and the outting log became random every time. But all went down finally because of vm critical.

I considered that the stuck was in the while loop. But no clear evidence. The problem occured recently but the reproduct would take 53 days.

Could u pls tell me, that my analysis had problem or something I missed?

Re: Infinite loop problem

Kai.Zou — Thu, 06 Aug 2009 06:40:23 GMT

Hi, the stack trace and show call output should be as the following:
note: the process entry function is dmm_main.

analy/system
sda> set process dmmprc1
sda> show stack

Process Stacks (on CPU 00)
--------------------------
Current Operating Stack (USER):
00000000.7AEE02E0 00000000.7AEE0318
00000000.7AEE02E8 00000000.0000000F
00000000.7AEE02F0 00000000.00000000
00000000.7AEE02F8 5653432E.00000000
SP => 00000000.7AEE0300 00000000.00018FB0 DMM_MAIN+18FB0 -> stuck point I think
00000000.7AEE0308 7AEE0530.010E270F DMM_MAIN+010E270F
00000000.7AEE0310 7AEE0330.010E00FF DMM_MAIN+010E00FF
00000000.7AEE0318 00000000.00000000
00000000.7AEE0320 7AEEAC88.0004A150 DMM_MAIN+4A150
00000000.7AEE0328 00000000.00049E18 DMM_MAIN+49E18

analy/system
sda> set process dmmprc1
sda> show call
sda> show call/next
DMMã ®callï¼ show call,show call/nextï¼
Call Frame Information
----------------------
Stack Frame Procedure Descriptor
Flags: Base Register = FP, No Jacket, Native
Procedure Entry: 00000000.00186260 DMM_MAIN+00186260
Return address on stack = 00000000.0017DCD4 DMM_MAIN+0017DCD4

Registers saved on stack
------------------------
7AEE2C48 00000000.000192C0 Saved R2 DMM_MAIN+192C0
7AEE2C50 00000000.00000005 Saved R3
7AEE2C58 00000000.0001863C Saved R4 DMM_MAIN+1863C
7AEE2C60 00000000.00000001 Saved R5
7AEE2C68 00000000.00000000 Saved R6
7AEE2C70 00000000.00000001 Saved R7
7AEE2C78 00000000.7AEEAC80 Saved R29

Re: Infinite loop problem

John Gillings — Thu, 06 Aug 2009 20:30:32 GMT

Kai,

>while (1) {
> call lib$find_file ( "*.csv", result buff,&context);
> if search result buff contains a "*" then
> return SS$_NORMAL;
> else
> rename the search result buff to "*.log"
>}
> call lib$find_file_end(&context);

It looks to me like LIB$FIND_FILE_END is unreachable. The only way out of your loop is the conditional return (which skips the LIB$FIND_FILE_END). If LIB$FIND_FILE fails for any reason (like running out of virtual memory) you're in an infinite loop.

Make sure you add a test of the return status of LIB$FIND_FILE and ensure that LIB$FIND_FILE_END is ALWAYS called when you've finished searching.

A very simple reformulation might be:

while (lib$find_file ( "*.csv", result buff,&context)){
if search result buff contains a "*" then
lib$find_file_end(&context);
return SS$_NORMAL;
else
rename the search result buff to "*.log"
}
call lib$find_file_end(&context);

I'm assuming this is C - one of the BIG issues I have with C and its brethren is the lack of discipline in control structures.

Too often programmers don't think about what control structures are really doing and cobble together something with "while(1)" and "break", or some monstrosity using the swiss army knife "for loop" abomination. True infinite loops are extremely rare in nature, so seeing one coded should ring alarm bells.

Go back to Dijkstra and learn about loop preconditions, postconditions, predicates, invariants and convergence. Make sure your loops have structural integrity and match the corresponding data structure being processed.

Re: Infinite loop problem

Kai.Zou — Fri, 07 Aug 2009 01:18:15 GMT

Dear John,
Thanks for your answer very much.
1. Now, I think, we had agreed with each other about the root cause of the problem, really?
2. So the problem should be able to be reproduct easily, shouldn't it? But when I tried to reproduct it with a test program, it always fell down, but not infinite loop.
3. The problem program was not written by me. I am just an analyzer, or maintenancer.
4. My customer will not agree to modify the issue unless clear evidence of the root cause had been found,now it became a delimmer for me.

So it must be sth. wrong during the analysis or this problem is really caused by a random behaviour?

Any way, thanks again for your sincerely help.
Thanks so much.