Pascal program crashing with -RMS-F-DME, dynamic memory exhausted

[labadie_1]

The article with the Dcl version

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c00352768

[Brian Reiter]

Hi Folks,

Thanks for all your help and advice. The engineering team have corrected the filehandling logic so the process no longer crashes. Luckily the NFS lgoical names proved to be a red-herring, that is a LD disk used to manage an exported NFS, which may its own strangeness to the mix.

The strange things is that I cannot see any signs of problems with the CTLPAGES etc. The only oddity is that at this time the disk queue length is quite high (somewhere between 5 and 10) at the point where the error during close occurs.

On a possibly related issue the DS10 has gone to the blue screen and is now busy displaying the following message:

$CLOSE: Channel xxxx is free and it should not be

Where xxxx is one of a pair of numbers.

We're slow adding instrumentation to the software, athough given its age (17 years, slightly younger than one of the other systems we maintain) there is very little will to expend too much effort.

Regards

Brian

[Hein van den Heuvel]

Hi,

I was wondering why Hoff brought up NFS, and finally noticed the attachment on the original topic.
The reported device is NFS$DISK, and the file name is lower case. Ah. More importantly perhaps, I now see the error is reported on CLOSE! That's kinda odd.
DME is most often on OPEN/CONNECT
DME is most often for process permanent files (Labadie's angle)

Brian,
- Is this indeed an NFS mounted file and do you have a feeling it might be related to that?
- Is the close 'special' like with a DISPOSE clause to submit or print?
- Could this be a process-permanent file (opened by DCL)?
- Does the process opens lots of files? (SHOW PROC /CHAN)
- Could this be an odd error report after a network timeout/disconnect?
- How often? When did it start? More often now?

Hein.

[Hein van den Heuvel]

Our replies crossed.
Thanks for the NFS clarification.

"$CLOSE: Channel xxxx is free and it should not be"

That does NOT look like a standard OpenVMS generated message.
I suspect it comes from the application or 3rd party tool.
Any detached processes or servers out there?
Any special helper tools?

How was the error handler changed?

"The engineering team have corrected the filehandling logic so the process no longer crashes. "

It should 'crash', or rather, it should not continue processing.
Typically a DME error is fatal.
The process is either out of memory, or the structures managing memory have been corrupted.
In either case trying to go on creates more trouble and hides the original error more.
Of course sometimes DME is 'overloaded' and/or exploited as a 'handy' code for application related events, but this _appears_ to come straight from RMS through the Pascal RTL.

Sometimes you want to either create a process dump for these errors to study at your leisure, or what I've done in the past is to call SYS$SETPRN "HELP ME" and SYS$HIBER()
Now you can poke at the process with ANALYZE /SYSTEM to try figure out its state.

Good luck!

[Brian Reiter]

Hi Hein,


From the first reply:

The file is not NFS mounted but is resident on a LD device (the LD device is exported to unix and VMS hosts). I'm curious to know whether or not it would cause any problems.

Nothing special on the close. Its a file which is opened.rewritten and closed every seconds.

Thre process has 60 or so open channels, the bulk of which are mailbox devices. Those which aren't mailboxes are section files.

I don't think the network is an issue, the file resides on a logical disk. I'm going to try moving it to the physical disk and see if that cures the problem.

As for the last questions, not sure. Certainly several times in the past week. Its been hinted at as "one of those things" but of course I can't get a firm answer.


From the second reply:

The engineering team added the ERROR := CONTINUE to the call to the Pascal CLOSE function and moved the call to the CLOSE function so it was always done regardless of the result of the OPEN. This seems to have solved the crash and the process stops reporing problems with the open after 20 seconds or so.

I can't see anything in our code which contains anything resembling that error message. The system consists of OpenVMS, TCPIP and our applications, there are no other third part applications running.

Looks like I'm going to need some good luck tracking this down.


Regards


Brian

[abrsvc]

One trick that I have used successfully is to have the process suspecnd itself when the condition is detected. This will at least provide a snapshot of the process status in regards to the RMS stuff. You should be able to use SDA to examine the process status. You can check open files, PIOpage usage etc. Since a process dump is not possible, this might be the next best thing.

Dan

[Hein van den Heuvel]

One of those high-jacked DME errors is within RMS itself when is can not longer keep track of all the open 'internal fabs'.

Those are managed through the IFI word in the FAB, but and is further limited to 16K by overloaded upper bits (to indicated process permanent files).

"Its a file which is opened.rewritten and closed every seconds."

Hmmm, there potential there to 'run out' if something is going wrong or done wrong.
How does the application find the file? Simple open, or wild-card parsing with lib$find_file or some such?
Failure to close down the parsing context could lead to odd behavior on the open/close.

fwiw,
Hein

[Jur van der Burg]

>The file is not NFS mounted but is resident on a LD device (the LD device is exported to
>unix and VMS hosts). I'm curious to know whether or not it would cause any problems.

That may work without an issue. Out of curiosity: which LD version do you use? Make sure you use the latest one (V9.5) if you happen to use LD tracing as it fixes a nasty floatingpoint register corruption.

Jur (lddriver author).

[Jim_McKinney]

fwiw, since I see that you're using 7.3-2...

We have a detached job that runs perpetually executing a DCL command procedure to monitor our application and environment. It loops through a series of exercises every 10 or 15 minutes. After 10 days of operation, or so, this job would fail with DME. Lots of of executables, lots of DCL, lots of opens and closes - but none of those were the issue. It exhuasts CTLPAGES. In my case I traced it to repeated use of the $BRKTHRU system service (via DCL REQUEST verb) which constructs a context block there that was not being released upon completion. I have no idea if this issue was fixed in 7.3-2 or not - my failures were with 7.3 and at the time it was expedient to add a bit of code for the application to just stop and restart itself after a week's operation to avoid the issue and I've not revisited this since to determine if the behavior is still there.

If you're not calling $BRKTHRU then this is obviously not your issue.

[Brian Reiter]

Hein,

The file is rewritten every 5 seconds, and the full file path is used very time. There are no wild carded searches etc. The code now seems to be logically correct.

Jur,
The verions of LD driver is that which ships with OpenVMS 7-3-2.

Jim,
No calls to $BRKTHRU in this code.

Regards

Brian