Pascal program crashing with -RMS-F-DME, dynamic memory exhausted

[Jur van der Burg]

Where is the container file located? On an nfs volume?

:Could excessive disk queue lengths cause the problem I'm seeing?

Can't image that that would be an issue.

Jur.

[Brian Reiter]

Hi Jur,

The container file resides on a physical disk, so I think I can safely rule out NFS.

What is interesting though is that when the error does occur, and using the modified vversion (i.e. ERROR:=CONTINUE on the CLOSE) the Pascal RTL returns error during close rather than the RMS-F-DME.

cheers

Brian

[Hein van den Heuvel]

>> The code change results in a temporary glitch which goes after 10 seconds

and

>> What is interesting though is that when the error does occur, and using the modified vversion (i.e. ERROR:=CONTINUE on the CLOSE) the Pascal RTL returns error during close rather than the RMS-F-DME.

I'm not sure I understand that, but maybe others do.

I guess the program now only sees the "PAS-F-ERRDURCLO, error during CLOSE" from a call to "status" and no longer get access to the underlying RMS error.

To get that the program may need to use a user_action routine, or PAS$FAB but suppose that after a close the fab is gone error or success (not sure)

http://h71000.www7.hp.com/doc/82final/6083/6083pro_020.html#error_recovery_param_sec

" If you specify ERROR := CONTINUE, the program continues to execute regardless of any error conditions encountered during execution of the procedure. If you specify this value, you should use the STATUS function to be certain that the I/O routine worked as expected. "

I'm leaning towards memory corruption and would review any Async / stall / timer operations based on hight activity rates changing behaviour.

Or just live with the current situation until it degrades further, but it would bother me to know that a program is doing uncontrolled stuff.

Cheers,
Hein.

[Hein van den Heuvel]

I thought I had something to help a little, but the timing is probably wrong.

It bothered me to get a DME error from a CLOSE.
So I checked the listings for PASCLOSE
There was a fix back in 1990.

>>> 1-009 - Zero RAB$L_STS and RAB$L_STV before closing file. CCC 25-Jul-1990.

So if a prior operation (OPEN?, READ?) failed with RMS$_DME but was continued from then in the old days the pascal signal routine could pick up the stale RAB error in favor of the close error (in the FAB).

I have to believe that a 1990 era fix made it into the RTL you are running on V7.3-2 which was well after 1990.
But still, I would consider using PAS$RAB to check RAB$L_STS for DME before the close, and report that.

It is much easier to imagine a DME error before CLOSE.
Is this TMS_timestamp.DAT special at all? Simple sequential? Indexed? Global Buffers?


fwiw,
Hein

[Brian Reiter]

Hi Hein

The timestamp file is a one line text file.


cheers

Brian

[Hoff]

Get the port or the replacement environment to the state where you can retire this application. You might try loading some patches here, but that path is not without risk.

If that does not suffice, then have somebody take a look at (more of) the application design and more of the whole environment around this DME bug (rather than this port hole that we're looking through), and see what can be done to provide a better way of implementing this.

There almost certainly will be more efficient schemes.

I'm guessing there's sequential processing here, and you're using these hack-files to pass a dollop of information (some sort of timestamp, apparently) along to some downstream processing.

That you are here dealing with design decisions and application configurations and this, well, hack-file code that was Good Enough some years back, but where you've scaled past what RMS and LD and the existing design can manage with this design.

Going faster might involve messages via mailboxes, might be passing the time stamp with the (hypothetical) batch job parameters, might be a database, might be a key-value store in a big relative file, might be shared memory, there are all sorts of approaches that might be feasible here.

RMS itself stinks at these dinky hack-file designs, too; there have been some substantial performance updates in various VMS releases to make this hack-file design go faster, but every design (an application, the RMS file system, VMS itself) eventually hits a wall (or three) as you try to scale it past its design target. Unix goes way-faster here; with these hack-file designs.

And the possibility of latent bugs within the application has not been eliminated here. The statement "Luckily the NFS lgoical names proved to be a red-herring, that is a LD disk used to manage an exported NFS, which may its own strangeness to the mix." implies that there are other decisions similar to this hack-file design; that statement certainly implies NFS is still around, and having NFS in a production design is inherently going to introduce transients. It's of of the elder versions of a modern distributed network; where any of the components can drop out from underneath you without warning.

Given what budgetary and corporate goals can be inferred here, I'd also suggest dusting off your own resume and updating your skills and your staff's skills in alignment with where your organization is now headed for computing, because this VMS environment is clearly a "run it into the ground" environment. Get this stuff ported. Or get the management to sign off on a port or a replacement, or on a plan to drain the current environment.

[Hoff]

> ...It's of of the elder versions ...

...NFS is one of the ancestral protocol versions of a modern distributed network; where any of the protocols and components can drop out from underneath you without warning...

[abrsvc]

In summary, perhaps it is time to take a complete overview of the application. There are a number of us here that can provide that service to you. It would seem that a close examination of the entire application to address the problem rather than attempting to "fix" the symptom is the better approach.

IF you can swing it, check the status of the close and set the process into SUSP state. Use SDA to show the open channels. Perhaps that simple step will show channels remaining open that should not be open. The DME suggests too many files open at one time (a common fault) given the available memory.

Just a thought...

Dan

[Brian Reiter]

Hmm, its a one line file written every 5 seconds so that in the case of some engineer on site powering off the system we can have a guess at how much downtime the system has had and record the fact accordingly.

We've seen this failure several times in the factory, once on a production quality machine and repeatedly on a DS10 with the minimum possible supporting hardware. On this DS10 the problem could be reproduced every time. Moving the file back off the logical disk onto the physical disk cures the problem.

The problem has never occurred at a customer site of which there several. I still have a couple of avenues to explore, but at the moment a 20 second window where the primary timestamp may be unknown is not ncessarily a problem, there are secondary sources used in case the file proves to be unreadable.

Given that the customer is very happy with these systems I suspect that porting to another platform will be met with some amount of laughter.

Regards

Brian

[Hoff]

This would take ten or fifteen minutes to rewrite a basic pinger, and probably a half day to unpack the hackery that's sprouted up around this.

I might well re-design the pinger (slightly) during the rewrite, and specifically move the pinger into the main loop of some of the critical processing.

There may well be an uptime tool or three in the Freeware or related resources, and there are SNMP and http-based uptime tools available.

And simply collecting T4 data continuously would also provide this information, too. T4 collection logs its performance data at one-minute intervals, after all. The time there would involve unpacking gaps in the time lines within the T4 data, and that's not complex.

Most of these approaches could be implemented in less time than has been collectively invested in this thread, too.

[Hoff]

Consider investigating why the boxes are being hard-powered, too. That can imply issues with the aggregate stability of the environment; that's a common brute-force server reset sequence.

[abrsvc]

A setup used for me requires a cluster license as well, but only 1 machine has the majority of licensing for products. I have a 2 node cluster over Fibre with the 2nd node only present to serve a set of disks for 3 member shadow sets. The primary system boots off of Root 0 (DSA0) and the second boots off of Root 1 (copy of DSA0 on a single drive). Since the second machine is a disaster recovery machine, it does not run any applications nor is it available to the users. If the main machine fails, human intervention is required because of instruments directly connected to the machine. Along with "moving" the instruments, the human at this point would halt the 2nd machine and boot it as machine 1. Once machine 1 is "repaired", it is booted as machine 2 off of root 1 (after a copy of DSA0 to a blank disk. This maintains the single machine boot off of the drive as well as keeping the system up most of the time.

Dan

[Hoff]

Wrong thread, Dan.

[abrsvc]

Don't quite know how that happened, but yes this should be in another thread. Sorry about that.

[Brian Reiter]

The power downs and restarts are often part of an attempt to get the whole system (several different server type boxes and potentially several 1000 devices connected via a variety of means).

Its not ideal but given what the system is (anyone who drives in the UK will likely have seen it action), its an understandable solution.