Pascal program crashing with -RMS-F-DME, dynamic memory exhausted

[Brian Reiter]

Hi folks,

I have an executable which sometimes terminates with the folowing error:

-RMS-F-DME, dynamic memory exhausted

Are there any particular quotas I should be looking at? The execurable was written in Pascal (6.0-107), running on OpenVMS 7-3-2 (sadly only up to Update 6). The problem is intermittent.

Regards

Brian

[abrsvc]

Use the $HELP/MESS DME command to get specifics, but a common cause of this is excessive open files. Check the code for repeated opens without corresponding closes.

For more specifics, post the stackdump if you have it and we can guide you on how to read that if you are not sure.

Dan

[Hein van den Heuvel]

Is it causing a problem more often now?
Anything change recently?
CONVERTs with bigger buckets --> more memory?
Time for a convert? Unlikely impact

>> Are there any particular quotas I should be looking at?

Good question.

DME refers to VIRTUAL memory shortage 'at the wrong time', that is while RMS needed it.

So you can to watch and/or increase PGFLQUOTA
You also may want to watch (F$GETJPI ?) FREP0VA and FREP1VA


Sysgen param IMGIOCNT can be use to move more RMS stuff in P1 space and decrease 'fragmentation' of that memory zone.

The better way to set that is per image using IOSEGMENT in the option file.
Far too few folks use that.
If you can not re-link readily, then you can just tweak the image header to fix this on th e fly, for example with the DCL below.

Now you should also look at factors influencing usage, specifically SET RMS.

Issue a SHOW RMS and see if there are oddly large, and/or changed numbers there?

Does this program open and close files with global buffers repeatedly?

Good luck,
Hein


$! IMGIOCNT.COM Hein van den Heuvel
$if p1.eqs."" then $inquire p1 "image file name"
$
$! libr /extr=$EIHDDEF/out=tt: sys$library:lib.mlb
$! EIHD$L_IMGIOCNT 60
$
$open/read/write file 'f$parse(p1,".exe")
$offset = 60 !Alpha. We'll assume exe matches current system.
$if f$getsyi("arch_name").eqs."VAX" then exit ! offset = 32
$read file header
$old = f$cvsi(offset*8,32,header)
$write sys$output "IMGIOCNT was ", old, "."
$if p2.nes.""
$then
$ header[offset*8,32] = 'p2
$ write/update/symbol file header
$ write sys$output "IMGIOCNT now ", p2,"."
$endif
$close file

[Robert Gezelter]

Brian,

I suggest that you consider enabling dumps on that image. Analyzing the resulting process dump file may be quite illuminating (see SET PROCESS/DUMP).

- Bob Gezelter, http://www.rlgsc.com

[H.Becker]

>>>Sysgen param IMGIOCNT can be use to move more RMS stuff in P1 space and decrease 'fragmentation' of that memory zone.

Which - as you probably know - affects all processes.

Relinking with an options file gives you the linker limit which on Alpha is very conservative. A relinked image only affects the P1 layout, when running this image.

>>>If you can not re-link readily, then you can just tweak the image header to fix this on th e fly, ...

Not really on the fly, it's a change on the disk.

Both, IOSEGMENT and IMGIOCNT are a number of pagelets, that is a virtual address space, in P1. P1 is limited to 1GB. Obviously that's too much but it is difficult to give a good upper limit for this. Although you have 32 bits for the IOSEGMENT in the image header, you should be very careful with the DCL script and numbers resulting in more than half of P1: there is much more than RMS buffers which needs to be in this memory region and the shown DCL script doesn't do any check, here.

[John Gillings]

re: Robert - enabling process dumps...

Excellent suggestion, but chances are whatever is causing the DME will probably block the creation of a dump file. Sometimes it even blocks issueing an error message.

Brian,

See SHOW PROCESS/MEMORY to see the process dynamic memory area. Unfortunately this doesn't have a /ID option, so you can only see it from the process itself. The SYSGEN parameters which control it are PIOPAGES and CTLPAGES. Not to be messed with lightly!

As well as everything else suggested, I'd also add checking for runaway logical name and/or symbol creation. I'd also check SHOW RMS for a non-zero value in disk multibuffer counts.

[Hoff]

Get NFS entirely out of the path. That stuff can lead to all sorts of wacky I/O errors. (I just got done debugging a seriously bad diagnostic being emitted by a rather more current NFS configuration, too. How bad? Did you know that an NFS mount "NOPRIV" diagnostic can mean "I don't have a compatible NFS server to talk to?" Neither did I.)

Make absolutely certain that your NFS is patched to current, as you're (also) operating in the same range with a very insidious floating point register corruption lurking within NFS. Que pueden causar errores oscuro; obscure bugs can follow.

Back within your own Pascal code, desk-check the open and write and close paths, as errors and failures to deal with errors can get tangled, and the results can be confusing at best. (The older and longer that code has been in service, the more likely it contains status check errors and IOSB and synchronization bugs, too.)

If you're not interested in making a career out of debugging this particular (or any other) code, then the next step can be to Instrument the code. VMS doesn't provide these routines, but there are source code examples around and you can add your own instrumentation. (In non-trivial code, it's often best to start with the diagnostics as the scaffolding and build the application code around that.)

[labadie_1]

John Gillings said

"See SHOW PROCESS/MEMORY to see the process dynamic memory area. Unfortunately this doesn't have a /ID option, so you can only see it from the process itself."

Well, if memory serves me, 2 articles in the HP VMS database were about doing it, one in DCL, 2) and an Sda extension called ctlpa, in order to do the same thing inside SDA.

I even know the author
:-)

[labadie_1]

See

OpenVMS - Example C: How to Monitor Another Process Using an SDA Extension

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c01597305

and for Itanium, replace NOTINPHYS with NOREAD

[labadie_1]

On Itanium

$ set noon
$ say :== write sys$output
$ if P1.eqs."" then exit
$! if f$getsyi("ARCH_NAME") .nes. "Alpha" then exit
$ if .not.f$privilege("CMKRNL")
$ then
$ say "the privilege CMKRNL is needed"
$ exit
$ endif
$ if f$extract(0,4,f$getsyi("version")).ges."V7.3"
$ then
$ goto follow
$ else
$ say "the SDA> repeat /until=condition is not yet available"
$ say "a workaround may be a high number of SDA> exam @."
$ say "until the message NOREAD (on Itanium) is received."
$ exit
$ endif
$follow:
$ cpn = 0
$ tot_n = %X0
$ tot_largest = %X0
$ pid = f$getjpi("","PID")
$ fntmp = "sys$scratch"+":"+f$getjpi("","pid")+".tmp"
$ fntmp_a = "sys$scratch:sda_" + pid + "_a.tmp"
$ fntmp_b = "sys$scratch:sda_" + pid + "_b.tmp"
$ fntmpall = "sys$scratch:sda_" + pid + "*.tmp"
$ open/write log 'fntmp
$ write log "$ set noon"
$ write log "$ def/user sys$error nla0:"
$ write log "$ def/user sys$output nla0:"
$ write log "$ ana/sys "
$ write log "set proc/id=''P1' "
$ write log "set out ''fntmp_a' "
$ write log "exam ctl$gq_allocreg"
$ write log "exam @."
$ write log "repeat/until=NOREAD"
$ close log
$ @'fntmp'
$ ! keep only the useful lines
$ sea 'fntmp_a' "00000000.7"/out='fntmp_b'
$ open/read log1 'fntmp_b
$ cp = 0
$loop:
$ read/end=end log1 line
$ cp = cp + 1
$ tot_'cp = "%x"+f$extract(20,8,line)
$ tot_n = tot_n + tot_'cp
$ if tot_'cp.gt.tot_largest then tot_largest = tot_'cp
$ goto loop
$ end:
$ close log1
$ tot_largest = tot_largest + 0
$ !delete/nolog/noconf 'fntmpall';*
$ say "sh proc/mem/id=''P1' -> ''tot_n' free ,and ''tot_largest' is the largest"
$ exit


and on Alpha

$ set noon
$ say :== write sys$output
$ if P1.eqs."" then exit
$! if f$getsyi("ARCH_NAME") .nes. "Alpha" then exit
$ if .not.f$privilege("CMKRNL")
$ then
$ say "the privilege CMKRNL is needed"
$ exit
$ endif
$ if f$extract(0,4,f$getsyi("version")).ges."V7.3"
$ then
$ goto follow
$ else
$ say "the SDA> repeat /until=condition is not yet available"
$ say "a workaround may be a high number of SDA> exam @."
$ say "until the message NOTINPHYS is received."
$ exit
$ endif
$follow:
$ cpn = 0
$ tot_n = %X0
$ tot_largest = %X0
$ pid = f$getjpi("","PID")
$ fntmp = "sys$scratch"+":"+f$getjpi("","pid")+".tmp"
$ fntmp_a = "sys$scratch:sda_" + pid + "_a.tmp"
$ fntmp_b = "sys$scratch:sda_" + pid + "_b.tmp"
$ fntmpall = "sys$scratch:sda_" + pid + "*.tmp"
$ open/write log 'fntmp
$ write log "$ set noon"
$ write log "$ def/user sys$error nla0:"
$ write log "$ def/user sys$output nla0:"
$ write log "$ ana/sys "
$ write log "set proc/id=''P1' "
$ write log "set out ''fntmp_a' "
$ write log "exam ctl$gq_allocreg"
$ write log "exam @."
$ write log "repeat/until=NOTINPHYS"
$ close log
$ @'fntmp'
$ ! keep only the useful lines
$ sea 'fntmp_a' "00000000.7"/out='fntmp_b'
$ open/read log1 'fntmp_b
$ cp = 0
$loop:
$ read/end=end log1 line
$ cp = cp + 1
$ tot_'cp = "%x"+f$extract(20,8,line)
$ tot_n = tot_n + tot_'cp
$ if tot_'cp.gt.tot_largest then tot_largest = tot_'cp
$ goto loop
$ end:
$ close log1
$ tot_largest = tot_largest + 0
$ !delete/nolog/noconf 'fntmpall';*
$ say "sh proc/mem/id=''P1' -> ''tot_n' free ,and ''tot_largest' is the largest"
$ exit

[labadie_1]

The article with the Dcl version

http://www11.itrc.hp.com/service/cki/docDisplay.do?docLocale=en&docId=emr_na-c00352768

[Brian Reiter]

Hi Folks,

Thanks for all your help and advice. The engineering team have corrected the filehandling logic so the process no longer crashes. Luckily the NFS lgoical names proved to be a red-herring, that is a LD disk used to manage an exported NFS, which may its own strangeness to the mix.

The strange things is that I cannot see any signs of problems with the CTLPAGES etc. The only oddity is that at this time the disk queue length is quite high (somewhere between 5 and 10) at the point where the error during close occurs.

On a possibly related issue the DS10 has gone to the blue screen and is now busy displaying the following message:

$CLOSE: Channel xxxx is free and it should not be

Where xxxx is one of a pair of numbers.

We're slow adding instrumentation to the software, athough given its age (17 years, slightly younger than one of the other systems we maintain) there is very little will to expend too much effort.

Regards

Brian

[Hein van den Heuvel]

Hi,

I was wondering why Hoff brought up NFS, and finally noticed the attachment on the original topic.
The reported device is NFS$DISK, and the file name is lower case. Ah. More importantly perhaps, I now see the error is reported on CLOSE! That's kinda odd.
DME is most often on OPEN/CONNECT
DME is most often for process permanent files (Labadie's angle)

Brian,
- Is this indeed an NFS mounted file and do you have a feeling it might be related to that?
- Is the close 'special' like with a DISPOSE clause to submit or print?
- Could this be a process-permanent file (opened by DCL)?
- Does the process opens lots of files? (SHOW PROC /CHAN)
- Could this be an odd error report after a network timeout/disconnect?
- How often? When did it start? More often now?

Hein.

[Hein van den Heuvel]

Our replies crossed.
Thanks for the NFS clarification.

"$CLOSE: Channel xxxx is free and it should not be"

That does NOT look like a standard OpenVMS generated message.
I suspect it comes from the application or 3rd party tool.
Any detached processes or servers out there?
Any special helper tools?

How was the error handler changed?

"The engineering team have corrected the filehandling logic so the process no longer crashes. "

It should 'crash', or rather, it should not continue processing.
Typically a DME error is fatal.
The process is either out of memory, or the structures managing memory have been corrupted.
In either case trying to go on creates more trouble and hides the original error more.
Of course sometimes DME is 'overloaded' and/or exploited as a 'handy' code for application related events, but this _appears_ to come straight from RMS through the Pascal RTL.

Sometimes you want to either create a process dump for these errors to study at your leisure, or what I've done in the past is to call SYS$SETPRN "HELP ME" and SYS$HIBER()
Now you can poke at the process with ANALYZE /SYSTEM to try figure out its state.

Good luck!

[Brian Reiter]

Hi Hein,


From the first reply:

The file is not NFS mounted but is resident on a LD device (the LD device is exported to unix and VMS hosts). I'm curious to know whether or not it would cause any problems.

Nothing special on the close. Its a file which is opened.rewritten and closed every seconds.

Thre process has 60 or so open channels, the bulk of which are mailbox devices. Those which aren't mailboxes are section files.

I don't think the network is an issue, the file resides on a logical disk. I'm going to try moving it to the physical disk and see if that cures the problem.

As for the last questions, not sure. Certainly several times in the past week. Its been hinted at as "one of those things" but of course I can't get a firm answer.


From the second reply:

The engineering team added the ERROR := CONTINUE to the call to the Pascal CLOSE function and moved the call to the CLOSE function so it was always done regardless of the result of the OPEN. This seems to have solved the crash and the process stops reporing problems with the open after 20 seconds or so.

I can't see anything in our code which contains anything resembling that error message. The system consists of OpenVMS, TCPIP and our applications, there are no other third part applications running.

Looks like I'm going to need some good luck tracking this down.


Regards


Brian

[abrsvc]

One trick that I have used successfully is to have the process suspecnd itself when the condition is detected. This will at least provide a snapshot of the process status in regards to the RMS stuff. You should be able to use SDA to examine the process status. You can check open files, PIOpage usage etc. Since a process dump is not possible, this might be the next best thing.

Dan

[Hein van den Heuvel]

One of those high-jacked DME errors is within RMS itself when is can not longer keep track of all the open 'internal fabs'.

Those are managed through the IFI word in the FAB, but and is further limited to 16K by overloaded upper bits (to indicated process permanent files).

"Its a file which is opened.rewritten and closed every seconds."

Hmmm, there potential there to 'run out' if something is going wrong or done wrong.
How does the application find the file? Simple open, or wild-card parsing with lib$find_file or some such?
Failure to close down the parsing context could lead to odd behavior on the open/close.

fwiw,
Hein

[Jur van der Burg]

>The file is not NFS mounted but is resident on a LD device (the LD device is exported to
>unix and VMS hosts). I'm curious to know whether or not it would cause any problems.

That may work without an issue. Out of curiosity: which LD version do you use? Make sure you use the latest one (V9.5) if you happen to use LD tracing as it fixes a nasty floatingpoint register corruption.

Jur (lddriver author).

[Jim_McKinney]

fwiw, since I see that you're using 7.3-2...

We have a detached job that runs perpetually executing a DCL command procedure to monitor our application and environment. It loops through a series of exercises every 10 or 15 minutes. After 10 days of operation, or so, this job would fail with DME. Lots of of executables, lots of DCL, lots of opens and closes - but none of those were the issue. It exhuasts CTLPAGES. In my case I traced it to repeated use of the $BRKTHRU system service (via DCL REQUEST verb) which constructs a context block there that was not being released upon completion. I have no idea if this issue was fixed in 7.3-2 or not - my failures were with 7.3 and at the time it was expedient to add a bit of code for the application to just stop and restart itself after a week's operation to avoid the issue and I've not revisited this since to determine if the behavior is still there.

If you're not calling $BRKTHRU then this is obviously not your issue.

[Brian Reiter]

Hein,

The file is rewritten every 5 seconds, and the full file path is used very time. There are no wild carded searches etc. The code now seems to be logically correct.

Jur,
The verions of LD driver is that which ships with OpenVMS 7-3-2.

Jim,
No calls to $BRKTHRU in this code.

Regards

Brian

[labadie_1]

Not that I am convinced that this is your problem, but while we are on various limits of a process, check peakchan

$ if P1.eqs."" then exit
$ awk := $ sys$common:[syshlp.examples.tcpip.snmp]gawk.exe
$ fillm = f$getjpi("''P1'","fillm")
$ filcnt = f$getjpi("''P1'","filcnt")
$ used = fillm - filcnt
$ say "Process ",p1," has a fillm ",fillm," uses ",used," and peak open channels
"
$ say " "
$ pipe (say "set proc/id=''P1'" ; say "eval @CTL$GL_CHINDX" ) | ana/sys | sea sy
s$pipe Decimal | awk/command="{ print $(NF-1) }" sys$pipe

Or get it on

http://dcl.openvms.org/stories.php?story=08/06/09/6459295

[Brian Reiter]

Hmm curious, changing the file name logical to point to the physical disk seems to have stopped the problem occurring.

I'll install the latest version of LD driver and put the logical back to normal and see if that cures the problem.

Otherwise I nay be looking at an intermittent issue with interactions beween our application and LD driver which only seem to occur when the loading is heavy.

Regards

Brian

[Brian Reiter]

Oh well, looks as though pointing the file at the lgoical disk brings the problem back.

It only seems to be an issue at high loads during our system startup and at some periods overnight. The code change results in a temporary glitch which goes after 10 seconds or so and has no other impact on the system (other than the timestamp file may be out by a few seconds).

cheers

Brian

[Jur van der Burg]

How exactly is LDdriver involved? What (exact) commands did you use to configure it?

Jur.

[Brian Reiter]

Hi Jur,

During startup the system will do an LD CREATE if the container file is missing and then always issue the LD CONNECT. The drive is always initialised as ODS5 with no highwater marking. The volume itself is quite small, normally 1 Gbyte.

No other LD commands are involved and the whole things has worked like a charm for the past 5 years or so.

Could excessive disk queue lengths cause the problem I'm seeing?


cheers

Brian