83A memory leak following recent patches?

[Peter Barkas]

A few days ago, I installed FIBRE V7, RMS V10, SHADOWING V5, SYS V16.

Since then Total committed paging file usage seems to be increasing.

I see that the RMS kit includes changes for global buffers, which we use, and I wonder if anyone else is experiencing a problem?

[Hein van den Heuvel]

Thanks for the heads up.

What makes you suspect RMS Global buffers?

There is a fix for those as part of the RMS patch kit, but I happened to have worked with the customer experiencing this problem and it has nothing to do with space allocation best I understood it.

When you mention 'increasing' can you be a little more specific?
Compared to what?
How much?
Pages, thousands of pages, millions?
And if it is lots, seemingly in large increments?
(like a global buffer worth ;-)
How Fast? every second? hour? day?
Per image activation? Per file open? Per...?

Perhaps use T4 to fnd a correlation?

If you can nail it down to a process (group) then maybe ANAL/SYS, SET PROC ..., SHOW PROC/RDE
Now use SHOW PROC/RMS=GBH.
You should see "GBH Address:" P2-address values corresponding with each (re)used RDE

Of course there may be other usages in the program.


Regards,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting.


Hein
sho wpr

[Peter Barkas]

Thanks Hein, I'll see if I can get specifics.

I have had to reboot once since installing the patches because page file was exhausted and had no problems previously.

I had seen a bunch of BASIC "Cannot open file" errors a while before that, but didn't get to the underlying system error.

[The Brit]

Peter,
Would probably also be a good idea to create/install a couple of large secondary page files, on some free storage. This would give you some breathing room, (if this is happening quickly or unpredictably)

Dave.

[Peter Barkas]

Yes, thank you.

I doubled the page file size when the reboot became necessary.

[Jan van den Ende]

Peter,

in addition to Dave's advise and doubling the pagefile, please keep in mind that IF you notice pagefile usage growing, you CAN (on the fly) create and install an extra pagefile (as much as the number of SYSGEN specified pagefiles allows).
New pagefiles get used by new processes, so existing processes do not benefit ( Gurus, is this still true in V8.x ? )
Bear in mind, that you need to do this timely!
50 % pagefile use indicates the need to do it NOW, and NOT wait some more minutes (!).

fwiw.

Proost.

Have one on me.

jpe

[Peter Barkas]

Jan, thank you.

I am standing by with my quiver of pagefiles.

[Peter Barkas]

Currently we have installed these patches only on a development system.

Only this system exhibits the problem.

If global buffers are turned off there is no problem.

It appears that the problem might be triggered when an RMS query is requested by an APACHE client and the client browser window is closed before the query completes.

[Volker Halle]

Peter,

do you have processes hanging around without disappearing ?

Does INSTALL LIST/GLOBAL or ANAL/SYS and SDA> SHOW GSD show something unusual ?

Volker.

[Peter Barkas]

Volker, thank you.

I had to reboot because SWAPPER was getting jolly busy, I'll try your suggestions when the problem presents itself again.

[Hoff]

Please force a crashdump when you next have to reboot.

Don't halt and reboot.

I typically encourage my clients to provide the full crashdump command sequence next to the console; this whether on a sticky-note on the console or as a page in the operational manual or otherwise.

[Hein van den Heuvel]

>> If global buffers are turned off there is no problem.

Ok, I believe you, but that's odd.
( And potentially very serious to OpenVMS )

How many files are we talking about, how many global buffers and and what bucket size.

You'd think that once you mapped the first user, the rest just joins in and does not cause further memory usage.

Are you creating lots of new files with lots of global buffers?

Are you using the New V8.3 global buffer FDL / SET FILE options to create more than 32K global buffers per file or to use a system wide default?

At any rate, the right first tool to investigate is: INSTALL LIST/GLOBAL

$ PIPE INSTALL LIST / GLOB | SEARCH SYS$PIPE RMS$GBS

Start counting and adding up! :-)

( I have DCL code and tools to backtranslate the DEV + FID to filename + GBC + BKZ )

Next tool might be
$ ANAL/SYSTEM ...
SDA> SET PROC
SDA> SHOW PROC/RMS=(FAB,GBH)

And you might use
SDA> SET PROC
SDA> SHOW PROC /CHAN

Good luck,
Hein.

[Peter Barkas]

Hoff, good idea, I'll do that.

It's a balance between diagnosing the problem and getting some real work done.

[Hein van den Heuvel]

Attached a script to list all RMS global sections in memory, by parsing INSTALL LIST/GLO output.
The script reaches out to the files to report the 'normal' GBC setting.
No extra provisions (yet) for the V8.3 features.

This attached file may or might not be the most recent version I have.
I should check, but no time now.
Send Email with 'thanks' or questions as appropriate.

Cheers,
Hein

[Peter Barkas]

Hein

Thank you.

Buffers are set using set file/glo=def.

No new files are created with global buffers.

There are over 100 files with bucket sizes ranging from 2 to 63 but typically only 5 or 10 (and usually the same 5 or 10) are opened by an APACHE client.

On the system in question it is probably not uncommon for there to be no clients active at any one time because the client is instantiated on each page submission and is active only for as long as it takes to respond to the request.

I'll do that analysis next time around.

I managed to trace one layer of the underlying system error following the BASIC "Cannot open file" errors in the web application.

This was, surprisingly, "%RMS-F-CRMP, CRMPSC system service failed to map global buffers" even though GBLPAGFIL, GBLPAGES and GBLSECTIONS seem more than adequate and that error was not apparent when running more weighty "VT" clients.

[Hein van den Heuvel]

Hmm, this is like pulling teeth!

How can you answer " set file/glo=def" without specifying the relevant SYSGEN params?
My psychic powers fail to divine those.

As per help :
" o DEFAULT-Requests RMS at runtime to recalculate the global cache size based on an algorithm that makes use of two global buffer SYSGEN parameters, GB_CACHEALLMAX and GB_DEFPERCENT."

As a somewhat random workaround, try SET RMS/GLO=2000

But really... you need to figure out what reasonable settings are in the context of what you want to accomplish.

Please consider using an outside resource if the skills needed to do so are (not yet) available in house.

The RMS V10 patch description mention a Global buffer related patch, but best I understood not in that particular area.

Was the V10 patch applied after having run with V9, or were some patch levels skipped?

>> This was, surprisingly, "%RMS-F-CRMP, CRMPSC system service failed to map global buffers"

Actually, that's no surprise. It is just about the only error you ever get when there is trouble with global buffer. Now I don't recall whether that is the STS or STV value. Try to have the application print out the STV value if you can.

And please be sure to provide feedback to HP about this issues. At the minimum they made a change in behaviour which requires documentation, or they actually broke it.

By working the numbers from our system (SYSGEN params, FILE SIZES, BUCKET SIZES) you should be able to calculate what's right and what's wrong.

Regards,
Hein van den Heuvel at gmail
HvdH Performance Consulting.

[Peter Barkas]

The problem I saw was "Total committed paging file usage" becoming larger and larger until the system became unusable and this happened only when global buffers were in use.

The figure seemed not to reduce even after all processes that open files with global buffers had terminated.

It is not clear to me why this should be the case but I don't have time to work on this just now, so I will discontinue use of global buffers on this system until there is more time.

[Peter Barkas]

Later

[Hein van den Heuvel]

For the benefit of other readers....

$ mcr sysgen show gb_
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
GB_CACHEALLMAX 50000 50000 100 -1 Blocks D
GB_DEFPERCENT 35 35 0 1000 Percent D

So by default the per-file global buffer default setting is limited to 25.

For a bucket size in the 5 - 10 block range that equates to 5,000 - 10,000 global buffers.... per file, if the file is more than 75 MB.

It also equates to about 1 GB of pagefile reservation when 40 files are targetted.

Be sure to do the match when setting global buffers on a 'bunch' of files!

Cheers,
Hein