topic Re: XFC and memory management in Operating System - OpenVMS

XFC and memory management

Jim_McKinney — Tue, 13 May 2008 11:22:39 GMT

The immediate environment is Alpha OpenVMS v7.3-2 but if I’d also be interested to know if behavior changes in subsequent VMS versions. The system in question was configured to use XFC with VCC_MAX_CACHE set to -1. Over time the XFC had become fully populated, consuming 50% of physical memory (4GB out of 8GB). With the typical full complement of users this system would normally never page or swap as there was expected to be adequate physical memory to support large working set extents for the expected number of processes. However, on one recent occasion, a large number of additional processes were created. One side effect of this was swapping. I was surprised by this – rather, I expected that the XFC would begin relinquishing some of that 50% of physical memory that it was consuming before swapping occurred – it didn’t and the swap file space was inadequate (this has been remedied – a secondary swap file was accidentally lost during a storage upgrade when several old small disks were consolidated into one larger one). Is this expected behavior or should swapping have been postponed until the XFC had surrendered some of what it was holding? Thanks for reading this and comments that you might have – more info can be provided if required.

Re: XFC and memory management

Wim Van den Wyngaert — Tue, 13 May 2008 11:31:23 GMT

Of an old thread :

Tested it on 7.3.

Workstation with 256 MB. VCC_MAX on -1 thus 128 MB is the max size.

Allocated 200 MB and made it dirty.

Again and again. At the end the pagefile was full but XFC still used 12 MB allocated(started the test with 77 MB).

BTW : I ran the malloc again for 5 MB and the system went into hang.

Wim

Re: XFC and memory management

labadie_1 — Tue, 13 May 2008 11:40:02 GMT

Hello

I guess you know there is the sda extension xfc
$ ana/sys
xfc
help

and
xfc sh file/brief
xfc show memory(/verify)
xfc sh summary/stat

and similar commands regularly may help monitor this.

This may vary with the patches applied on your system.

Regards

Gerard

Re: XFC and memory management

Jim_McKinney — Tue, 13 May 2008 11:47:13 GMT

Perhaps I should phrase this another way - if there's a run on the (memory free list) bank, how is it expected that the bank will be funded? - by trimming of the XFC or by outswapping idle processes? Who yields first? I'm interested in the ruleset that is applied here.

Re: XFC and memory management

Wim Van den Wyngaert — Tue, 13 May 2008 11:48:29 GMT

To be complete : I used XFC 2.0 dd aug-2002. The last version is of JUL-2004.

Wim

Re: XFC and memory management

Jim_McKinney — Tue, 13 May 2008 11:56:32 GMT

$ pipe prod show history | sear sys$pipe xfc
DEC AXPVMS VMS732_XFC V4.0 Patch Install 10-SEP-2007 20:19:07

$ ana/sys

OpenVMS (TM) system analyzer

SDA> xfc show sum

XFC Summary (current time 13-MAY-2008 05:55:37.58)
--------------------------------------------------
Extended File Cache V1.0 (Mar 8 2007 11:18:39)

Re: XFC and memory management

labadie_1 — Tue, 13 May 2008 12:17:13 GMT

There is a patch XFC V4, have you applied it ?

The releases notes say, among other things

The most recent patch is : VMS732_XFC-V0400

5.2 Problems addressed in this kit

5.2.1 Limit the Maximum Number of Closed Files Retained in
Cache

5.2.1.1 Problem Description:

Excessive use of non-paged pool by XFC closed files could
lead to non-paged pool allocation failures and bugchecks.

The maximum number of closed files that XFC will retain
in the cache is now limited so that XFC will use no more
than one half of the original size of non-paged pool (the
SYSGEN parameter NPAGEDYN). This value will be
approximately 1500 files per megabyte of non-paged pool.
For example, if the SYSGEN parameter NPAGEDYN is
30,000,000 then then XFC will retain no more than about
45,000 closed files in the cache.

Re: XFC and memory management

Wim Van den Wyngaert — Tue, 13 May 2008 12:58:49 GMT

Did some reading and read that the reclaiming is somewhat slow. So may be it's a question of waiting ...

Also : in VMS 7.3 doc I read

$ RUN SYS$SYSTEM:SYSMAN
SYSMAN> RESERVED_MEMORY SHOW VCC$MIN_CACHE_SIZE /NOGLOBAL_SECTION

should return the minimum size of the VCC. But not on my 7.3 system.

Wim

Re: XFC and memory management

Wim Van den Wyngaert — Tue, 13 May 2008 13:08:05 GMT

Mmm you have to add the reserved memory yourself. I didn't.

Wim

Re: XFC and memory management

Jim_McKinney — Tue, 13 May 2008 13:10:36 GMT

With VMS 7.3-2 the SDA XFC extension will permit you to see the minimum cache size... a convenient system shows the following.

SDA> XFC SHOW MEM
[ . . . ]
Boottime Maximum Cache Size : 402653184 ( 384.0 MB)
Current Maximum Cache Size : 402653184 ( 384.0 MB)
Total Allocated Cache Memory : 396904832 ( 378.5 MB) Peak: 397094784 ( 378.7 MB)
Minimum cache size : 8052736 ( 7.7 MB)
[ . . . ]

Re: XFC and memory management

Wim Van den Wyngaert — Tue, 13 May 2008 13:18:27 GMT

And the min size is missing in 7.3.

Re: XFC and memory management

Wim Van den Wyngaert — Tue, 13 May 2008 13:44:40 GMT

Did the same test again. Started with 58 MB and went down to 9.3 MB. And hang. Killed a decw process, waited a few minutes and the system came out of hang. 9.31 MB allocated.

Fwiw

Wim

Re: XFC and memory management

Wim Van den Wyngaert — Tue, 13 May 2008 13:49:16 GMT

And as soon as I went out of phusical memory the XFC cache was decreased to 9.3 MB. So that's ok. Wonder why it wasn't decreased in your case.

BTW : did you mean paging or swapping ?

Wim

Re: XFC and memory management

Jim_McKinney — Tue, 13 May 2008 14:15:25 GMT

I meant swapping - but paging to the page file was occurring as well. Processes were being trimmed back to their WSQUOTAs but the XFC wouldn't open its fingers and let go any of the memory that it had - at least not quickly enough to save this system.