Free list size on mon system (alpha)

[Kendall McCarthy_2]

I have a question regarding the mon system screen . I notice that the free list was down to around 14,000 on the free list graph. I was assured that this is not an issue. Is this the case? (running VMS 7.3-2 on alpha)

[Jim_McKinney]

Probably not - you bought the memory so you might as well use it, right?

Is XFC active - see your SHOW MEMORY output - and if so how much memory is it holding? It's adaptive and takes what isn't used up to a configurable threshold - if need be, it'll automatically return memory to the free list so that user processes can be satisfied.

Any outswapped processes?

[Ian Miller.]

14,000 compared to how many pages of memory. For a quick look at memory use then MONITOR CLUSTER gives a percent.

However, what is the real problem?

[Kendall McCarthy_2]

Vol is in full XFC mode .

How do I tell how much it is holding?

[Jim_McKinney]

Example:

$ sh mem/full
System Memory Resources on 1-APR-2008 13:25:41.91

Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (768.00MB) 98304 5893 81292 11119

Extended File Cache (Time of last reset: 9-DEC-2007 12:09:40.43)
Allocated (MBytes) 366.89 Maximum size (MBytes) 384.00


This system has 768 MB and the XFC is using nearly half of it (366 MB).

[Kendall McCarthy_2]

monitor cluster gives me 99% memory used on this node. There are no issues . I just saw the low free list and saw it as a potential issue.

[Kendall McCarthy_2]

$ sho mem/full System Memory Resources on APRIL 1, 2008 04:31 PMPhysical Memory Usage (pages): Total Free In Use Modified Main Memory (4.00GB) 524288 38297 372854 113137Extended File Cache (Time of last reset: 16-MAR-2008 00:52:09.78) Allocated (MBytes) 987.21 Maximum size (MBytes) 2048.00


Although at this point free list is at 40,000

[Hein van den Heuvel · Accepted solution]

>> I notice that the free list was down to around 14,000 on the free list graph
>> I was assured that this is not an issue. Is this the case?

Sounds low, but it is probably just fine.
It could indicate that it is time to review load, growth, scaling.
14,000 down from 15,000? Out of 100,000 or out out 2,000,000 (16GB)

Memory on the free list is just wasted while sitting there. It is nice to have it as a reserve but it is better put to use in the XFC cache... knowning that will give back when neeeded.

So the "In use" or "Allocated" memory the XFC is a better indication for the memory pressure, assuming it is not artificailly limited by "maximum Size." As a bonus it is reported in Mbytes (or Gbytes as appropriate) so we don't have to think whether those 14,000

>> How do I tell how much it is holding?

$SHOW MEMORY /FULL

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Hein van den Heuvel]

Ah, time-collision on those replies.
I see some details now.

I would worry a little, but not a lot. I would try to plot XFC and Free memory over the past weeks and and interpret that in the larger picture of system usage and application updates. Did it step down when a new database was added, or did it creep down over time.
More users, or new functionality?
(Many) more users expected or a new plateau reached?
Those are the real questions, not the raw number out of context.

Cheers,
Hein.

[Jim_McKinney]

Here's yours reformatted ( check that "retain format(spacing)" box below your reply when you compose ):

$ sho mem/full
System Memory Resources on APRIL 1, 2008 04:31 PM
Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (4.00GB) 524288 38297 372854 113137

Extended File Cache (Time of last reset: 9-DEC-2007 12:09:40.43)
Allocated (MBytes) 987.21 Maximum size (MBytes) 2048.00


It shows that you've got 987 MB currently in use. So, nearly 1/4 of your memory is being used by the Cache. So, you've got no issues vis-a-vis the free list. You might examine more of that SHOW MEMORY output to see the efficiency of your XFC cache.

[Jim_McKinney]

To a point should probably consider the delta between what the XFC is using and its minimum size (seen below) as part of the free list.

Extended File Cache (Time of last reset: 9-DEC-2007 12:09:40.43)
Allocated (MBytes) 366.89 Maximum size (MBytes) 384.00
Free (MBytes) 0.21 Minimum size (MBytes) 3.12

I say "to a point" because the XFC adds value - potentially lots of value depending upon your applications - so if you starve it then you lose that potential. You can see the value in the "hit rates" present in that same SHOW MEMORY display. Simplistically, if your hit rates are small then either your cache is too small or your application is bypassing it.

[Ian Miller.]

IIRC the free list acts as a secondary cache for pages removed from a process working set i.e. they can be returned to the process working set (soft fault) rather than read from disk (hard fault). This means pages on the free list are not wasted and having a decent number of them helps this to occur.

I may be misremembering this however and will have to dig out the Big Black Book and re-read.

[Ian Miller.]

The advice given about measuring the size of the free list, XFC cache, fault rates etc is the best idea.

Is the application performance good enough and has it changed recently?

[Mark Hopkins_5]

A way to look at the XFC dynamic memory use is
to use the following command in SDA -

SDA> XFC SHOW HISTORY

XFC keeps about 3 days worth of data including
the amount of dynamic memory allocated. This
is currently done in 10 minute intervals
(originally 20 minutes). With the latest XFC
images, there is an additional command -
'XFC SHOW HISTORY/DEVELOPER' which shows
information more interesting to the maintainers,
but which does show the amount of trimming and
expansions for each interval.

In general, if XFC has grown to the maximum
size (by default 50% of physical memory) and
has not been trimmed, then the system is
probably not memory stressed.

Mark Hopkins

[vms systems management]

Is this showing me that extended file cache is exausted?


$ sho mem
System Memory Resources on APRIL 17, 2008 10:54 AM

Physical Memory Usage (pages): Total Free In Use Modified
Main Memory (4.00GB) 524288 13543 391831 118914

Extended File Cache (Time of last reset: 9-APR-2008 11:27:53.81)
Allocated (GBytes) 1.04 Maximum size (GBytes) 2.00
Free (GBytes) 0.00 Minimum size (GBytes) 0.00
In use (GBytes) 1.03 Percentage Read I/Os 93%
Read hit rate 69% Write hit rate 0%
Read I/O count 362373181 Write I/O count 26111572
Read hit count 250410167 Write hit count 0
Reads bypassing cache 42 Writes bypassing cache 2573913
Files cached open 2012 Files cached closed 528
Vols in Full XFC mode 0 Vols in VIOC Compatible mode 8
Vols in No Caching mode 0 Vols in Perm. No Caching mode 0

[Hein van den Heuvel]

"vms systems management",
Are you that same person as "Kendall McCarthy" or working on the same sytem?

(Both have 4GB memory, but that's not enough to tell really :-)

>> Is this showing me that extended file cache is exausted?

Are you refering to the "Free (GBytes) 0.00 "? Don't worry about that!
It's the "In use (GBytes) 1.03 " that counts.

What would worry me some is "Read hit rate 69%". That's below typical.
It could mean that the application usage patterns to not allow for good caching (lots, of data, little repeat access).
Or it could mean that the XFC could use some more memory.

Is this a cluster with actively write shared files?
You may want to drill down to the per-file cache usage and effectiveness.
Come to my Boot Camp session in Nashua NH in May (Nice!), or to the HP Tech Forum in Vegas in June (Hot!) and I'll tell you all about it!
http://h71000.www7.hp.com/symposium/may_2008/abstracts_all.html ... search XFC

I would review the Modified Page settings.
There is almost 1GB right there, if I read my numbers right. Isn't that 10x too much?
Is that a deliberate choice? Please explain!

Hope this helps some.
Contact me ofline if you think you'd like more help from me.

Best regards,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[vms systems management]

Here is what I am seeing. The free list slowly gets lower and lower daily until we restart the application. Then the free list goes back up. I am thinking that there is something in the application that has a memory leak . Is there a way to identiy a process that is not relinquishing its used memory?

[Jim_McKinney]

With an XFC active, I would expect that over time the "In use" bytes of the the XFC would grow as they are taken off the free list. This behavior would continue until either the XFC's "Maximum size" is reached or the free list is depleted to the point where the XFC starts replenishing it. Does this match your observation?

[vms systems management]

Yes it is . What happens when the Max size in extended file cahch is reached?

[Jim_McKinney]

Presuming that VMS doesn't have other memory demands that will cause the XFC to return memory that it has acquired, once it reaches its maximum size it will rotate out never (or infrequently) accessed files to make room for new ones that are "in demand". It (the XFC) will remain "full" (though the content may change) as long as there is no reason for it to surrender the memory back to VMS. This is normal, expected, and desired behavior.

[John Gillings]

>The free list slowly gets lower and lower
>daily until we restart the application.

This is NOT a problem. Memory is there to be used. OpenVMS will use it. That is good!

>I am thinking that there is something in
>the application that has a memory leak .
>Is there a way to identiy a process that
>is not relinquishing its used memory?

Don't look at the free list. It doesn't tell you anything about memory leaks.

Instead look at your virtual memory consumption, and page file useage. For an individual process watch PAGFILCNT and/or VIRTPEAK (F$GETJPI items). Graph the values over time. A "normal" process will grow over time, but will tend to plateau or asymptote towards a maximum. If it grows without limit you may have a leak. Use the heap analyzer to work out where. (see the DEBUG manual for details).

Note that minor leaks really only matter for long lived applications. Although that's a fairly sloppy attitude from an engineering perspective, the reality is it's not economically sound to go chasing every tiny the drip in an application that only runs a few hours or days and never gets near its virtual memory limits.

Large leaks will manifest as processes hitting their PGFLQUOTA or some other memory limit. DEBUG or HEAP ANALYZER are the weapons of choice to determine how and why.