XFC cache min size

[John Eerenberg]

A few questions regarding the stability and setting of the XFC cache.

1) Does the VMS documentation state there is a minimum size for the XFC Cache (i.e., vcc_max_cache)? sysgen show legal values from 0 to -1 in MBs).

2) HP analyzed a crash dump and said the system hung from vcc_max_cache being set too low (i.e., 16MB). They recommend setting it to 100MB. My questions is, why not 50MB? why not 10MB? What is magical about 100MB as a minimum size?

3) Isn't this really a work around for a bug in VMS's XFC code?

john

[Mark Hopkins_5 · Accepted solution]

1) Does the VMS documentation state there is a minimum size for the XFC Cache (i.e., vcc_max_cache)? sysgen show legal values from 0 to -1 in MBs).

No. Even if set to 0, XFC will allocate
a minimum amount at boot time to allow
booting and some work. I'll see about
putting some warnings in the next versions
of the system manager's manual.

2) HP analyzed a crash dump and said the system hung from vcc_max_cache being set too low (i.e., 16MB). They recommend setting it to 100MB. My questions is, why not 50MB? why not 10MB? What is magical about 100MB as a minimum size?

The system involved had 3GB of memory (if we're talking about the same problem). 100MB
seemed like a value that was sure not to
cause problems. 50MB or 25MB might be
enough, but without knowing more about the
number of files which need to be opened on
the system, 100MB is pretty much sure to
be safe and give some performance benefit
as well.

) Isn't this really a work around for a bug in VMS's XFC code?

Yes. XFC could do a better job of continuing when constrained in this way. For
one thing, it would really have helped if XFC sent opcom messages when it realized it
was wedged up and unlikely to get free. In
addition, it might also be nice to output
a message at boottime if memory looks tight
(the hard part is deciding what this might
mean).

Currently, XFC divides the memory
it does have into memory for data structures
(meta-data) and buffers. In this case, we
ran out of memory for a structure for a
file open. It recovered as much as possible,
but there wasn't any more to recover. The
problem is that the current design doesn't
allow moving memory from the buffer space
to the data structure space. Fixing that
will require a fair amount of surgery on
the memory management code. I have some
ideas to help which should get into the
next release and set of remedials.

XFC is really designed to work better with
larger amounts of memory. On this system,
the cache hit rate was less than 2%. In
addition, XFC was doing a lot of work to
scrounge up more memory which added more
overhead. My guess is that the system
would probably have had better performance
with caching turned off altogether (I'm
curious how VIOC behaves on this system
with 16MB of memory).

Mark

[John Eerenberg]

> "The system involved had 3GB of memory (if we're talking about the same problem)."

Sounds like we are talking about the same system.
They had a potential memory problem. The app vendor reported to me that they seemed to have run out of page/swap file space which meant, in theory, on 3GB of RAM, about 4-5GB was swapped out. Setting vcc_max_cache to 16MB meant I could norrow down the page/swap problems.

> "100MB seemed like a value that was sure not to cause problems. 50MB or 25MB might be enough, but without knowing more about the number of files which need to be opened on the system, 100MB is pretty much sure to be safe and give some performance benefit as well. "

My goal was simply to restore response time to something they could live with and if more memory was needed, recommend how much. I couldn't do any indepth performance analysis, but I am guessing the IO to the database file was not getting a whole lot out of the cache at 240MB (the way I found it) and this afternoon's 500MB size. Curious because the database should be doing better. Don't know.
I wasn't too concerned with cache hit rates since I was just trying to get to the root of the problem; knowning the potential CPU overhead involved helps too.

> "Currently, XFC divides the memory it does have into memory for data structures (meta-data) and buffers."
Thanks for this info. It will help me in the future. I might even consider setting nocache on certain volumes at other sites (just a thought).

> "XFC is really designed to work better with larger amounts of memory. On this system, the cache hit rate was less than 2%. In addition, XFC was doing a lot of work to scrounge up more memory which added more overhead. My guess is that the system would probably have had better performance with caching turned off altogether."
With the cache at 240MB, the hit rate was around 37% (if memory serves).
In retrospect, I wish I had simply turned it off. But at 4AM . . .

> "(I'm curious how VIOC behaves on this system with 16MB of memory)."
I had a 128MB alphastation with 16MB in the VIOC. Worked fine . . .

Maybe you and I will be on the same concall later today?

Thanks for all your information.

john

[Mark Hopkins_5]

Many databases turn off caching directly on
each I/O (Oracle does this) and XFC won't
help the database performance directly. Even
here, XFC can help by reducing overhead for I/O that is cached. Predicting the performance impact of various combinations of cache memory is very tricky. The results
are often counter-intuitive.

The cache effectiveness for a file can be
seen using either the XFC SDA extensions
(e.g. XFC SHOW FILE/BRIEF) or from DCL -
$ SHOW MEMORY/FILE=$7$DKA100:[000000]TEST.TXT
System Memory Resources on 31-MAR-2004 15:54:46.19

Extended File Cache File Statistics:

_$7$DKA100:[000000]TEST.TXT;1 (closed)
Caching is enabled, active caching mode is Write Through
Allocated pages 11307 Total QIOs 11306
Read hits 7122 Virtual reads 11306
Virtual writes 0 Hit rate 62 %
Read aheads 3092 Read throughs 11306
Write throughs 0 Read arounds 0
Write arounds 0

Total of 1 file for this volume

For this particular problem, setting the
volume nocache won't help. The reason is
that XFC still needs to track open files to
maintain cache coherancy in a cluster. It
is perfectly reasonable and common to have
a volume mounted no cache on one node and
/cache on another. An additional problem is
that currently mounting /nocache also turns
off all file system caches (XQP) which may
be important.

Turning off caching on a node (setting
vcc_cache to 0) will turn off caching
cluster-wide.

The question about VIOC was aimed at performance. VIOC uses different memory
management techniques and much, much smaller
data structures. The question is what kind
of hit rates would you get on this system.
The default of 16MB for VIOC (3200 pagelets)
is almost always too small for good performance on a big, active system.

I haven't been asked to join the concall, but
I'll be in all afternoon and am more than
happy to talk.

mark

[Guinaudeau]

i red in another thread that HP recommended you a min size of 128MB.

we have some test systems with release 7.3-2 with main memory of 512MB or 768MB and VCC_MAX_SIZE=-1 and VCC$MIN_CACHE_SIZE, where XFC shrinks to a smaller cache size than 128MB.

eg,
main memory = 768MB
VCC$MIN_CACHE_SIZE=44MB
XFC allocated = 66MB

It is not feasible to reserve 100MB or 128MB for XFC on that machine, and since this is a test system (production systems have rather large memory and 100MB XFC), the alternative would be to switch off the feature or mount/nocache.

The test system works properly most of the time, but we have from time to time unexplained crash where the system hung, usually after several stop/start of the application. We can halt the machine and crash dump. Actually this happened today.

I would like to know whether one can quickly identify the sort of XFC troubles described, for example using a SDA XFC subcommand. I noticed in the last dump that there was no more PECB freepkt in the XFC SHOW MEMORY statistics.

I can send some statistics of this crash dump if it helps.

louis

[Ian Miller.]

post the clue file for the crash as a start

[Guinaudeau]

jan,

i attached CLUE informations. because the crash was operator forced, i dont know whether it will be very helpfull in that particular case.

i attached SDA mem + XFC informations from the dump : since memory shortage has been sometimes troublesome on that machine, i checked for free pages, nonpaged pool, pagefile and that was all right at the time the machine no more worked properly.

the machine did not hang completely. one could no more log in the machine using either SET HOST or TELNET, and interactive outswapped sessions no more reacted. the running application continued to react properly and to answer to network commands.

louis