InterSystems Cache and XFC...

[Rich Hearn]

What are peoples experiences using XFC with InterSystems Cache on VMS? Has anyone tried (or is now doing) this?

Tnx,
Rich

[Jim Geier_1]

I am managing an InterSystems Cache environment with 350 GB of databases on a production cluster of 3 AlphaServer ES45 systems with about 1400 interactive users running GE/IDX applications. We also have a test cluster with 10 Cache configurations, each with about 350 GB of databases. I have asked this question before of GE/IDX and InterSystems but cannot find the official answer. I believe the answer was that they had not tested the XCF, that because they focussed on using the large shared memory section on OpenVMS, that using the XFC may not help. They did not say, No, don't do it, but they did not encourage it, either. I'm continuing to look for the official answer.

Having seen the XFC cause dramatic performance improvements on other, non-Cache systems, I'd surely like to use a feature of the operating system that might improve I/O performance at no additional cost.

[Jim_McKinney]

Cache does its own caching. Presumably that caching is very specialized and has been optimized for the behavior of Cache and can do a better job than the XFC for its IOs. Using the XFC on the CACHE.DAT files in addition to what Cache is already doing would be redundant. So, presuming a system whose primary function is to serve the Cache database, it would seem a better use of memory if more were devoted to Cache's cache. Simplistically speaking, if you have a gig of memory available for caching that you want to devote to your database, then assign it all to Cache's cache rather than, for example, split it evenly between Cache and the XFC. In practice however, you'll probably want to use the XFC for those non-Cache files that your system accesses (you can enable/disable XFC caching on a per-volume or per-file basis). Fortunately, the XFC is adaptive and will use what memory's available. In my DBMS experiences, best overall performance is observed when the DBMS cache is fattened - so, determine what amount you need for Cache and let the XFC do the best it can with the rest.

[Wim Van den Wyngaert]

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=923054&admit=-682735245+1148024888670+28353475

You can mark the directory or db files of the dsm (=cache) with /nocache.

Wim

[Ian Miller.]

See also
http://dba.openvms.org/phorum/read.php?17,23,23#msg-23

Basically you disable XFC for the Cache files.

[Rich Hearn]

Thank you all for the replies. I gather no one is actually doing it, since no one's given me any statements that they are. I do appreciate all of the answers. Thank you again, and have a great day.

Rich

[Jim Geier_1]

GE Healthcare (IDX) and InterSystems do not have an official position on the use of the XFC in environments where Cache is running on OpenVMS. On a system in which Cache is running, there are many other file operations that could benefit from the XFC, even when the dominant use of the system is the applications in the Cache configuration.

When configuring the XFC on such a system, be sure to allocate most of the memory to Cache, and limit the amount of memory available to the XFC. As Jim McKinney mentioned, application-specific caching is almost always more efficient compared to general-purpose caching. I would only enable the XFC on a system where there is plenty of memory, and a large amount of free memory during peak times.

Caching can be disabled on the CACHE.DAT files with the SET FILE /CACHING_ATTRIBUTE=NO_CACHING command. Or, if there are disks housing nothing but CACHE.DAT files, the disk may be mounted using MOUNT /SYSTEM /NOCACHE to disable caching for that disk.

[Rob Young_4 · Accepted solution]

Jim writes:

When configuring the XFC on such a system, be sure to allocate most of the memory to Cache, and limit the amount of memory available to the XFC. As Jim McKinney mentioned, application-specific caching is almost always more efficient compared to general-purpose caching. I would only enable the XFC on a system where there is plenty of memory, and a large amount of free memory during peak times.

Caching can be disabled on the CACHE.DAT files with the SET FILE /CACHING_ATTRIBUTE=NO_CACHING command. Or, if there are disks housing nothing but CACHE.DAT files, the disk may be mounted using MOUNT /SYSTEM /NOCACHE to disable caching for that disk.

---

I'd add something else. Very much depends
on environment. Let's say you have 4 Cache
instances per node and 4 GBytes of physical.
As per Intersystems RECs, you would size
your RESERVED_MEMORY to 2 MByte * # users
per instance.

So let's say that works out to 128 MBytes.
What would you do? I guess do what they ask,
now XFC being turned on is working with
2 GBytes and each instance 128 MBytes. Do
you turn off XFC on Cache.dat files? I'd
say no as when you go in SDA and view XFC
hits you see good numbers *in many cases*,
poor in other cases (regarding cache.dat).
So clearly XFC is helping - double caching
and all! Now where vendor caching wins out
is they don't have to flush their cache.
XFC will flush cache on those volumes as
write demon timer runs down every 80 seconds
so you will always see a ping-ponging of
cache pages for volumes that are pretty
write active.

One are you would see very good hits is
down through the installation (OPG]CACHE.DAT
, MGR]CACHE.DAT area). Those cache.dat
are read-only (my guess) for security, etc.

Now maybe you have a single instance with
1000 users and 4 GBytes of memory. With
that as a model, I would run with XFC
disabled on cache.dat files as per Intersystems, they would want you to size
your reserved memory at or near 2 GBytes.

[Wim Van den Wyngaert]

If I remember correctly from tests that I did 2 years ago :

If you give a lot of memory to XFC and you go out of virtual memory, the memory used by XFC is not reclaimed, or at least not as I would expect. Then you better have BI and AI journaling on your Cache stuff (we don't on DSM and may loose 1 second of data on each crash).

Wim

[Richard J Maher]

Hi Rich,

You might want to check if DSM uses io$m_novcache internally for its i/o.

FWIW, my vote is turn off XFC for database files especially as Cache seems smart enough to use the reserved memory registry. (Like Orrible Oracle). If only Rdb did :-( Still, they're trying to push people towards XFC. But then if all your eggs are in the one Row Cache basket I guess your options are limited.

Regards Richard Maher

[Tom O'Toole]

Rich,

You haven't clarified: is your main concern 1) to improve cache performance or 2) to improve performance of non cache file I/O on the same system (but you're concerned about data safety with XFC enabled)?

If 1), what you get from ^GLOSTAT when you cache system is busy (do a timed run of a minute or so)... As I recall Intersystems says 'global efficiency' over 100 is 'good'. It appears to be (global refs)/(block reads+block writes). As others have said, increasing the size of the global buffer memory is probably the best way to improve this.

Has anybody out there done any measurements of global efficiency as a function of the size of their cache global buffer pool? These parameters are unfortunately inconvenient to change on heavily used systems:-)

[comarow]

Someone commented that during their tests two years ago that XFC took too long in releasing memory. It is my understanding that the recent ECOs have focused on releasing memory more quickly. I'd love to see those tests replicated.

I *think* XFC is smart enough not the cache the Cache's cache.

[Rich Hearn]

Thank you to *all* of you who have taken the time to answer/respond. Been a busy past 2 weeks or so for me - haven't gotten back here to respond. I apologize.

Tom has a question for me so I'll address it. What I'm looking to do is see if I *have* to turn off XFC - currently, it's on. I have been concerned about the "double overhead" of caching what Cache` caches. The responses have been *very* informative for me and worthwhile, so I'm going to leave it turned on, but mark each cache` disk as /nocache during mounting (DB disks only). It seems that that would make the most sense. The GLOSTAT info is very inconvenient since a shutdown is tough to get. Almost as easy as "defraging a DB" by restoring it from tape :^)
Thanks again to everyone for your thoughts and responses.

Rich

[Mike Kier]

I just recently took responsibility for an IDX Cache system.

I put a Set File /Caching=No_Caching dev:[*...]cache.dat;* temporarily (one-shot)in the system startup before the IDX startup so that the existing Cache files would not be cached (and this attribute should be inherited by any copies). And then I enabled XFC with the default settings. We run with a 1GB Cache reserved memory area and the XFC is running about 1.5GB (on a 4GB DS20e).

XFC read hits are running about 20% and none of the Cache databases are cached. Seems to be performing well.

Mike
Group Health Associates, A TriHealth Company
Cincinnati, OH

[Volker Halle]

Rich,


but mark each cache` disk as /nocache during mounting (DB disks only).


Mounting a disk /NOCACHE will also disable all file system caches - this may not be what you want. There is no qualifier yet, which allows you to just disable the XFC data cache per mounted volume.

Consider to go with SET FILE/NOCACHE on the database files to explicitly tell XFC to not cache those files, as already suggested by others.

Volker.

[Rich Hearn]

Volker,

The only files on those disks would be Cache` files so, since I don't want them cached anyway, it would be ok to do. VMS would not be expected to cache them.

Rich

[Tom O'Toole]

Hi Rich,

What's your global efficiency with GLOSTAT now, and what's the current size of "globals" in the .CPF?

I just did a 300 second measurement here:

global efficiency - 126

With a 2.44Gb vms reserved section used for cache globals - approx. 700 users on this node (one of four in a clustered cache configuration).

Anyone else who is using perhaps larger sections care to post?

[Hein van den Heuvel]

Mike Kier wrote on Jun 6, 2006 16:09:06
>>----------------------------
>> I just recently took responsibility for an IDX Cache system.

Hi Mike,

Are you the OpenVMS Ambassadors Mike Kier from Ohio?
Sean me an Email some day.
Good to see you got a good gig!
Congrats!

Cheers,
Hein.

[Mike Kier]

Hi Hein,

Yup, one and the same. Great to hear from you - it was good to see your photo from the Bootcamp!

It took a lot longer than I'd hoped to find a position and its still got some uncertainty around it (another company undergoing acquisition and reductions - I feel right at home) but its great to be back working with VMS again.

I've only been on board about 6 weeks on a 3 month contract-to-hire, so it was too late to make it to the Bootcamp this time around, but maybe next year!

I'm at mike_kier@cgha.com.

Mike

[Rich Hearn]

Tom,

This is a 2 node cluster (v7.3-2, Cache 5.0.05, ES47's - 16GB mem, EVA5000 1.9 TB)
On the Master node, efficiency is 193%, on the "slave" it's 60%. I'm guessing that's probably ok since the "slave" doesn't do it's own writes & has to go through all the communication protocol processes (SW_1-7), so I'm expecting it to be less; no?

We normally have ~450 users on each node and the current size of GLOBALS is 1418 in the *.CPF

FWIW - Show mem/cache/full shows:
In use (GBytes) 1.44

Rich

[Tom O'Toole]

I think any system in a vms cache cluster can do its own writes... Yesterday I was seeing less global efficiency on the master here than some others (it seems to vary a lot).

[Cass Witkowski]

If you think about it. If you have XFC enabled on disks with the Cache' database on it then you are first reading the data into XFC and then into the Cache's own in memory cache. This means that the CPU is doing a lot of moving of memory around.

I don't know if XFC will cache files that are open for write on multiple nodes. So it may be a moot point anyway.

[Tom O'Toole]

One thing I've noticed in connection with this - the absolute counters roll over quickly on a large system - giving goofy results. I zeroed the counters on all nodes yesterday and some of the counters on the master have already rolled over. In particular, the global refs. appears to be an unsigned 32 bit counter.

Maybe we should think about saving/zeroing them every night around the time journals are rolled, or even more frequently.

[Rich Hearn]

Wow, thank you all for your thoughts, advice, & experiences.

I meant to get back to Mike K & Volker H earlier, but forgot to put it in my previous response.
I was thinking mount/nocache for the disk because it would be less overhead than individual files on the disk since there are only cache' files on the disk - no? I was thinking the directory caching would still be active - yes? Am I mis-understanding this?

Cass W - I didn't really want caching turned on; just "stumbled" on to finding it on when I was checking on a "system slow" problem (that has "disappeared" btw and not been seen since... gotta love it) I thought I had it off. I agree on the extra CPU activity.

Tom - I'm trying the ^GLOSTAT counter zero-ing now - I'll let you know how it goes. Seems like a good idea tho'.

Rich

[Volker Halle]

Rich,

when mounting a disk /NOCACHE, you are turning off all caching (including the XQP=File System caches). See $ HELP MOUNT/CACHE

Volker.