Higher memory utilization after OS upgrade

[roose]

Hi folks,

I would like to ask for some advise regarding our systems. We have a 2-node ES80 cluster currently running on OpenVMS v7.3-2. Last January 24, we upgraded the memory on the nodes on this cluster from 4GB to 8GB, and upgraded the OS from v7.3-1 to v7.3-2. Before the memory and OS upgrade, we did an autogen with feedback on the nodes. And after the upgrades, we did the autogen as well. However, now, it seems that our memory utilization have gone up, even with supposedly same loading before the memory upgrade. I have been investigating this using Performance Advisor and it seems that the majority of the memory allocation is being used by "IO cache". Can someone help me understand how we can reduce this IO cache allocation? Which SYSGEN parameter should we take to achieve this? Else, what benefits do we get if we just maintain this IO cache allocation?

We have also observed some increase on the CPU utilization, but I still need to investigate further on this as well.

I have attached a PSPA output of our node's CPU and memory utilization prior and after the memory and OS upgrade.

We are having a scheduled downtime sometime this April and we would like to do the fine-tuning then.

Thanks in advance to those who will help!

[Hoff]

Free memory is a resource that can be used.

I/O from cache is fast.

I/O from disk -- even an EVA -- is slower.

XFC is using available free physical memory as a (larger) I/O cache, as part of an on-going effort toward reducing the numbers of disk I/Os required and toward speeding application performance.

As for changes in local application requirements, XFC is coded to transparently release this physical memory back, if and when your local applications require added physical memory.

XFC is using idle memory, and it is polite about its use. XFC is borrowing this free and idle physical memory to speed your I/O.

If the XFC cache hit rates don't support it, you can certainly configure XFC not to use the free memory. Me? I'd leave it, and I'd also look at enabling and utilizing RMS Global Buffers, and other related I/O optimizations. I might well choose to toss more memory in, and allow XFC to grow its caches. (I have not looked at your performance data, however.)

Stephen Hoffman
HoffmanLabs

[Hoff]

ps: more CPU is expected when shoveling data through cache. This is a trade-off, and using extra CPU and memory moves is usually preferable to disk I/O, in terms of aggregate application performance.

Load up the T4 probes and establish a performance baseline. From that, you can make decisions around tuning.

And remember that a CPU running near capacity in user mode, or a system with little free memory might not be a bad thing. Tuning isn't as simple as it once was. (If it was ever truely simple.)

The real question is around application performance, and around the current performance bottleneck. And T4 can help here.

[Jan van den Ende]

Roose,

you should NOT want to reduce memory utilisation by the cache!

Roughly speaking, any data that has been brought in from disk also goes into that cache, (FIFO), and if another request for it is made before it is flushed... HEY, it IS already in memory! Map or copy it(whichever is appropriate), and we skipped the processing of and waiting for a disk transfer.
Effectively, you have bought IO performance which you paid for with memory.

hth

Proost.

Have one on me.

jpe

[Volker Halle]

Roose,

PSPA nicely graphs the IO cache in a different colour. If you look at all other usage of memory - aside from the XFC cache - and factor in the doubling of memory, I don't think your memory utilization has increased much.

Utilizing much of the available memory is not a bad thing. XFC will release it's IO cache memory, if it's required for process workingsets. Over-utilizing memory only hurts, if it leads to increased paging and/or swapping. Do you see any significant paging IO rates ?

Volker.

[John Gillings]

Roose,

You paid good money for that memory, so I'd hope that OpenVMS will use it and give you value for your hard earned dollars. Simply put, XFC will utilise any memory not in use by applications to improve your I/O performance. It will return memory to applications on demand.

The only downside, is all your performance monitoring screens look scary with memory utilisation (hopefully!)in the high 90's. Please consider this a very good thing and trust that OpenVMS will maximise your ROI on memory.

You may be able to tune the cache so it leaves some memory free (I don't know off hand how, because I can't imagine why anyone would want to do it!), but the net result will almost certainly be worse performance overall.

[Martin Hughes]

Memory is cheap and plentiful these days. I'd suggest that in most modern configurations I/O is the main area in which performance gains and losses are made. You might want to use PSPA to compare your direct I/O and disk I/O queue-lengths before and after the change.

Also, consider that page faulting and swapping are more significant indicators of memory shortage than utilisation. Although these symptoms can also result from a poorly tuned system, no matter how much memory is available.

[Jon Pinkley]

Martin Hughes wrote:

>>>Memory is cheap and plentiful these days.

However, this is an ES80, and for that, even third party memory isn't cheap.

That said, it is much better to let the memory be used by the cache that to have 50% in the free pool.

Roose, did the users think the system was slower or faster after the upgrade? Do jobs complete faster or slower. Do you care if there is a higher CPU utilization or a lower free memory pool as long as the work is getting done? (hopefully faster than it was before the upgrade)

[roose]

Thanks to those who took their time to answer my questions.

With these inputs, I'll go ahead and dig deeper on analyzing the performance data of our systems specially on memory and CPU utilization. So far, I do am seeing a number of hard pagefaults taking place as well on our nodes, but fortunately, we have not received any complaints from the users yet concerning performance degradation. Unfortunately, we are unable to implement a process yet on how we can measure application response time and performance. Hopefully, once we are able to do this, I can further correlate the application metrics to our system performance much easier.

Just one last question though, aside from XFC, are there any other component that I must look at that is contributing to our IO cache?

[Volker Halle]

Roose,

the memory shown as 'IO Cache' should all be used by XFC.

$ SHOW MEM/CACHE

Volker.

[roose]

Volker,

Thanks for your reply. Based on the command, it only has the XFC.

On that command as well, I noticed that the Read hit rate of our XFC is just around 74% on one node, and 86% on another node. Does this mean that the caching is not effective on our nodes? If not, what are the ways we can make it more effective?

[Volker Halle]

Roose,

on both nodes in your cluster, you are saving about 100 Read-IOs/second through the use of the XFC cache. What else would your memory be used for, if not for the XFC cache ?

You may also have noticed, that the IO load (read/write ratio) is quite different on those 2 systems. XFC is a generic method to cache disk data. Depending on the application, there may be even more specific methods to cache IO data (RMS global buffers etc.).

Volker.

[Wim Van den Wyngaert]

In contrast with what is being said here, I found in 2004 that XFC memory is NOT released when applications need it. Don't have the data anymore but perhaps it's better to test it.

WIm

[Martin Hughes]

You can use PSPA to find out which processes are generating the hard page faults. This may simply be the result of some processes with insufficient working set quota's.

Have you run AUTOGEN since the time you did it immediately after the upgrade? I.E. has it been run in the new configuration after a period of load?.

[Wim Van den Wyngaert]

Tested it on 7.3.

Workstation with 256 MB. VCC_MAX on -1 thus 128 MB is the max size.

Allocated 200 MB and made it dirty.

Again and again. At the end the pagefile was full but XFC still used 12 MB allocated(started the test with 77 MB).

Also read this http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=533755

BTW : I ran the malloc again forr 5 MB and the system went into hang.

Wim

[Jon Pinkley]

If you use PSPA to look for processes with high pagefaults, check the for number of image activations, as pages from the image file have to get faulted in. In the last version of PSPA I used (prior to being sold to CA), there was column IMGCNT that indicated number of image activations. Even with sufficient working set limits, image activations will cause page faults.

[Jon Pinkley]

Wim Van den Wyngaert describes problems with XFC in v7.3.

However, there were fixes since then. See release notes for 7.3-2.

Wim, have you reproduced your test in 7.3-2?

Jon

[Mark Hopkins_5]

I'm the XFC maintainer.

First, get the latest XFC remedial for V7.3-2.
I just finished testing V4 for V7.3-2 and it
should be available within the next week or
so. ( I'm not sure whether it is available without prior version support, if not V3 is
fine). There are lots of performance
improvements and bugfixes over the original
V7.3-2 release.

On most systems (particularly large memory
systems like these), XFC works fine out of
the box and you shouldn't have to worry about
tuning it.


In particular, the memory trimming code has
been vastly improved (actually rewritten).
A recent fix was to eliminate thrashing when
memory reclamation is happening.
We also do a better job of detecting
not enough memory at boot time (here I
mean 32MB or so). No matter what, XFC needs
memory to work and allocates about 4MB
permanently at boot time and even if
constrained may allocate more to prevent
hanging because of deadlock (I think that I
have all those fixed).


Low memory is not at all a problem on these
two systems.

In general, the hit rates on these two
systems look reasonable. It is interesting
that node S1A01 has over 80% writes. This
is very unusual - I'm a little curious about
the type of load here. The read cache
actually helps write performance since the
read I/Os aren't competing for bandwidth.

Low IO hits rates (e.g. < 30%) are not
necessarily a sign of poor cache performance.
We have seen some systems where the IO hit
rate was low, but the block hit rate was
high (30% for the former and over 70% for
the later). The reason being an application
doing 3 block reads with almost a zero hit
rate. The larger IOs were for the most part
being satisfied out of cache. The latest
versions of XFC are now tracking this data,
both in aggregate and in time series.

Since the cache on S1A01 has not grown to
full size, I'm guessing that this system
is more memory constrained and XFC is either
being trimmed or is not expanding.

Mark Hopkins