Higher memory utilization after OS upgrade

[roose]

Hi folks,

I would like to ask for some advise regarding our systems. We have a 2-node ES80 cluster currently running on OpenVMS v7.3-2. Last January 24, we upgraded the memory on the nodes on this cluster from 4GB to 8GB, and upgraded the OS from v7.3-1 to v7.3-2. Before the memory and OS upgrade, we did an autogen with feedback on the nodes. And after the upgrades, we did the autogen as well. However, now, it seems that our memory utilization have gone up, even with supposedly same loading before the memory upgrade. I have been investigating this using Performance Advisor and it seems that the majority of the memory allocation is being used by "IO cache". Can someone help me understand how we can reduce this IO cache allocation? Which SYSGEN parameter should we take to achieve this? Else, what benefits do we get if we just maintain this IO cache allocation?

We have also observed some increase on the CPU utilization, but I still need to investigate further on this as well.

I have attached a PSPA output of our node's CPU and memory utilization prior and after the memory and OS upgrade.

We are having a scheduled downtime sometime this April and we would like to do the fine-tuning then.

Thanks in advance to those who will help!

[Hoff]

Free memory is a resource that can be used.

I/O from cache is fast.

I/O from disk -- even an EVA -- is slower.

XFC is using available free physical memory as a (larger) I/O cache, as part of an on-going effort toward reducing the numbers of disk I/Os required and toward speeding application performance.

As for changes in local application requirements, XFC is coded to transparently release this physical memory back, if and when your local applications require added physical memory.

XFC is using idle memory, and it is polite about its use. XFC is borrowing this free and idle physical memory to speed your I/O.

If the XFC cache hit rates don't support it, you can certainly configure XFC not to use the free memory. Me? I'd leave it, and I'd also look at enabling and utilizing RMS Global Buffers, and other related I/O optimizations. I might well choose to toss more memory in, and allow XFC to grow its caches. (I have not looked at your performance data, however.)

Stephen Hoffman
HoffmanLabs

[Hoff]

ps: more CPU is expected when shoveling data through cache. This is a trade-off, and using extra CPU and memory moves is usually preferable to disk I/O, in terms of aggregate application performance.

Load up the T4 probes and establish a performance baseline. From that, you can make decisions around tuning.

And remember that a CPU running near capacity in user mode, or a system with little free memory might not be a bad thing. Tuning isn't as simple as it once was. (If it was ever truely simple.)

The real question is around application performance, and around the current performance bottleneck. And T4 can help here.

[Jan van den Ende]

Roose,

you should NOT want to reduce memory utilisation by the cache!

Roughly speaking, any data that has been brought in from disk also goes into that cache, (FIFO), and if another request for it is made before it is flushed... HEY, it IS already in memory! Map or copy it(whichever is appropriate), and we skipped the processing of and waiting for a disk transfer.
Effectively, you have bought IO performance which you paid for with memory.

hth

Proost.

Have one on me.

jpe

[Volker Halle]

Roose,

PSPA nicely graphs the IO cache in a different colour. If you look at all other usage of memory - aside from the XFC cache - and factor in the doubling of memory, I don't think your memory utilization has increased much.

Utilizing much of the available memory is not a bad thing. XFC will release it's IO cache memory, if it's required for process workingsets. Over-utilizing memory only hurts, if it leads to increased paging and/or swapping. Do you see any significant paging IO rates ?

Volker.

[John Gillings]

Roose,

You paid good money for that memory, so I'd hope that OpenVMS will use it and give you value for your hard earned dollars. Simply put, XFC will utilise any memory not in use by applications to improve your I/O performance. It will return memory to applications on demand.

The only downside, is all your performance monitoring screens look scary with memory utilisation (hopefully!)in the high 90's. Please consider this a very good thing and trust that OpenVMS will maximise your ROI on memory.

You may be able to tune the cache so it leaves some memory free (I don't know off hand how, because I can't imagine why anyone would want to do it!), but the net result will almost certainly be worse performance overall.

[Martin Hughes]

Memory is cheap and plentiful these days. I'd suggest that in most modern configurations I/O is the main area in which performance gains and losses are made. You might want to use PSPA to compare your direct I/O and disk I/O queue-lengths before and after the change.

Also, consider that page faulting and swapping are more significant indicators of memory shortage than utilisation. Although these symptoms can also result from a poorly tuned system, no matter how much memory is available.

[Jon Pinkley]

Martin Hughes wrote:

>>>Memory is cheap and plentiful these days.

However, this is an ES80, and for that, even third party memory isn't cheap.

That said, it is much better to let the memory be used by the cache that to have 50% in the free pool.

Roose, did the users think the system was slower or faster after the upgrade? Do jobs complete faster or slower. Do you care if there is a higher CPU utilization or a lower free memory pool as long as the work is getting done? (hopefully faster than it was before the upgrade)

[roose]

Thanks to those who took their time to answer my questions.

With these inputs, I'll go ahead and dig deeper on analyzing the performance data of our systems specially on memory and CPU utilization. So far, I do am seeing a number of hard pagefaults taking place as well on our nodes, but fortunately, we have not received any complaints from the users yet concerning performance degradation. Unfortunately, we are unable to implement a process yet on how we can measure application response time and performance. Hopefully, once we are able to do this, I can further correlate the application metrics to our system performance much easier.

Just one last question though, aside from XFC, are there any other component that I must look at that is contributing to our IO cache?

[Volker Halle]

Roose,

the memory shown as 'IO Cache' should all be used by XFC.

$ SHOW MEM/CACHE

Volker.