Operating System - OpenVMS
1748195 Members
4166 Online
108759 Solutions
New Discussion юеВ

Performance DECREASE after CPU/memory upgrade

 
Willem Grooters
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

Martin,

Got the tools: handy to have them around! I;ll have to some tuning on processes to it's good to have them.
Well, I ran perf_mon, and I found the CPU-intensive application uses 92-100% CPU when the system is 'idle'. FYI: it runs at priority 3, so it will only be active if nothing else requires the CPU. At the moment, the system is mostly 'idle' ;-). But still, this job takes more CPU time than anticipated.
Quota shows it's NOT running out of any so that can not be the case either.
Willem Grooters
OpenVMS Developer & System Manager
Martin P.J. Zinser
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

Hello Willem,

one thing that might need adaption after moving to a faster CPU is the Quantum Sysgen parameter.
Default is 20. Looking around on our systems it seems that 5 might not be such a bad value to move to.

Greetings, Martin
Todd Maurer
Advisor

Re: Performance DECREASE after CPU/memory upgrade


NOTE: Assuming that the issue here is a CPU intensive batch job running without much competition, then lower quantum is not a good recommendation. Lowering quantum favors better round robin servicing many processes where the number of processes in COM+CUR states typically exceeds the number of CPUs in the system.

Another suggestion is to compare the accounting information of the job before the upgrade with the accounting information from after the upgrade. See what has changed besides just the elapsed time. If the amount of CPU time higher? Is the number of page faults higher? Is the number of BIOs or DIOs different?

If the CPU time increased then we're likely talking something like a hardware or memory issue as with a faster CPU the time should decrease. Note that memory latencies are accounted for in CPU time.

Another idea is to graph the 10 hour process stats. You may discover that rather than a uniform 25% slower process, that something more drastic happens for a shorter period of time. Pull the T4 toll off the Freeware CD or download ECP from the network. Both tools are capable for this type of analysis.

http://h71000.www7.hp.com/openvms/products/ecp/
You may also want to go to the freeware CD
OpenVMS is here to stay
Willem Grooters
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

Alas, I don't have these statistics, but it's not too much trouble to reverse, a bit anyway. I planned downtime this weekend anyway to add an internal device...
I'll remove the added memory (being of different type (64/board iso 128)) and let the system run for a considerable time and measure, and then put it back and do the same.
Probably that would shine a light. And NO, I won't forget to run autogen ;-)

So 10 hours would be sufficient - then I'll have the data on Monday.

Stay tuned...
Willem Grooters
OpenVMS Developer & System Manager
Martin P.J. Zinser
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

Hello,

just some background on the tools mentioned:

t4 is essntially a Monitor grabber, which dumps all Monitor statistics into a CSV file. Analysis afterwards using Excel on a PC
If you had different performance metrics to add this could be done with moderate effort (but I do not think this applies in your case)

ECP comes with a DECwindows interface that you can use on your workstation. One thing to keep in mind with ECP is that it resubmits itself automatically and stores the results by default on the system disk. Either make sure to kill it once you got your results or make sure you clean the ECP directory in regular intervalls.

Greetings, Martin

P.S. Todd is correct Quantum only helps if you have more than one process competing for the CPU. OTOH I think this is exactly what he suspects ;-)

P.P.S. As for the memory, the PWS has 6 memory banks, how is the memory distributed over these. Do you have the same modules for all slots?
Willem Grooters
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

Martin:
Bank 0+1 = 128 (non-Compaq) 128Mb - a total of 256, this is the original set.
Bank 2-5 (genuine Digital...) 64Mb SIMMs (high porfile ones), I couldn't find a clue on the type, but it's said to be PWS specific.

SRM shows:

PROCESSOR
DECchip (tm) 21164A-2 Pass 600Mhz 96Kb SCache

2Mb BCache

MEMORY

Bank Size Base Addr
----- ----- ---------
0 256Mb 00000000
1 128Mb 10000000
2 128Mb 18000000

tested meory = 512

(I may have missed a line or two)

I'm collecting data at the moment....
Willem Grooters
OpenVMS Developer & System Manager
Hein van den Heuvel
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade


When switching to higher speed CPU, a CPU bound process should go faster. Period.
If not, then something is broken (like an L2 cache).

It might not go as much faster as the relative frequency increase if there are other bottleneck in play, notably memory latency & bandwidth.

So here we, the folks following this discussion, have a problem. You changed two, if not three parameters at the same time: cpu speed, memory quantity, memory quality.

As with the CPU speed, incremental memory can only be goodness... if everything alse stays the same. Typically memorys size increase will NOT speed up a CPU bound application, but it will make the system look more healthy (fewer outswapped processes).

The only way I could see memory size have an influence for a CPU bound VMS application is if said application is experiencing lots of memory resolved pagefaults: global valid, freelist or modified page list faults. If those happen a lot, clearly the working set needs to be increased and/or the free list management set to be less aggresive.

We have no indication of the memory size increase having influenced changes in working set or free/modified page list settings (no autogen) so I do not expect this to be an explanation for a reduction in speed.

So there is really on one explanation left... the 'quality' of the memory must have decreased. The latency has increased and/or the bandwidth has decreaced.

The PWS documentation I have looked at does NOT suggested it uses an interleaving memory controller. Too bad... otherwise we could blame that :-). We have practical experiences where on a Turbolaser (GS140) where cpu speed seemed slower by adding memory. There the original memory was 2*2GB at 4x interleaving and the additional 1GB module reduced the overall interleaving to 1x, reducing bandwitdh.

If it is not interleaving, it must be raw speed, like using 70ns DIMMs where 50ns DIMMs were used before. That would reduce measured CPU speed... if the application was memory access intense. (IMHO all but the 'calculate pi to the max degree' cpu apps are memory bound.)

I'd strongly suggest to rip out those 4 new (small) memory sets and go back to the original memory config, if only from a purist benchmarking point of view: only ever change one param at a time even if it 'can only help'. Now test just the new CPU. After that re-add the memory and re-measure the speed.

In closing, perhaps stating the obvious for many readers, please realize that a CPU waiting for slow memory to fill a cache line is NOT seen as IDLE but as 100% busy. Yes, the application is idly waiting to execute the next step, but the CPU is blocked in doing so and will not be 'context switched' or some such. It will be and appeal 100% CPU busy.

Hope this helps some,
Sorry for the longish story.

Met vriendelijke groetjes,
Hein.



Martin P.J. Zinser
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

Hi,

this is more an answer/request to Hein...

As Willem said, he can not find the type of memory the Digital boards actually are. Well, without having seen them I do bet they do have a 54-***** partnumber somewhere on them.

Is there anyway a mere mortal not working for
Digital/Compaq/hp can find out what this translates to in "normal" order numbers like MSP01 etc.?

A resource like this would certainly help me bunches and I bet many other customers too.

Greetings, Martin
Willem Grooters
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

It IS the memory causing the problem.

I let EPC run from Sunday 22:00 till Monday 10:00, the calculation process took about 92-100% CPU and about 10,5 hours of CPU time to complete.
Yesterday, I removed the memory, Autogen'd again and started EPC for the same period (22:00 - 10:00). I haven't seen the analysis yet, but it;s own calculation shows the proces will use less than 8 hours of CPU time to complete, meaning a gain of approx 6%. Still less then expected but much better than the lost 25%...

Anyone interrested in 8*64Mb DIGITAL memory (max CPU-speed 433Mhz)?
Willem Grooters
OpenVMS Developer & System Manager
Hein van den Heuvel
Honored Contributor

Re: Performance DECREASE after CPU/memory upgrade

> It IS the memory causing the problem.

'course it is. Logic never fails :-).


> the proces will use less than 8 hours of CPU
> time to complete, meaning a gain of approx 6%.
> Still less then expected but much better than the lost 25%...

dunno... how does one increase the speed for a PWS?
Do you just slap in a new CPU, which uses a higher clock
multiplier and leave the rest of the box alone?
Or do you turn up the system clock and with that the memory speed?

See... if the memory speed remained the same, and your
application is sensitive to memory performance (as proven
by your 'old memory' experiment) then the only time you
see the new CPU speed helping is when it is working
out of cache. So then the 6% is actually nice.
That's why faster/fastest clocked Alpha's went to a
full 16MB cache: to match cpu cpeed with memory feed.
That's why ES45's are such great little boxes as
they have lotsa 'excess' memory bus power.

btw: on the old GS140 we used to play benchmarking games.
You could turn knobs for clock speed and multiplier.
So let's say for sake of the argument you could pick
100mhz * 5 (500) or 80Mhz * 7 (560).
Which setting performed better?
Well, it depends! (as always).
Tests runing in cache would do better on 560 Mhz.
Tests runing with memory bus constraints run better
at 500 mhz cpu speed because the backplane bus was
clocked at 100.
The TL could be clocked from 62.6M through 100M.
The 88M was typical high-end.
For example the old 625 boards really ran more like 612 for 7*88.
Best I recall the 100M was never stable enough to ship and support.


> Anyone interrested in 8*64Mb DIGITAL memory (max CPU-speed 433Mhz)?

:-).