topic Re: Performance DECREASE after CPU/memory upgrade in Operating System - OpenVMS

Performance DECREASE after CPU/memory upgrade

Willem Grooters — Fri, 22 Aug 2003 07:39:58 GMT

My dear collegues,

I consider this weird.
My (old ;)) Digital PWS500au had 256 Mb memory and (as the name suggests) a 500Mhz CPU. I run a CPU-intensive application on it that takes about 8 hours to complete. A lot of other processes are loaded as well and a number of them (Decnet and (inactive) TCPIP) are in HIBO or LEFO state - which doesn't really matter.
Well, I doubled memory and installed a faster (600Mhz) CPU, and changed CPU-clock accordingly. No problem for SRM, it signals the CPU speed is 600Mhz and the system has 512Mb memory, nor a problem for VMS: I have no HIBO or LEFO processes left (good).
However: the CPU-intensive process now takes about 10,5 hours to complete - runs about 25% SLOWER where I would expect a gain of about 15% (be a bit pessimistic....)

Can anyone explain this and give me some idea what to do (I _should_ run autogen???)

Re: Performance DECREASE after CPU/memory upgrade

Martin P.J. Zinser — Fri, 22 Aug 2003 14:31:01 GMT

Hello Willem,

running AUTOGEN is certainly a first step.
It might be interesting to actually look at the report from AUTOGEN to see where it wants to change parameters.
Else you might want to check if the CPU is actually maxed out when you run your job or if there is still "free" CPU left.

Re: Performance DECREASE after CPU/memory upgrade

Mike Naime — Sat, 23 Aug 2003 01:04:11 GMT

If you did not run Autogen after adding the additional memory, your OS and Apps may not really be aware that it is all there to be used.

Whenever you change your hardware platform, you MUST run autogen to allow the system to make necessary changes to allow for that additional hardware to really be used.

If you want to see the changes that it makes save your old parameters, and do a diff between the old and new parameters.

EX:
$ Copy alphavmssys.par Alphavmssys.old
$ mcr sysgen
use current
set /output .old
show/all
show/spec
exit
(*) make edits to modparams.dat
$ @sys$update:autogen getdata setparams nofeedback
$ mcr sysgen
use current
set /output .new
show/all
show/spec
exit
$diff/par/match=1/width=80/out=x.dif .old .new
$ type x.dif
verify that the edits in modparams took hold and are correct, note other changes
and verify they look ok as well.

When you are happy, Reboot the box!

Re: Performance DECREASE after CPU/memory upgrade

Martin P.J. Zinser — Sat, 23 Aug 2003 02:18:41 GMT

Assuming your system has been up long enough
under load, you can also use autogen with savparams and feedback

Re: Performance DECREASE after CPU/memory upgrade

Willem Grooters — Mon, 25 Aug 2003 06:02:40 GMT

Thanks to all - I'll run autogen anyway.
Someone else gave me a hint it could be caused by the hardware, e.g. because the memory isn't responsive enough: processor speed 600Mhz but memory would fit up to 433, for instance, which would cause an overall decrease of memery access - even if memory that _can_ handle this speed is accessed.
Anyone an idea about this?

Re: Performance DECREASE after CPU/memory upgrade

Antoniov. — Mon, 25 Aug 2003 11:08:08 GMT

Hi Willen,
AUTOGEN can help you because some system parameters (that have value for 256 Mb and 500MHz CPU) don't work correctly with new hardware.
For example (but it's only a example)PAGEDYN could be set for 256 Mb and limit use of Dynamic Pool. If System need increase pool can overalloc new heap but system performance can down. Similar is NPAGEDYN and so on.
AUTOGEN solve theese problems.
Bye.

Re: Performance DECREASE after CPU/memory upgrade

Martin P.J. Zinser — Mon, 25 Aug 2003 16:38:19 GMT

Hello Willem,

there is nothing in the old SOC that points in the direction of non-matching main memory. For all PWS from the 433au to the 600au the MSP01-C,D,E,F are listed as supported memory modules.

Greetings, Martin

Re: Performance DECREASE after CPU/memory upgrade

Willem Grooters — Wed, 27 Aug 2003 11:49:37 GMT

I've done autogen (saveparams setparams) and rebooted (well - cycled power, due to some hardware problems I cannot reboot ;-().
Overall, it seems a bit better but the CPU-intensive program still is considerably slower than before the upgrade.
Could it be that due to extended memory, more processes are kept in memory - using at least some CPU time from time to time? (Although I don't believe this ought to lead to a decreament of 25%).

Any other ideas are welcomed...

Re: Performance DECREASE after CPU/memory upgrade

Martin P.J. Zinser — Wed, 27 Aug 2003 12:25:27 GMT

Hello Willem,

to get a first overview how your system uses the memory and if there still might be idle CPU have a look at

http://www.decus.de:8080/www/vms/sw/perf_meter.htmlx

to check if your compile process is maybe running out of quota the following tool might be useful

http://www.decus.de:8080/www/vms/sw/show_quota.htmlx

Greetings, Martin

Re: Performance DECREASE after CPU/memory upgrade

Antoniov. — Thu, 28 Aug 2003 06:11:56 GMT

Hi Willen,
can you post
a)Before change, other use can work when CPU intensive proc. was running? And now?
b)Can you increase priority process?
If you set prio=16, you set process as real-time so it goes out from round robin; setting prio=16 maybe dangerous because other process could go to stall and you must ready to decrease prio less then 16.

Martin's suggestion it's a safer good idea!

Bye,
Antoniov

Re: Performance DECREASE after CPU/memory upgrade

Willem Grooters — Thu, 28 Aug 2003 06:18:35 GMT

Martin,

Got the tools: handy to have them around! I;ll have to some tuning on processes to it's good to have them.
Well, I ran perf_mon, and I found the CPU-intensive application uses 92-100% CPU when the system is 'idle'. FYI: it runs at priority 3, so it will only be active if nothing else requires the CPU. At the moment, the system is mostly 'idle' ;-). But still, this job takes more CPU time than anticipated.
Quota shows it's NOT running out of any so that can not be the case either.

Re: Performance DECREASE after CPU/memory upgrade

Martin P.J. Zinser — Thu, 28 Aug 2003 14:07:19 GMT

Hello Willem,

one thing that might need adaption after moving to a faster CPU is the Quantum Sysgen parameter.
Default is 20. Looking around on our systems it seems that 5 might not be such a bad value to move to.

Greetings, Martin

Re: Performance DECREASE after CPU/memory upgrade

Todd Maurer — Thu, 28 Aug 2003 15:11:05 GMT

NOTE: Assuming that the issue here is a CPU intensive batch job running without much competition, then lower quantum is not a good recommendation. Lowering quantum favors better round robin servicing many processes where the number of processes in COM+CUR states typically exceeds the number of CPUs in the system.

Another suggestion is to compare the accounting information of the job before the upgrade with the accounting information from after the upgrade. See what has changed besides just the elapsed time. If the amount of CPU time higher? Is the number of page faults higher? Is the number of BIOs or DIOs different?

If the CPU time increased then we're likely talking something like a hardware or memory issue as with a faster CPU the time should decrease. Note that memory latencies are accounted for in CPU time.

Another idea is to graph the 10 hour process stats. You may discover that rather than a uniform 25% slower process, that something more drastic happens for a shorter period of time. Pull the T4 toll off the Freeware CD or download ECP from the network. Both tools are capable for this type of analysis.

http://h71000.www7.hp.com/openvms/products/ecp/
You may also want to go to the freeware CD

Re: Performance DECREASE after CPU/memory upgrade

Willem Grooters — Fri, 29 Aug 2003 06:04:38 GMT

Alas, I don't have these statistics, but it's not too much trouble to reverse, a bit anyway. I planned downtime this weekend anyway to add an internal device...
I'll remove the added memory (being of different type (64/board iso 128)) and let the system run for a considerable time and measure, and then put it back and do the same.
Probably that would shine a light. And NO, I won't forget to run autogen ;-)

So 10 hours would be sufficient - then I'll have the data on Monday.

Stay tuned...

Re: Performance DECREASE after CPU/memory upgrade

Martin P.J. Zinser — Fri, 29 Aug 2003 14:56:18 GMT

Hello,

just some background on the tools mentioned:

t4 is essntially a Monitor grabber, which dumps all Monitor statistics into a CSV file. Analysis afterwards using Excel on a PC
If you had different performance metrics to add this could be done with moderate effort (but I do not think this applies in your case)

ECP comes with a DECwindows interface that you can use on your workstation. One thing to keep in mind with ECP is that it resubmits itself automatically and stores the results by default on the system disk. Either make sure to kill it once you got your results or make sure you clean the ECP directory in regular intervalls.

Greetings, Martin

P.S. Todd is correct Quantum only helps if you have more than one process competing for the CPU. OTOH I think this is exactly what he suspects ;-)

P.P.S. As for the memory, the PWS has 6 memory banks, how is the memory distributed over these. Do you have the same modules for all slots?

Re: Performance DECREASE after CPU/memory upgrade

Willem Grooters — Sun, 31 Aug 2003 19:33:03 GMT

Martin:
Bank 0+1 = 128 (non-Compaq) 128Mb - a total of 256, this is the original set.
Bank 2-5 (genuine Digital...) 64Mb SIMMs (high porfile ones), I couldn't find a clue on the type, but it's said to be PWS specific.

SRM shows:

PROCESSOR
DECchip (tm) 21164A-2 Pass 600Mhz 96Kb SCache

2Mb BCache

MEMORY

Bank Size Base Addr
----- ----- ---------
0 256Mb 00000000
1 128Mb 10000000
2 128Mb 18000000

tested meory = 512

(I may have missed a line or two)

I'm collecting data at the moment....

Re: Performance DECREASE after CPU/memory upgrade

Hein van den Heuvel — Sun, 31 Aug 2003 23:23:46 GMT

When switching to higher speed CPU, a CPU bound process should go faster. Period.
If not, then something is broken (like an L2 cache).

It might not go as much faster as the relative frequency increase if there are other bottleneck in play, notably memory latency & bandwidth.

So here we, the folks following this discussion, have a problem. You changed two, if not three parameters at the same time: cpu speed, memory quantity, memory quality.

As with the CPU speed, incremental memory can only be goodness... if everything alse stays the same. Typically memorys size increase will NOT speed up a CPU bound application, but it will make the system look more healthy (fewer outswapped processes).

The only way I could see memory size have an influence for a CPU bound VMS application is if said application is experiencing lots of memory resolved pagefaults: global valid, freelist or modified page list faults. If those happen a lot, clearly the working set needs to be increased and/or the free list management set to be less aggresive.

We have no indication of the memory size increase having influenced changes in working set or free/modified page list settings (no autogen) so I do not expect this to be an explanation for a reduction in speed.

So there is really on one explanation left... the 'quality' of the memory must have decreased. The latency has increased and/or the bandwidth has decreaced.

The PWS documentation I have looked at does NOT suggested it uses an interleaving memory controller. Too bad... otherwise we could blame that :-). We have practical experiences where on a Turbolaser (GS140) where cpu speed seemed slower by adding memory. There the original memory was 2*2GB at 4x interleaving and the additional 1GB module reduced the overall interleaving to 1x, reducing bandwitdh.

If it is not interleaving, it must be raw speed, like using 70ns DIMMs where 50ns DIMMs were used before. That would reduce measured CPU speed... if the application was memory access intense. (IMHO all but the 'calculate pi to the max degree' cpu apps are memory bound.)

I'd strongly suggest to rip out those 4 new (small) memory sets and go back to the original memory config, if only from a purist benchmarking point of view: only ever change one param at a time even if it 'can only help'. Now test just the new CPU. After that re-add the memory and re-measure the speed.

In closing, perhaps stating the obvious for many readers, please realize that a CPU waiting for slow memory to fill a cache line is NOT seen as IDLE but as 100% busy. Yes, the application is idly waiting to execute the next step, but the CPU is blocked in doing so and will not be 'context switched' or some such. It will be and appeal 100% CPU busy.

Hope this helps some,
Sorry for the longish story.

Met vriendelijke groetjes,
Hein.

Re: Performance DECREASE after CPU/memory upgrade

Martin P.J. Zinser — Mon, 01 Sep 2003 01:02:36 GMT

Hi,

this is more an answer/request to Hein...

As Willem said, he can not find the type of memory the Digital boards actually are. Well, without having seen them I do bet they do have a 54-***** partnumber somewhere on them.

Is there anyway a mere mortal not working for
Digital/Compaq/hp can find out what this translates to in "normal" order numbers like MSP01 etc.?

A resource like this would certainly help me bunches and I bet many other customers too.

Greetings, Martin

Re: Performance DECREASE after CPU/memory upgrade

Willem Grooters — Tue, 02 Sep 2003 06:15:20 GMT

It IS the memory causing the problem.

I let EPC run from Sunday 22:00 till Monday 10:00, the calculation process took about 92-100% CPU and about 10,5 hours of CPU time to complete.
Yesterday, I removed the memory, Autogen'd again and started EPC for the same period (22:00 - 10:00). I haven't seen the analysis yet, but it;s own calculation shows the proces will use less than 8 hours of CPU time to complete, meaning a gain of approx 6%. Still less then expected but much better than the lost 25%...

Anyone interrested in 8*64Mb DIGITAL memory (max CPU-speed 433Mhz)?

Re: Performance DECREASE after CPU/memory upgrade

Hein van den Heuvel — Tue, 02 Sep 2003 13:30:11 GMT

> It IS the memory causing the problem.

'course it is. Logic never fails :-).

> the proces will use less than 8 hours of CPU
> time to complete, meaning a gain of approx 6%.
> Still less then expected but much better than the lost 25%...

dunno... how does one increase the speed for a PWS?
Do you just slap in a new CPU, which uses a higher clock
multiplier and leave the rest of the box alone?
Or do you turn up the system clock and with that the memory speed?

See... if the memory speed remained the same, and your
application is sensitive to memory performance (as proven
by your 'old memory' experiment) then the only time you
see the new CPU speed helping is when it is working
out of cache. So then the 6% is actually nice.
That's why faster/fastest clocked Alpha's went to a
full 16MB cache: to match cpu cpeed with memory feed.
That's why ES45's are such great little boxes as
they have lotsa 'excess' memory bus power.

btw: on the old GS140 we used to play benchmarking games.
You could turn knobs for clock speed and multiplier.
So let's say for sake of the argument you could pick
100mhz * 5 (500) or 80Mhz * 7 (560).
Which setting performed better?
Well, it depends! (as always).
Tests runing in cache would do better on 560 Mhz.
Tests runing with memory bus constraints run better
at 500 mhz cpu speed because the backplane bus was
clocked at 100.
The TL could be clocked from 62.6M through 100M.
The 88M was typical high-end.
For example the old 625 boards really ran more like 612 for 7*88.
Best I recall the 100M was never stable enough to ship and support.

> Anyone interrested in 8*64Mb DIGITAL memory (max CPU-speed 433Mhz)?

:-).