Performance DECREASE after CPU/memory upgrade

[Willem Grooters]

My dear collegues,

I consider this weird.
My (old ;)) Digital PWS500au had 256 Mb memory and (as the name suggests) a 500Mhz CPU. I run a CPU-intensive application on it that takes about 8 hours to complete. A lot of other processes are loaded as well and a number of them (Decnet and (inactive) TCPIP) are in HIBO or LEFO state - which doesn't really matter.
Well, I doubled memory and installed a faster (600Mhz) CPU, and changed CPU-clock accordingly. No problem for SRM, it signals the CPU speed is 600Mhz and the system has 512Mb memory, nor a problem for VMS: I have no HIBO or LEFO processes left (good).
However: the CPU-intensive process now takes about 10,5 hours to complete - runs about 25% SLOWER where I would expect a gain of about 15% (be a bit pessimistic....)

Can anyone explain this and give me some idea what to do (I _should_ run autogen???)

[Martin P.J. Zinser]

Hello Willem,

running AUTOGEN is certainly a first step.
It might be interesting to actually look at the report from AUTOGEN to see where it wants to change parameters.
Else you might want to check if the CPU is actually maxed out when you run your job or if there is still "free" CPU left.

[Mike Naime]

If you did not run Autogen after adding the additional memory, your OS and Apps may not really be aware that it is all there to be used.

Whenever you change your hardware platform, you MUST run autogen to allow the system to make necessary changes to allow for that additional hardware to really be used.

If you want to see the changes that it makes save your old parameters, and do a diff between the old and new parameters.

EX:
$ Copy alphavmssys.par Alphavmssys.old
$ mcr sysgen
use current
set /output .old
show/all
show/spec
exit
(*) make edits to modparams.dat
$ @sys$update:autogen getdata setparams nofeedback
$ mcr sysgen
use current
set /output .new
show/all
show/spec
exit
$diff/par/match=1/width=80/out=x.dif .old .new
$ type x.dif
verify that the edits in modparams took hold and are correct, note other changes
and verify they look ok as well.

When you are happy, Reboot the box!

[Martin P.J. Zinser]

Assuming your system has been up long enough
under load, you can also use autogen with savparams and feedback

[Willem Grooters]

Thanks to all - I'll run autogen anyway.
Someone else gave me a hint it could be caused by the hardware, e.g. because the memory isn't responsive enough: processor speed 600Mhz but memory would fit up to 433, for instance, which would cause an overall decrease of memery access - even if memory that _can_ handle this speed is accessed.
Anyone an idea about this?

[Antoniov.]

Hi Willen,
AUTOGEN can help you because some system parameters (that have value for 256 Mb and 500MHz CPU) don't work correctly with new hardware.
For example (but it's only a example)PAGEDYN could be set for 256 Mb and limit use of Dynamic Pool. If System need increase pool can overalloc new heap but system performance can down. Similar is NPAGEDYN and so on.
AUTOGEN solve theese problems.
Bye.

[Martin P.J. Zinser]

Hello Willem,

there is nothing in the old SOC that points in the direction of non-matching main memory. For all PWS from the 433au to the 600au the MSP01-C,D,E,F are listed as supported memory modules.

Greetings, Martin

[Willem Grooters]

I've done autogen (saveparams setparams) and rebooted (well - cycled power, due to some hardware problems I cannot reboot ;-().
Overall, it seems a bit better but the CPU-intensive program still is considerably slower than before the upgrade.
Could it be that due to extended memory, more processes are kept in memory - using at least some CPU time from time to time? (Although I don't believe this ought to lead to a decreament of 25%).

Any other ideas are welcomed...

[Martin P.J. Zinser]

Hello Willem,

to get a first overview how your system uses the memory and if there still might be idle CPU have a look at

http://www.decus.de:8080/www/vms/sw/perf_meter.htmlx


to check if your compile process is maybe running out of quota the following tool might be useful

http://www.decus.de:8080/www/vms/sw/show_quota.htmlx

Greetings, Martin

[Antoniov.]

Hi Willen,
can you post
a)Before change, other use can work when CPU intensive proc. was running? And now?
b)Can you increase priority process?
If you set prio=16, you set process as real-time so it goes out from round robin; setting prio=16 maybe dangerous because other process could go to stall and you must ready to decrease prio less then 16.

Martin's suggestion it's a safer good idea!

Bye,
Antoniov

[Willem Grooters]

Martin,

Got the tools: handy to have them around! I;ll have to some tuning on processes to it's good to have them.
Well, I ran perf_mon, and I found the CPU-intensive application uses 92-100% CPU when the system is 'idle'. FYI: it runs at priority 3, so it will only be active if nothing else requires the CPU. At the moment, the system is mostly 'idle' ;-). But still, this job takes more CPU time than anticipated.
Quota shows it's NOT running out of any so that can not be the case either.

[Martin P.J. Zinser]

Hello Willem,

one thing that might need adaption after moving to a faster CPU is the Quantum Sysgen parameter.
Default is 20. Looking around on our systems it seems that 5 might not be such a bad value to move to.

Greetings, Martin

[Todd Maurer]

NOTE: Assuming that the issue here is a CPU intensive batch job running without much competition, then lower quantum is not a good recommendation. Lowering quantum favors better round robin servicing many processes where the number of processes in COM+CUR states typically exceeds the number of CPUs in the system.

Another suggestion is to compare the accounting information of the job before the upgrade with the accounting information from after the upgrade. See what has changed besides just the elapsed time. If the amount of CPU time higher? Is the number of page faults higher? Is the number of BIOs or DIOs different?

If the CPU time increased then we're likely talking something like a hardware or memory issue as with a faster CPU the time should decrease. Note that memory latencies are accounted for in CPU time.

Another idea is to graph the 10 hour process stats. You may discover that rather than a uniform 25% slower process, that something more drastic happens for a shorter period of time. Pull the T4 toll off the Freeware CD or download ECP from the network. Both tools are capable for this type of analysis.

http://h71000.www7.hp.com/openvms/products/ecp/
You may also want to go to the freeware CD

[Willem Grooters]

Alas, I don't have these statistics, but it's not too much trouble to reverse, a bit anyway. I planned downtime this weekend anyway to add an internal device...
I'll remove the added memory (being of different type (64/board iso 128)) and let the system run for a considerable time and measure, and then put it back and do the same.
Probably that would shine a light. And NO, I won't forget to run autogen ;-)

So 10 hours would be sufficient - then I'll have the data on Monday.

Stay tuned...

[Martin P.J. Zinser]

Hello,

just some background on the tools mentioned:

t4 is essntially a Monitor grabber, which dumps all Monitor statistics into a CSV file. Analysis afterwards using Excel on a PC
If you had different performance metrics to add this could be done with moderate effort (but I do not think this applies in your case)

ECP comes with a DECwindows interface that you can use on your workstation. One thing to keep in mind with ECP is that it resubmits itself automatically and stores the results by default on the system disk. Either make sure to kill it once you got your results or make sure you clean the ECP directory in regular intervalls.

Greetings, Martin

P.S. Todd is correct Quantum only helps if you have more than one process competing for the CPU. OTOH I think this is exactly what he suspects ;-)

P.P.S. As for the memory, the PWS has 6 memory banks, how is the memory distributed over these. Do you have the same modules for all slots?

[Willem Grooters]

Martin:
Bank 0+1 = 128 (non-Compaq) 128Mb - a total of 256, this is the original set.
Bank 2-5 (genuine Digital...) 64Mb SIMMs (high porfile ones), I couldn't find a clue on the type, but it's said to be PWS specific.

SRM shows:

PROCESSOR
DECchip (tm) 21164A-2 Pass 600Mhz 96Kb SCache

2Mb BCache

MEMORY

Bank Size Base Addr
----- ----- ---------
0 256Mb 00000000
1 128Mb 10000000
2 128Mb 18000000

tested meory = 512

(I may have missed a line or two)

I'm collecting data at the moment....

[Hein van den Heuvel]

When switching to higher speed CPU, a CPU bound process should go faster. Period.
If not, then something is broken (like an L2 cache).

It might not go as much faster as the relative frequency increase if there are other bottleneck in play, notably memory latency & bandwidth.

So here we, the folks following this discussion, have a problem. You changed two, if not three parameters at the same time: cpu speed, memory quantity, memory quality.

As with the CPU speed, incremental memory can only be goodness... if everything alse stays the same. Typically memorys size increase will NOT speed up a CPU bound application, but it will make the system look more healthy (fewer outswapped processes).

The only way I could see memory size have an influence for a CPU bound VMS application is if said application is experiencing lots of memory resolved pagefaults: global valid, freelist or modified page list faults. If those happen a lot, clearly the working set needs to be increased and/or the free list management set to be less aggresive.

We have no indication of the memory size increase having influenced changes in working set or free/modified page list settings (no autogen) so I do not expect this to be an explanation for a reduction in speed.

So there is really on one explanation left... the 'quality' of the memory must have decreased. The latency has increased and/or the bandwidth has decreaced.

The PWS documentation I have looked at does NOT suggested it uses an interleaving memory controller. Too bad... otherwise we could blame that :-). We have practical experiences where on a Turbolaser (GS140) where cpu speed seemed slower by adding memory. There the original memory was 2*2GB at 4x interleaving and the additional 1GB module reduced the overall interleaving to 1x, reducing bandwitdh.

If it is not interleaving, it must be raw speed, like using 70ns DIMMs where 50ns DIMMs were used before. That would reduce measured CPU speed... if the application was memory access intense. (IMHO all but the 'calculate pi to the max degree' cpu apps are memory bound.)

I'd strongly suggest to rip out those 4 new (small) memory sets and go back to the original memory config, if only from a purist benchmarking point of view: only ever change one param at a time even if it 'can only help'. Now test just the new CPU. After that re-add the memory and re-measure the speed.

In closing, perhaps stating the obvious for many readers, please realize that a CPU waiting for slow memory to fill a cache line is NOT seen as IDLE but as 100% busy. Yes, the application is idly waiting to execute the next step, but the CPU is blocked in doing so and will not be 'context switched' or some such. It will be and appeal 100% CPU busy.

Hope this helps some,
Sorry for the longish story.

Met vriendelijke groetjes,
Hein.

[Martin P.J. Zinser]

Hi,

this is more an answer/request to Hein...

As Willem said, he can not find the type of memory the Digital boards actually are. Well, without having seen them I do bet they do have a 54-***** partnumber somewhere on them.

Is there anyway a mere mortal not working for
Digital/Compaq/hp can find out what this translates to in "normal" order numbers like MSP01 etc.?

A resource like this would certainly help me bunches and I bet many other customers too.

Greetings, Martin

[Willem Grooters]

It IS the memory causing the problem.

I let EPC run from Sunday 22:00 till Monday 10:00, the calculation process took about 92-100% CPU and about 10,5 hours of CPU time to complete.
Yesterday, I removed the memory, Autogen'd again and started EPC for the same period (22:00 - 10:00). I haven't seen the analysis yet, but it;s own calculation shows the proces will use less than 8 hours of CPU time to complete, meaning a gain of approx 6%. Still less then expected but much better than the lost 25%...

Anyone interrested in 8*64Mb DIGITAL memory (max CPU-speed 433Mhz)?

[Hein van den Heuvel]

> It IS the memory causing the problem.

'course it is. Logic never fails :-).


> the proces will use less than 8 hours of CPU
> time to complete, meaning a gain of approx 6%.
> Still less then expected but much better than the lost 25%...

dunno... how does one increase the speed for a PWS?
Do you just slap in a new CPU, which uses a higher clock
multiplier and leave the rest of the box alone?
Or do you turn up the system clock and with that the memory speed?

See... if the memory speed remained the same, and your
application is sensitive to memory performance (as proven
by your 'old memory' experiment) then the only time you
see the new CPU speed helping is when it is working
out of cache. So then the 6% is actually nice.
That's why faster/fastest clocked Alpha's went to a
full 16MB cache: to match cpu cpeed with memory feed.
That's why ES45's are such great little boxes as
they have lotsa 'excess' memory bus power.

btw: on the old GS140 we used to play benchmarking games.
You could turn knobs for clock speed and multiplier.
So let's say for sake of the argument you could pick
100mhz * 5 (500) or 80Mhz * 7 (560).
Which setting performed better?
Well, it depends! (as always).
Tests runing in cache would do better on 560 Mhz.
Tests runing with memory bus constraints run better
at 500 mhz cpu speed because the backplane bus was
clocked at 100.
The TL could be clocked from 62.6M through 100M.
The 88M was typical high-end.
For example the old 625 boards really ran more like 612 for 7*88.
Best I recall the 100M was never stable enough to ship and support.


> Anyone interrested in 8*64Mb DIGITAL memory (max CPU-speed 433Mhz)?

:-).

[Martin P.J. Zinser]

Well given the fact that the 64 MB modules in my PWS are certainly not "high profile" but rather use flat memory chips it very well might be you do not have PWS memory at your hands but just something that happend to kind-of-sort-of work.
As I said there should be a many digit partnumber either on the board or on a label on one of the chips that somebody from within the company formerly known as Digital (Hint ;-) should be able to translate into an intelligible part number for outsiders.

[Willem Grooters]

So, the measurements.
I compared both analysis's and put the greater differences into the attached file. What's isn't shown is equal on boths runs, or differ less then 1% (as far as the printout can show)
As shown, CPU utilisation dropped from avarage almost 95% to 86%. prove memory was the problem?

Hein: on a PWS there's a dipswitch setting to be changed, and I've been told that this setteing controls both CPU as memroy bus (and PCI). I don't know whether this is true; anyway: SRM tells the CPU runs on 600Mhz.
So that setting IS correct.

Well, good to know where the problem is, and what I could do about it (get faster memory).

Thanks all for their suggestions. This thread can be considered closed (unless someone can help me out with the RIGHT memory ;-))

[Willem Grooters]

Martin,

Ok, you posted your reply when I was busy processing mine....

"high profile" - I meant: they're twice as high as the originally installed (about 2" in stead of abot 1"). The chips are flat,as usual.
On the plate it's (really!) "D I G I T A L' - the famous boxed characters. I could look for a partnumber easily (when I get home again)...

[Willem Grooters]

...
and here it is:

I found 2 numbers on the plain:

5024808-01 A01 and 54024829-DA A01

Since the latter number is also on a small sticker on one of the memory chips, I guess that's the product number.

The memory chips are all SEC KM44S4020AT-G10 from Korea. Whatwever that number may mean but itr mnay ring a bell somewhere.

I would be delighted if someone could tell me to what speed (Alpha) these SIMMs are useful. Does nayone use these SIMMs in an Intel workstation?

(That's final....)

[Martin P.J. Zinser]

Well they do not coincide with the 5025091-01 C01
resp. 54-25092-BA on the 64 MB banks I do have in my desk which supposedly are original PWS memory :-)

If or if not they both actually were sold as MSP01 I think only hp can tell (If Hein comes up with something on this, please be generous with points for him ;-))