CPU upgrade and Dedicated CPU Lock Manager

[Michail Bondarenko]

Hello,

Have the cluster from 4 nodes. 3xGS1280 and 1xGS80. CPU configuration is 2x10CPU(GS1280)+8CPU(GS1280)+8CPU(GS80). After adding 2x6 additional CPU at 2 nodes (2x16CPU at GS1280) cause too much performance degradation.
VMS version 8.2

After new CPU "MP Synchronization" is ~600% at both upgraded nodes. Temporary soliution was disabled 6 CPU from VMS. Performance is the same before upgrade.
But in future we need to use more CPU.

I found I can use "Dedicated CPU Lock Manager", but:
1. OS version 8.2
2. I'm using MC adapter to connect nodes in cluster
3. I'm using Fiber Chanel HBA to connect SAN disks.
4. I'm using Fast Path Devices

Could I try to use this manager or upgrading my system v8.3?

[Hein van den Heuvel]

>>> After adding 2x6 additional CPU at 2 nodes (2x16CPU at GS1280) cause too much performance degradation.

What were they thinking?
Why did they believe more CPUs would help?
I mean it might, but it needs to be reasoned out!

>>> After new CPU "MP Synchronization" is ~600% at both upgraded nodes.

Q1> what was it before?
Q2> Which spinlock is the main culprit? Did you get details? ( @SYS$EXAMPLES:SPL )

>>> I found I can use "Dedicated CPU Lock Manager",

Does the application use the lock manager at all? Probably yes, and probably it is highly desirable to use a dedicated lock manager on the nodes with more than 8 CPUs, but just switching it on without due investigation is much like investing hunderds of thousands of dollars in more CPUs just because you can.

So have someone do the homework, probably based on MONITOR, T4, and SDA SPL data.

>> 1. OS version 8.2

That's just silly to have stayed there. Upgrade, re-asses, get out the cavalery if need be.

>> Could I try to use this manager or upgrading my system v8.3?

Probably both, but what is this 'try' word? get some facts behind the proposals!

What kind of application is this, and notably what database technique is used? Oracle RAC ? Oracle RDB ? RMS (indexed) files ? homegrown?


Hope this helps some more.
Hein van den Heuvel
HvdH Performance Consulting

[Andy Bustamante]

There's always one answer that fits for performance questions: "It depends."

As Hein points out, you really need to gather information on what the application(s) and users are doing to consume resources. Define a baseline, make a change and measure the impact on the system.

One option may be to removethe GS-80 completely and move to the GS-1280s. Again, "it depends." Measure change and meaure.

Upgrading to VMS 8.3, soon to be 8.4 may provide some benefits. The dedicated lock manager may or may not be helpful.

It can take days of reviewing an environment, depending on complexity and applications before a reasonable suggestion might be made. Here, you're going to get generic gather data, try it and gather more data replies. Using a consultant, possibly allowing remote access, may be the most graceful and fastest way to document and begin this process. There are several folks here who can provide these services.

[Bob Blunt]

Fully agree with both Hein and Andy. I'd go a bit further, though, with some general recommendations. OpenVMS works best when your configuration changes are measured and planned before the hardware is purchasd and arrives for installation. There are operating system-internal setup and hardware configuration details that are necessary and require consideration before you add processors, memory, I/O... OpenVMS has greatly improved it's use of SMP systems and highly complex hardware like the GS1280 must absolutely be configured optimally for their workload to run properly.

The dedicated lock manager also requires very specific setup and configuration in addition to setting up other processes to use the correct CPUs for optimal performance.

This sort of work requires a LOT of initial work and setup and an extensive knowledge of your environment. None of this should be approached casually.

bob

[P Muralidhar Kini]

Hi Michail,

I also have the same views as already mentioned by others.

>> So have someone do the homework, probably based on MONITOR, T4, and
>> SDA SPL data.
Thats right.
As Hein has suggested, you need to use these tools in order to find out why the
MP synch is too high. Once you collect the above data you would know which
spinlock are the ones where you are causing a high MP sync.

For using Monitor, refer help
$ HELP MONITOR

For SPL tracing , refer help
$ANAL/SYS
SDA>SPL

For Dedicated CPU lock manager, Refer
http://h71000.www7.hp.com/doc/73final/6491/6491pro_015.html

T4 is something, you have to download and install
For T4, Refer
http://h71000.www7.hp.com/openvms/products/t4/

Hope this helps.

Regards,
Murali

[John McL]

It sounds like massive amounts of Lock traffic bouncing around the cluster.

What are you running on these systems? Is it in-house software or third-party software? Are the machines used homogenously (i.e. same apps on all systems?)

If the software is in-house then there maybe there are changes that can be made to lock handling.(e.g. use system-wide locks rather than cluster-wide where possible). If you run third-party software then maybe running on one machine but having warm standby (i.e. ability to startup on another machine) is a possibility.

Is I/O intensive software running on all nodes? May be you can split the cluster in two and divide the applications.

Adding more machines to a cluster is not always the best answer if it is going to generate huge amounts of inter-node traffic.

[P Muralidhar Kini]

Hi Michail,

Also,
You may find the following link usefull -
http://forums13.itrc.hp.com/service/forums/questionanswer.do?threadId=1044656

Hope this helps.

Regards,
Murali

[Michail Bondarenko]

we are using GemBase application. Gembase works RMS (indexed) files. All 4 nodes are using the same disks and DB, I can not split cluster.

[abrsvc]

To properly assist with this issue, a site visit should be scheduled. Where are you located. While all here can contribute ideas, a better understanding +/or view of the actual application and system should cut down the time to resolve this.

Where are you located?

Dan

[Hein van den Heuvel]

>> Gembase works RMS (indexed) files.

RMS Global buffers will be the answer.

Regards,
Hein RMS van den Heuvel

[P Muralidhar Kini]

Hi Michail,

As has been already suggested, you need to use the SDA SPL extension to find
out which spinlock is causing the high MP Sync value.
It may be just one spinlock that is causing this issue.

If you are lucky, May be this issue has already been reported and fixed also.

>> Hein RMS van den Heuvel
hmm. The word RMS getting in your name doesnt suprise me!.

Regards,
Murali