High MPSYNC - help

[Volker Halle]

Tom,

VMS732_UPDATE-V0100 included VMS732_MQ-V0100. VMS732_MQ-V0300 was included in VMS732_UPDATE-V0500.

There may be an 'interesting' fix in VMS732_MQ-V0300:

5.2.1 Performance Degradation

5.2.1.1 Problem Description:

When a single global section contains thousands of pshared objects, large multiprocessor systems can experience poor performance and very high MP_Synch times.


It's a pity, that HP does not keep old VMS patch descriptions online. For a problem like this, you would be very interested in that kind of information. I have kept all those patch descriptions stored locally.

Old patch descriptions are also kept online by openvms.org and decuserve.org - you'll find links in:

http://www.openvms.org/pages.php?page=Patches

Volker.

[Volker Halle]

Tom,

I've now also find old VMS patch descriptions in Ask Compaq (nowadays called: IT resource center - Search Assistant):

http://www5.itrc.hp.com/service/james/CPQhome.do

Volker.

[Hein van den Heuvel]

Jim wrote: "Oracle... lots of IOs, lots of locking, lots of CPUs. Wild guess here... are you using dedicated CPU lock manager?"

Hmmm, best I know Oracle does NOT use the VMS lock manager, so a dedicated lock manager would not help for that.

Thomas,
Is this a single system solution or 3-tier?
Is the problem on the DB server or the app side? The Millenium app can have lots of locking.

What are the lock rates according to monitor (MONI LOCK|DLOCK) or T4 (check LCK73 params)

Speaking of T4... does the 'Correlate' button show anything interesting (once you remove all per-cpu mode data).

An other potential cause for high MPSYNC is the network stack. Are you using Multinet per chance? If using VMS TCP/IP then make sure the scaleable kernel is enabled.

What does Cerner support suggest might be the cause. They ought to know!

Mostly though, listen carefully to Volker and try to report what is atually consuming MPSYNC through the SPL data.

Good luck!

Hein van den Heuvel
HvdH Performance Consulting.

[Thomas Thacker]

The problem is not on the system that serves the Oracle database. It on a node that hosts the application SRVxxxx processes (271 of them).

The lock rates in T4 for all LCK73 items is zero. Monitor lock/dlock shows there are no deadlocks.

The correlate function does not indicate indicate anything interesting (at least to me).

We are using TCPIP Service for OpenVMS V5.4 ECO 5. The scaleable kernel is enabled.

Opening a ticket with Cerner was next on my list. I just wanted to understand the problem better and do some basic research and troubleshooting first.

I have attached the output from an SPL run today (after the dedicated lock mgr change). It's not the prime MPSYNC time, but it's still higher than normal.

Thanks everyone for the responses.

Regards,
Tom

[Volker Halle]

Tom,

I'm missing the SPL ANALYZE output. This may be due to an error reported:

(10) CPU 5 has acquired spinlock at 0X835F4B80 at incorrect IPL; CPU already associated with spinlock at 0X82565380. Returning...

This looks like a possible spinlock synchronization issue or an error in SPL ANALYZE.

The output file should start with a node summary CPU statistics. Maybe try again. There is an example output in the System Analysis Tools manual:

http://h71000.www7.hp.com/doc/82FINAL/6549/6549pro_030.html#command_124

Volker.

[Thomas Thacker]

I've attached the SPL output. The summary info is in this output file.

Thanks,
Toom

[Ian Miller.]

In SDA what does
SHOW SPINLOCK/ADDR=835F4B80
show ?

[Hein van den Heuvel]

Yeah, that 835F4B80 really stands out and would seem to explain 90% of the MPSYNC time.


Spinlock % Time Held Acquires/sec Average Hold % Time Spinning
------------ ----------- ------------ ------------
835F4B80 88.8 490.6 2080501 513.6


Spinlock
Caller's PC
---------
835F4B80
80162804 PSHARED_OBJECT_CREATE_C+007F4
80162874 PSHARED_OBJECT_CREATE_C+00864

[Volker Halle]

Tom,

... and this EXACTLY matches the symptoms solved in VMS732_MQ-V0300, which you do not yet have installed.

A great example for SPL tracing - I hope you would allow me to use this for my next DECUS crashdump or SDA extension training.

Volker.

[Thomas Thacker]

It does not show much.

Show spinlock/addr=835F4B80

System dynamic spinlock structures
----------------------------------
Unknown Address 835F4B80
Owner CPU ID None DIPL 00000006
Ownership Depth FFFFFFFF Rank FFFFFFFF
Timeout Interval 007FFFFF Share Array 00000000

I have no problem with using the SDA ouput for training.

I'll get the patch installed as soon as I can get a downtime scheduled.

Thank again,
Tom