DEADLOCK_WAIT parameter setting

[Peter van den Bogaart]

Hello,

At our site a discussion is started about the DEADLOCK_WAIT parameter.

One of our development teams asked us to decrease the DEADLOCK_WAIT parameter from 5 to 1 (second) because it would increase the performance of one application dramaticly (as they have tested). As they say the performance of the application was not good because of RDB (dead) locks. (and no, fixing the application is not an option as they say).

As system management group we were not involved in the tests but at the operational system (with DEADLOCK_WAIT set to 5) we hardly see any deadlock-searches or deadlock-finds for this application (DecPS and RMU Show locks/mode=block)

The VMS system management team is not to cute about the change and also has doubts about the impact on this and other applications and the (VMS) system.

We have several questions we don't have an answer and we hope someone can give the answers or at least start a discussion.

These questions are :

1. Is the Deadlock-search process a multi threaded process what can consume multiple (all) CPU's power.

2. How long can a deadlock search take (I realize it depends on the situation but I hope someone have seen and measured it)

3. What happens (if it can happen) when a deadlock-search takes more time as the DEADLOCK_WAIT setting? By example the DEADLOCK_WAIT is set to one 1 and a DeadLock-search takes 1.5 second.

Environment :
System : AlphaServer ES45 / 4*EV68 1250Mhz/8 GB memory/ SAN storage
OS : VMS 7.3-2 with (almost) latest patches NO Cluster
DB : Oracle Rdb V7.0-73

The application is a business critical application what has to be up 24x7x365 (wishfull). Other business critical RDB applications are on the same machine.



I hope some of you have experience with a simular situation or can give other advise.

Thx, Peter

[Martin P.J. Zinser]

Hello Peter,

can't comment on the details here, since I do not do much with Rdb, but I did check on one of our business critical Rdb systems (GS1280, VMS 8.3, Rdb 7.2), and DEADLOCK_WAIT is set to 1 on the box. No ill effects on the VMS side of the house with this setting.

Greetings, Martin

[Peter van den Bogaart]

Tnx for your fast reply Martin.
Do you have any idea what the default value is in VMS 8.3. In vms 7.3-2 and as long as I remember it was 10. As you say you are not involved in the RDB part you probably have no idea if and why the value of 1 is changed from the default value.

[John Gillings · Accepted solution]

Peter,

DEADLOCK_WAIT set at 10 (the default) is really rather silly. It reflects the extreme conservatism in OpenVMS engineering when it comes to changing defaults for system parameters, rather than any carefully considered logic.

The default was probably even conservative for the VAX 780 that it was most likely initially set for. Today on an Itanium it's laughable.

Deadlock searching is not polling, it's only done when necessary. Note what the docs say:

"DEADLOCK_WAIT defines the number of seconds that a lock request must wait before the system initiates a deadlock search on behalf of that lock."

So, you have to have a lock request waiting for 5 seconds before anything happens.

At compute speeds, in a "normal" processing environment, a lock will either be granted very quickly or take a "long" time. Where "long" could be approximated as anything more than a few milliseconds.

There's no practical difference between 1 and 5 seconds - they're both "very long". In other words, if the request waits for 1 second, it will probably wait for 5 seconds or more. For most workloads, I wouldn't expect to see any significant difference in the number of deadlock searches with DEADLOCK_WAIT reduced to 1. On the other hand, for processing that depends on deadlocks being broken, I'd expect a significant improvement in response (since we'll see it after 1 second instead of 5).

One would hope that most workloads don't depend on deadlock searches, but I believe there are some Rdb operations that do. I've heard advice from Rdb experts to reduce DEADLOCK_WAIT.

How long does it take? That's a "piece of string" question. It depends on the number of locks you have (and with Rdb involved that usually means "Plenty"!). However, an ES45 can crunch through a serious quanity of locks in 1 second. I'd be astonished if you found searches exceeded 1 second! Without clusters, all lock activity is local, so it's as fast as it gets. Even if a single search exceeded 1 second, that doesn't mean you'd saturate a CPU for any significant time because it's all self limiting, and you could fix it instantly by resetting the parameter.

I believe this is a very low risk suggestion, with no potential downside, and significant potential upside. I'd recommend you go ahead. On the other hand you do NOT want to set it to zero.

Get yourself some statistics. MONITOR LOCK will show you the deadlock search rate and the deadlock find rate. Run it for a typical day to get yourself a baseline, then change the parameter and measure again.

(It's a pity the parameter doesn't have a bit more granularity - even 1 second is a very long time to wait on a deadlock).

[John Gillings]

Peter,
sorry, I didn't answer everything...

1) No deadlock searches will not consume all your CPU

2) - it depends, see previous response

3) It's not polling, so you can't get into an infinite loop

>Do you have any idea what the
>default value is in VMS 8.3

Still 10 on all platforms. Hey it took 20 years for the default QUANTUM to come down from 200msecs to 50msecs. Anyone care to guess what percentage of real world processes atually get to the end of their quantum?

Remember SYSGEN defaults are the extreme end of conservatism, they're intended to guarantee ANY openvms system will be bootable, nothing more, nothing less.

FWIW, we're a real live, definitely NOT just wishful, 24x7x365 shop with significant Rdb activity in production. We have DEADLOCK_WAIT set to 1 on all our systems.

[Jon Pinkley]

Peter,

PvdB> Do you have any idea what the default value is in VMS 8.3?

The default on Alpha V8.3 is still 10 seconds.

$ sho sys/nopro
OpenVMS V8.3 on node EISNER 30-OCT-2008 20:54:36.76 Uptime 91 12:20:44
$ mcr sysgen sho deadlock_wait
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
DEADLOCK_WAIT 10 10 0 -1 Seconds D
$

PvdB> As you say you are not involved in the RDB part you probably have no idea if and why the value of 1 is changed from the default value.

Google search for DEADLOCK_WAIT returned these items as well as others

Most likely, this is the source of the recommendation:

The Secrets of Performance (Thilo Lauer, Helmut Ammer)

http://www.decus.de/slides/sy2005/06_04/2G01.pdf

"DEADLOCK_WAIT=1 ! It ain't 1982 any longer"

Here's a 2005 presentation that is worth reading: http://download.oracle.com/otndocs/products/rdb/pdf/rdbtf05_locking.pdf

DEC Rdb Guide to Database Performance and Tuning

http://starlet.deltatel.ru/HyperReader/sys$common/syshlp/dymfaa42.decw$book?Chunk=612

And from the "HP OpenVMS Version 8.3 New Features and Documentation Overview" http://h71000.www7.hp.com/doc/83FINAL/6679/6679PRO.HTML


5.3 Deadlock Wait
OpenVMS V8.3 now supports the ability for a process to declare a sub-second deadlock wait time for the lock manager. This can be done with the $SET_PROCESS_PROPERTIES system service and the new item code PPROP$C_DEADLOCK_WAIT. The sub-second deadlock wait time overrides the system parameter DEADLOCK_WAIT time. In addition, the $GETJPI system service and F$GETJPI lexical function allow the retrieval of this time by using the item codes of JPI$_DEADLOCK_WAIT and DEADLOCK_WAIT. See the HP OpenVMS System Services Reference Manual and HP OpenVMS DCL Dictionary for more information on the usage.

The system parameter DEADLOCK_WAIT is in second units, so the smallest value you can set is 1 second. The sub-second deadlock wait time set via the system service $SET_PROCESS_PROPERTIES is valid only for the current image and is cleared during image rundown. The parameter passed is in 100nsec units and cannot exceed 1 sec. If the value is too small, then it is increased to a minimum value of 10msec. You can also call this system service to clear a previously set sub-second value by passing a parameter value of zero. Note the following example:



[...]
#define TEN_MSEC 100000

uint64 dead_wait;
uint64 prev_value;

//
// Set a 0.5 second deadlock wait time for the current process
//
dead_wait = 50 * TEN_MSEC;
status = sys$set_process_properties ( 0, 0, 0, PPROP$C_DEADLOCK_WAIT, dead_wait, &prev_value );
[...]

----------------------------

Deadlock searches can be expensive from a high IPL processing point of view, but the processing power has increased a lot since the 10 second default value was chosen. As long as the complexity of resource trees hasn't increased, then the percentage of time spent on deadlock searches should have gone down.

Ideally, deadlock protocols should attempt to avoid deadlocks, but that isn't always possible. I wonder what problem (and customer) requested the V8.3 enhancement to allow sub second deadlock wait times?

Good Luck,

Jon

[labadie_1]

IOTA (I/O Tima allowance) is another parameter that has an outdated default.

May be Vms should have default values according to the architecture (Vax, Alpha, Itanium) , and a basic autogen, based on the Cpu and the memory available, should run on the first boot ?

An idea, others may be better.

[Jon Pinkley]

JP> Ideally, deadlock protocols should attempt to avoid deadlocks, but that isn't always possible.

I meant to say "Ideally, locking protocols ..."

[Peter van den Bogaart]

Thnx all for the remarks and help. As far as what I see now we probably go for the 1 second option.
Monday I'll start a monitor recording to take the baseline.
And if someone has more pro or cons, please leave them here. Will keep the discussion open for a cpl of days.

[Jan van den Ende]

Peter,

>>>
Thnx all for the remarks and help.
<<<

Dan is het nu ongeveer tijd voor de enige valuta-van-dank: geef wat points!
En ze kosten jou alleen een paar seconden...

Proost.

Neem d'r eentje van me.

jpe