topic Re: DEADLOCK_WAIT parameter setting in Operating System - OpenVMS

DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Thu, 30 Oct 2008 19:51:26 GMT

Hello,

At our site a discussion is started about the DEADLOCK_WAIT parameter.

One of our development teams asked us to decrease the DEADLOCK_WAIT parameter from 5 to 1 (second) because it would increase the performance of one application dramaticly (as they have tested). As they say the performance of the application was not good because of RDB (dead) locks. (and no, fixing the application is not an option as they say).

As system management group we were not involved in the tests but at the operational system (with DEADLOCK_WAIT set to 5) we hardly see any deadlock-searches or deadlock-finds for this application (DecPS and RMU Show locks/mode=block)

The VMS system management team is not to cute about the change and also has doubts about the impact on this and other applications and the (VMS) system.

We have several questions we don't have an answer and we hope someone can give the answers or at least start a discussion.

These questions are :

1. Is the Deadlock-search process a multi threaded process what can consume multiple (all) CPU's power.

2. How long can a deadlock search take (I realize it depends on the situation but I hope someone have seen and measured it)

3. What happens (if it can happen) when a deadlock-search takes more time as the DEADLOCK_WAIT setting? By example the DEADLOCK_WAIT is set to one 1 and a DeadLock-search takes 1.5 second.

Environment :
System : AlphaServer ES45 / 4*EV68 1250Mhz/8 GB memory/ SAN storage
OS : VMS 7.3-2 with (almost) latest patches NO Cluster
DB : Oracle Rdb V7.0-73

The application is a business critical application what has to be up 24x7x365 (wishfull). Other business critical RDB applications are on the same machine.

I hope some of you have experience with a simular situation or can give other advise.

Thx, Peter

Re: DEADLOCK_WAIT parameter setting

Martin P.J. Zinser — Thu, 30 Oct 2008 20:23:55 GMT

Hello Peter,

can't comment on the details here, since I do not do much with Rdb, but I did check on one of our business critical Rdb systems (GS1280, VMS 8.3, Rdb 7.2), and DEADLOCK_WAIT is set to 1 on the box. No ill effects on the VMS side of the house with this setting.

Greetings, Martin

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Thu, 30 Oct 2008 20:33:05 GMT

Tnx for your fast reply Martin.
Do you have any idea what the default value is in VMS 8.3. In vms 7.3-2 and as long as I remember it was 10. As you say you are not involved in the RDB part you probably have no idea if and why the value of 1 is changed from the default value.

Re: DEADLOCK_WAIT parameter setting

John Gillings — Fri, 31 Oct 2008 00:30:41 GMT

Peter,

DEADLOCK_WAIT set at 10 (the default) is really rather silly. It reflects the extreme conservatism in OpenVMS engineering when it comes to changing defaults for system parameters, rather than any carefully considered logic.

The default was probably even conservative for the VAX 780 that it was most likely initially set for. Today on an Itanium it's laughable.

Deadlock searching is not polling, it's only done when necessary. Note what the docs say:

"DEADLOCK_WAIT defines the number of seconds that a lock request must wait before the system initiates a deadlock search on behalf of that lock."

So, you have to have a lock request waiting for 5 seconds before anything happens.

At compute speeds, in a "normal" processing environment, a lock will either be granted very quickly or take a "long" time. Where "long" could be approximated as anything more than a few milliseconds.

There's no practical difference between 1 and 5 seconds - they're both "very long". In other words, if the request waits for 1 second, it will probably wait for 5 seconds or more. For most workloads, I wouldn't expect to see any significant difference in the number of deadlock searches with DEADLOCK_WAIT reduced to 1. On the other hand, for processing that depends on deadlocks being broken, I'd expect a significant improvement in response (since we'll see it after 1 second instead of 5).

One would hope that most workloads don't depend on deadlock searches, but I believe there are some Rdb operations that do. I've heard advice from Rdb experts to reduce DEADLOCK_WAIT.

How long does it take? That's a "piece of string" question. It depends on the number of locks you have (and with Rdb involved that usually means "Plenty"!). However, an ES45 can crunch through a serious quanity of locks in 1 second. I'd be astonished if you found searches exceeded 1 second! Without clusters, all lock activity is local, so it's as fast as it gets. Even if a single search exceeded 1 second, that doesn't mean you'd saturate a CPU for any significant time because it's all self limiting, and you could fix it instantly by resetting the parameter.

I believe this is a very low risk suggestion, with no potential downside, and significant potential upside. I'd recommend you go ahead. On the other hand you do NOT want to set it to zero.

Get yourself some statistics. MONITOR LOCK will show you the deadlock search rate and the deadlock find rate. Run it for a typical day to get yourself a baseline, then change the parameter and measure again.

(It's a pity the parameter doesn't have a bit more granularity - even 1 second is a very long time to wait on a deadlock).

Re: DEADLOCK_WAIT parameter setting

John Gillings — Fri, 31 Oct 2008 00:41:04 GMT

Peter,
sorry, I didn't answer everything...

1) No deadlock searches will not consume all your CPU

2) - it depends, see previous response

3) It's not polling, so you can't get into an infinite loop

>Do you have any idea what the
>default value is in VMS 8.3

Still 10 on all platforms. Hey it took 20 years for the default QUANTUM to come down from 200msecs to 50msecs. Anyone care to guess what percentage of real world processes atually get to the end of their quantum?

Remember SYSGEN defaults are the extreme end of conservatism, they're intended to guarantee ANY openvms system will be bootable, nothing more, nothing less.

FWIW, we're a real live, definitely NOT just wishful, 24x7x365 shop with significant Rdb activity in production. We have DEADLOCK_WAIT set to 1 on all our systems.

Re: DEADLOCK_WAIT parameter setting

Jon Pinkley — Fri, 31 Oct 2008 04:05:29 GMT

Peter,

PvdB> Do you have any idea what the default value is in VMS 8.3?

The default on Alpha V8.3 is still 10 seconds.

$ sho sys/nopro
OpenVMS V8.3 on node EISNER 30-OCT-2008 20:54:36.76 Uptime 91 12:20:44
$ mcr sysgen sho deadlock_wait
Parameter Name Current Default Min. Max. Unit Dynamic
-------------- ------- ------- ------- ------- ---- -------
DEADLOCK_WAIT 10 10 0 -1 Seconds D
$

PvdB> As you say you are not involved in the RDB part you probably have no idea if and why the value of 1 is changed from the default value.

Google search for DEADLOCK_WAIT returned these items as well as others

Most likely, this is the source of the recommendation:

The Secrets of Performance (Thilo Lauer, Helmut Ammer)

http://www.decus.de/slides/sy2005/06_04/2G01.pdf

"DEADLOCK_WAIT=1 ! It ain't 1982 any longer"

Here's a 2005 presentation that is worth reading: http://download.oracle.com/otndocs/products/rdb/pdf/rdbtf05_locking.pdf

DEC Rdb Guide to Database Performance and Tuning

http://starlet.deltatel.ru/HyperReader/sys$common/syshlp/dymfaa42.decw$book?Chunk=612

And from the "HP OpenVMS Version 8.3 New Features and Documentation Overview" http://h71000.www7.hp.com/doc/83FINAL/6679/6679PRO.HTML

5.3 Deadlock Wait
OpenVMS V8.3 now supports the ability for a process to declare a sub-second deadlock wait time for the lock manager. This can be done with the $SET_PROCESS_PROPERTIES system service and the new item code PPROP$C_DEADLOCK_WAIT. The sub-second deadlock wait time overrides the system parameter DEADLOCK_WAIT time. In addition, the $GETJPI system service and F$GETJPI lexical function allow the retrieval of this time by using the item codes of JPI$_DEADLOCK_WAIT and DEADLOCK_WAIT. See the HP OpenVMS System Services Reference Manual and HP OpenVMS DCL Dictionary for more information on the usage.

The system parameter DEADLOCK_WAIT is in second units, so the smallest value you can set is 1 second. The sub-second deadlock wait time set via the system service $SET_PROCESS_PROPERTIES is valid only for the current image and is cleared during image rundown. The parameter passed is in 100nsec units and cannot exceed 1 sec. If the value is too small, then it is increased to a minimum value of 10msec. You can also call this system service to clear a previously set sub-second value by passing a parameter value of zero. Note the following example:

[...]
#define TEN_MSEC 100000

uint64 dead_wait;
uint64 prev_value;

//
// Set a 0.5 second deadlock wait time for the current process
//
dead_wait = 50 * TEN_MSEC;
status = sys$set_process_properties ( 0, 0, 0, PPROP$C_DEADLOCK_WAIT, dead_wait, &prev_value );
[...]

----------------------------

Deadlock searches can be expensive from a high IPL processing point of view, but the processing power has increased a lot since the 10 second default value was chosen. As long as the complexity of resource trees hasn't increased, then the percentage of time spent on deadlock searches should have gone down.

Ideally, deadlock protocols should attempt to avoid deadlocks, but that isn't always possible. I wonder what problem (and customer) requested the V8.3 enhancement to allow sub second deadlock wait times?

Good Luck,

Jon

Re: DEADLOCK_WAIT parameter setting

labadie_1 — Fri, 31 Oct 2008 07:08:26 GMT

IOTA (I/O Tima allowance) is another parameter that has an outdated default.

May be Vms should have default values according to the architecture (Vax, Alpha, Itanium) , and a basic autogen, based on the Cpu and the memory available, should run on the first boot ?

An idea, others may be better.

Re: DEADLOCK_WAIT parameter setting

Jon Pinkley — Fri, 31 Oct 2008 08:44:01 GMT

JP> Ideally, deadlock protocols should attempt to avoid deadlocks, but that isn't always possible.

I meant to say "Ideally, locking protocols ..."

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Fri, 31 Oct 2008 16:44:38 GMT

Thnx all for the remarks and help. As far as what I see now we probably go for the 1 second option.
Monday I'll start a monitor recording to take the baseline.
And if someone has more pro or cons, please leave them here. Will keep the discussion open for a cpl of days.

Re: DEADLOCK_WAIT parameter setting

Jan van den Ende — Fri, 31 Oct 2008 18:10:49 GMT

Peter,

>>>
Thnx all for the remarks and help.
<<<

Dan is het nu ongeveer tijd voor de enige valuta-van-dank: geef wat points!
En ze kosten jou alleen een paar seconden...

Proost.

Neem d'r eentje van me.

jpe

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Fri, 31 Oct 2008 23:04:31 GMT

Jan,
I will give points for sure!!! I just want to see if more reactions come in!
I appreciate all answers and will value them next week. Hope that is ok.

Re: DEADLOCK_WAIT parameter setting

Richard J Maher — Fri, 31 Oct 2008 23:05:23 GMT

Hi Peter,

As everyone else said, set DEADLOCK_WAIT to 1.

But I think that for performance you should be looking at moving to VMS 8.3. (I believe the lock tree remastering in a cluster has been much improved) Also, if anyone has experience of dedicating a CPU to a very busy DLM then I'd like to hear the experiences!

VLM and Fast I/O with Rdb Global Buffers is probably also worth looking up.

Cheer Richard Maher

PS. Be careful not to listen to everything you hear from Rdb engineering as you'll end up only opening your DB on one node :-( Of course their fanatical obsession with Row Cache and single-node configurations has had nothing to do with Rdb position in the VMS market place now has it?

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Fri, 31 Oct 2008 23:09:22 GMT

Richard,
Thanks for your reply. If it was our choice we would go to 8.3 but so far it is a management decision not to upgrade.
DLM is not an issue because it is a standalone (non cluster) node.
Peter

Re: DEADLOCK_WAIT parameter setting

Wim Van den Wyngaert — Mon, 03 Nov 2008 12:15:30 GMT

If decreasing the param increases performance significantly (wall time I asume), I would say your application has deadlocks all the time and needs to detect them fast (and it handles them too : they redo the transaction).

The reason why you have deadlocks is very often because not all programs handle father-sun relations in the same sequence. And thus one program first locks the father while others first lock the sun. This should be avoided. May be there is just 1 program misbehaving ...

fwiw

Wim

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Mon, 03 Nov 2008 13:00:58 GMT

Wim,
Avoiding the deadlocks by the code (or dbms design) is unfortunally out of our control. I hope your and others comment can help us to convince the developers.
Thanks for your advise.

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Mon, 03 Nov 2008 13:09:01 GMT

The result of this and other discussions is:
The parameter DEADLOCK_WAIT will be decreased by one (from 5 to 1) day by day instead of a big bang from 5 ==> 1.
Thanks all for your comment.

Re: DEADLOCK_WAIT parameter setting

Richard J Maher — Mon, 03 Nov 2008 22:04:31 GMT

Hi Peter,

Being cautious/prudent never hurt anybody, but the sooner the deadlock is identified the sooner the victim will be selected and the resources freed up. Concurrency levels have to be boosted.

As to why you're getting deadlocks being "out of your control", I suggest someone (in devlopment/support?)look at the DB Key being returned in the DEADLOCK message and work out if it's an index node or data page and then see if something as simple as a RESERVING clause or CONSISTENCY LEVEL on the transaction will fix things.

Cheers Richard Maher

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Mon, 03 Nov 2008 23:03:17 GMT

Thnx Richard,
I will pass your tip to the developers and the (technical) DBA. I hope it will push them in the right direction.

Re: DEADLOCK_WAIT parameter setting

Peter van den Bogaart — Wed, 05 Nov 2008 12:23:00 GMT

Thanks all for your responces.
It seems I have been a bit to generous giving points. I used the dutch school system instead of the ITRC standard. But I judged all the answers the same so I don't think it is a big deal.
As xpected, the "real and only" answer is not here but it helped us a lot anyway to decide to change the deadloack_wait parameter in steps from 5 to 1.
Peter