topic Re: Finding Record Locks in Operating System - OpenVMS

Finding Record Locks

Robert Atkinson — Mon, 16 Oct 2006 05:17:41 GMT

I know there are a few utilities to find the owner of record locks, but I've never managed to get any of them working.

Could anyone give me the SDA commands to manually find record locks?

Thanks, Robert.

Re: Finding Record Locks

Jim_McKinney — Mon, 16 Oct 2006 08:47:37 GMT

Though I haven't used it, I understand that AMDS is capable of locating the source of blocked locks.

You might also just simply hop into SDA and

SDA> show resources/contention

and see if you 'get lucky'.

Regarding your question... the answer is dependent upon how the record locks originate. Is this a cluster? More importantly, is it RMS that is generating the record locks? Anyone attempting to respond to this would likely want to know.

I don't believe that the format of an RMS lock is documented. (From memory) the resource name consists of the string "RMS$", the 3 words of the file-id, and the associated device lock name.

Re: Finding Record Locks

Hein van den Heuvel — Mon, 16 Oct 2006 17:40:23 GMT

I'll attach one of those utilities, but it is also hard to get to work. It needs CMEXEC privs, and you give it a pid, or a device spec as argument. I have some more tools, but those are not mine,

It would help to describe the circumstances better. If one process is waiting for an other holding a lock, then SDA is easy.
Just find the PID of the waiting process. Then:
SDA> SHOW PROCECS /LOCK

Now a record lock will show as an 8 byte long lock name with the VBN and RECORD ID encoded. The parrent of the lock should be a RMS$ file lock as the first reply indicates. The tool helps with this, formatting the lock, and dumping the record.

If a process starts, but dumps out on a locked record, then it is harder to tell, unless you have the target ID or KEY value.
You can use the key with DCL READ/KEY=xxx/WAIT and then use SDA on the DCL process (yourself) as per above.

Or use SHOW DEVICE/FILE and use SDA on each process havign the file open. The tool helps with this situation also.

What is the exact real problem you are trying to solve?

What is the secondary problem you have with the tools. Which tools? Where can we look at them?

You'll find an older version of my tools in:
http://h71000.www7.hp.com/freeware/freeware60/rms_tools

'next' to it you'll find a tool called 'rms_locks' which I have not tried but sounds like it might do the job.

Hope this helps,
Hein.

Re: Finding Record Locks

Andy Bustamante — Mon, 16 Oct 2006 19:47:50 GMT

In addition to AMDS, Availability Manager can display "interesting locks" in a system or cluster. See http://h71000.www7.hp.com/openvms/products/availman/docs.html. AMDS and Availability Manager are two sides of the same gem. The data manager display in AM can function on a recent Windows box, which does have a plus or two.

Andy

Re: Finding Record Locks

Robert Atkinson — Tue, 17 Oct 2006 04:10:20 GMT

To clarify, we are using RMS indexed files.

We often find programs running in batch 'hang' waiting for a record lock to be released. These locks are taken out by interactive users updating a particular record. As time goes by, a chain of locks can start to accumulate, as more and more users try to access the same record.

Although I do know the file that's being locked and the process PID that's trying to access the record, I don't know which exact record is being locked.

Ultimately, I need to know which user is locking the record the batch process is trying to access, but with 100+ users accessing that particular file, it makes the job difficult.

I had a look at Availability Manager yesterday, but wasn't 100% sure it would give the information I required. I may still install it on our test node and see what it can tell me.

Hein - could you email me an OBJ of your tool - ratkinson-at-tbs-ltd-dot-co-dot-uk.

My assumption is that I'm going to have to cobble together a mixture of tools to do this, which is very annoying, as the code within the application we use is capable of telling you who is locking the record but only sends a message every 10 minutes!

Rob.

Re: Finding Record Locks

Wim Van den Wyngaert — Tue, 17 Oct 2006 05:02:14 GMT

Robert,

Users are updating the records while the batch job is running. So, why don't you modify the batch job to read regardless of locks ?

Fwiw

Wim

Re: Finding Record Locks

Robert Atkinson — Tue, 17 Oct 2006 05:26:31 GMT

Because the batch process is also updating the records (stock figures).

Due to the way it's designed, there is no other choice but to wait for the lock to be released, from either the interactive processes or the batch jobs.

Rob.

Re: Finding Record Locks

Volker Halle — Tue, 17 Oct 2006 07:41:08 GMT

Robert,

AMDS/Availabilty Manager exactly does this job for you. You need to run RMDRIVER (@SYS$STARTUP:AMDS$STARTUP START) on all members of the cluster, which may be involved accessing those ISAM files.

If you run the GUI (data collector), you need to start collecting Lock Contention data. AMDS/Availability Manager will then find blocked locks and also identify blocking locks and show appropriate messages in the Event Windows and also record them in the AvailManLock.Log file.

This only works, if the process is actually waiting for the lock, i.e. the $ENQ operation is pending.

Volker.

Re: Finding Record Locks

Robert Atkinson — Tue, 17 Oct 2006 07:51:51 GMT

Thanks Volker.

Are there any 'gotchas' in the AM installation I should know about?

Rob.

Re: Finding Record Locks

Volker Halle — Tue, 17 Oct 2006 08:11:05 GMT

Rob,

I would suggest that you read the installation notes...

If your OpenVMS systems have multiple LAN devices, you need to define the correct device in AMDS$LOGICALS.COM. The AMDS LAN protocol only works on the same LAN segment, so you need to connect your AM data collector (GUI) to the same LAN segment as your OpenVMS nodes. You may also install and run the GUI on a Windows system.

Here are some examples of lock related AM messages (from the Availability Manager event log - V2.6A):

LCKCNT AXTST1 possible contention for resource QMAN$JBC_ALIVE_01

LCKBLK AXTST1 JOB_CONTROL blocking resource QMAN$JBC_ALIVE_01

LCKWAT AXTST1 QUEUE_MANAGER waiting for resource QMAN$JBC_ALIVE_01 granted to QUEUE_MANAGERQUEUE_MANAGER on node AXTST1

Note that the contents of the LCKWAT message is incorrect regarding the blocking process name - this has already been reported.

The Lock Contention information in the AvailManLock.Log file is more detailled and correct.

Note that you can also follow the blocked locks using SDA, but this is very tedious and time-consuming, especially in a cluster.

Volker.

Re: Finding Record Locks

Andy Bustamante — Tue, 17 Oct 2006 11:47:45 GMT

Adding to Volker's comment, AM also has the ability to configure access levels, see the documentation for configuration option of SYS$MANAGER:AMDS$DRIVER_ACCESS.DAT. The default access is monitoring. You also have the ability to adjust quotas, force quorum to be recalculated or even crash a node remotely.

I presently use Windows 2000 server with AM on the local LAN segment. With Terminal Services I can control the PC remotely. Access is restricted by VPN accounts.

Andy

Re: Finding Record Locks

Hein van den Heuvel — Tue, 17 Oct 2006 12:18:07 GMT

>>> Hein - could you email me an OBJ of your tool - ratkinson-at-tbs-ltd-dot-co-dot-uk.

You can download a (slightly older) executable from the OpenVMS freeware site, or request an account on eisner.decus.org or the deathrow cluster (just google +openvms +deathrow)

>>> My assumption is that I'm going to have to cobble together a mixture of tools to do this, which is very annoying, as the code within the application we use is capable of telling you who is locking the record but only sends a message every 10 minutes!

IF your (batch) application waits for the lock then my tool will do it all, giving it the batch job process ID. ANAL/SYS... set proce 'batch job'... SHOW PROC/LOCK will also quickly show who is halding the lock.

Good luck,
Hein.

Re: Finding Record Locks

B Claremont — Tue, 17 Oct 2006 17:48:55 GMT

You might want to take a look at the Lotsa Lock chapter in Bruce Ellis's "Hitchhikers Guide to VMS" for more hints on writing code to access lock information.

Bruce Claremont
www.MigrationSpecialties.com

Re: Finding Record Locks

Jim_McKinney — Tue, 17 Oct 2006 18:13:46 GMT

Attached is a VAX-to-Alpha port of Bruce Ellis' blocking_lock program found in his HitchHiker book.

Re: Finding Record Locks

Robert Atkinson — Wed, 18 Oct 2006 02:50:54 GMT

Doesn't look like Bruce's program works on my VMS 7.3-2 system, but thanks for sending it in Jim :-

ALPHA_ROB$ link showlock.obj
%LINK-W-NUDFSYMS, 9 undefined symbols:
%LINK-I-UDFSYM, EXE$CVT_EPID_TO_PCB
%LINK-I-UDFSYM, EXE$SIGTORET
%LINK-I-UDFSYM, LCK$GL_MAXID
%LINK-I-UDFSYM, LCK$GQ_IDTBL
%LINK-I-UDFSYM, SCH$GL_PCBVEC
%LINK-I-UDFSYM, SMP$ACQUIRE
%LINK-I-UDFSYM, SMP$GL_FLAGS
%LINK-I-UDFSYM, SMP$RELEASE
%LINK-I-UDFSYM, SYS$VERSION_BEGIN
%LINK-W-USEUNDEF, undefined symbol SYS$VERSION_BEGIN referenced
in psect $LINKAGE offset %X00000040
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol SMP$RELEASE referenced
in psect $LOCK_LINKAGE_2 offset %X00000050
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol SMP$ACQUIRE referenced
in psect $LOCK_LINKAGE_2 offset %X00000060
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol SMP$GL_FLAGS referenced
in psect $LOCK_LINKAGE_2 offset %X00000070
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol EXE$SIGTORET referenced
in psect $LOCK_LINKAGE_2 offset %X00000090
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol LCK$GL_MAXID referenced
in psect $LOCK_LINKAGE_2 offset %X000000B0
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol EXE$CVT_EPID_TO_PCB referenced
in psect $LOCK_LINKAGE_2 offset %X000000C0
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol SCH$GL_PCBVEC referenced
in psect $LOCK_LINKAGE_2 offset %X000000D0
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1
%LINK-W-USEUNDEF, undefined symbol LCK$GQ_IDTBL referenced
in psect $LOCK_LINKAGE_2 offset %X000000D8
in module BLOCKING_LOCK file OPS$DATA:[ROBERT]SHOWLOCK.OBJ;1

GAMMA_ROB$$$ set proc/priv=all
GAMMA_ROB$$$ showlock 20C026DA
%SYSTEM-F-ACCVIO, access violation, reason mask=00, virtual address=000000000000001C, PC=0000000000020850, PS=0000001B
%TRACE-F-TRACEBACK, symbolic stack dump follows
image module routine line rel PC abs PC
SHOWLOCK 0 0000000000020850 0000000000020850
0 FFFFFFFF80273ED4 FFFFFFFF80273ED4
GAMMA_ROB$$$

Re: Finding Record Locks

Volker Halle — Wed, 18 Oct 2006 02:53:56 GMT

Rob,

this is a privileged program, which needs to link to the executive:

$ LINK/SYSEXE showlock

Note that you will need CMKRNL and that this program might then be able to crash your system.

Volker.

Re: Finding Record Locks

Robert Atkinson — Wed, 18 Oct 2006 03:23:28 GMT

Thanks Volker, that works a treat now :-

GAMMA_ROB$$$ showlock
pid> 20C032E6

PID 20C032E6 belongs to process RAA_RobertAtki3 owned by user RAA
which has requested a PW mode lock with LKID 090060E2.

LKID: 090060E2
Resource name (ascii) ....R...
Resource name (hex) 00000000 00000000 00000052 00000001
00000000 00000000 00000000 00000000
Parent Resource name (ascii) RMS$...Ã ...ROBERT ...
Parent Resource name (hex) 5245424F 52020000 CA880015 24534D52
00000000 00000000 00202020 20202054

This lock request is currently blocked by a PW mode lock held by
process RAA_RobertAtki2 with PID 20C026DA and owned by user RAA.

I'll give it a go in a real situation and see if it works.

Rob.

Re: Finding Record Locks

Volker Halle — Wed, 18 Oct 2006 03:23:57 GMT

Rob,

let me add an advice:

If you get undefined symbols during a LINK operation, NEVER try to run the resulting image with CMKRNL (or CMEXEC) privileges enabled.

ACCVIOs, which will result when the code tries to reference those undefined symbol locations in KERNEL mode, will almost certainly cause a system crash.

In your case, the program died with ACCVIO before reaching kernel mode, so you were lucky.

Volker.

Re: Finding Record Locks

Robert Atkinson — Wed, 18 Oct 2006 03:27:24 GMT

Thanks for the head-up.

I never get involved with low-level languages, just stuff like PHP, so this one's certainly news to me.

Glad I ran it on our test node now!

Rob.

Re: Finding Record Locks

Jim_McKinney — Wed, 18 Oct 2006 06:27:55 GMT

> ACCVIOs, which will result when the code tries to reference those undefined symbol locations in KERNEL mode, will almost certainly cause a system crash.

To trap this sort of thing, I included an exception handler that is active in kernel mode, in the port of Bruce's program.