HPE Community read-only access December 15, 2018
This is a maintenance upgrade. You will be able to read articles and posts, but not post or reply.
Hours:
Dec 15, 4:00 am to 10:00 am UTC
Dec 14, 10:00 pm CST to Dec 15, 4:00 am CST
Dec 14, 8:00 pm PST to Dec 15, 2:00 am PST
Operating System - OpenVMS
cancel
Showing results for 
Search instead for 
Did you mean: 

About use of RAB$V_NQL No Query Locking

 
SOLVED
Go to solution
Jose Antonio Rocha
Occasional Advisor

About use of RAB$V_NQL No Query Locking

Hello all

I have a customer that uses RMS indexed files only, he has a lot of aplications that access these files using a CLUSTER of 5 Alphaservers. The locking is very hign and many aplications only READ the files, I need an small example in BASIC or FORTRAN in order to access this files only to read and does not make a call to the Lock Manager in order to reduce the LOCK and permit that other processes can access the same file to READ, MODIFY or WRITE.
15 REPLIES
Robert Gezelter
Honored Contributor

Re: About use of RAB$V_NQL No Query Locking

Jose,

I would hazard a guess that it is desired to open the file for read-only, not actually ignore locking.

For a start, see the FORTRAN Programmer's Guide. In particular, the OPEN statement has a "READONLY" option.

As far as Lock Manager performance, what are the actual locking statistics, and what is the scale of the hardware (e.g., which processors are involved in the cluster).

- Bob Gezelter, http://www.rlgsc.com
Jess Goodman
Esteemed Contributor

Re: About use of RAB$V_NQL No Query Locking

Jose,

We find that it is very efficient and safe to turn off all RMS record locking for programs that open indexed files READONLY, SHARED. Note that file locking and RMS bucket locking will still be in effect.

I have attached a Fortran routine, SET_RAB_ROP, that I wrote for this and similar purposes. You can use this routine in your READONLY, SHARED applications.
After your program open the file using, for example, the logical unit in variable LUN, call my routine like this:

INCLUDE '($RABDEF)'
CALL SET_RAB_ROP( LUN, RAB$M_RRL.OR.RAB$M_NLK, , )

Then before running your applications you should disable query locking. Easiest way is with:
$ SET RMS_DEFAULT/QUERY_LOCK=DISABLE [/SYSTEM]

This option was added in VMS 7.2-1H1 and it is perfectly safe, even with /SYSTEM, unless you have applications that check for the RMS alternate success status values RMS$_OK_RLK or RMS$_OK_RRL (which you would never even see anyway if you are using Fortran READs).

This approach will stop the default Fortran behavior of returning record-locked errors. This assumes you are ok with reading the record as is, instead of waiting for an in-progress update to it to complete (which you could do by calling SET_RAB_ROP with option RAB$M_WAT).
I have one, but it's personal.
Hein van den Heuvel
Honored Contributor
Solution

Re: About use of RAB$V_NQL No Query Locking

Jose,

RAB$V_NQL (No Query Locking) is easy enough. Just just flip (set) that bit and the RECORD locks dissappear.

In BASIC and FORTRAN you'd need a useropen.
The key lines for BASIC would be:

FUNCTION LONG MY_USEROPEN (FABDEF FAB, RABDEF RAB, LONG CHANNEL)
%INCLUDE "$RABDEF" %FROM %LIBRARY "SYS$LIBRARY:BASIC$STARLET"
:
RAB::RAB$W_ROP_2 = RAB$M_NQL
:
SYS$CONNECT(RAB)

However, I suspect that the effect will NOT be what you expect. It would only provide a small, but perhaps still useful reduction in lock activity.
The reason for this is that bucket lock activity is AT LEAST two time record lock activity and often 3 or 5 times!

IF you believe RMS locks are hurting, THEN you may want to review basic RMS file tuning as a first step (Index depth, Global buffers).

To stop all locking you can open the file READONLY, like Robert indicates, but you must specify (in RMS terms) FAC=GET, SHR=GET
So NO updates allowed.
Again I suspect that this is NOT the effect you intent. You probably want to allow (sporadic) updates, but just want the readers not to cause lock activity.

Sadly, RMS does not provide that option. You might think that SHR=UPI would accomplish this, but that's ignored. The reasoning for this is the possibility of partial bucket writes (notably with fragmentation in play) and that DEFERRED write by an updater _could_ send a reader into 'la-la-land' without interlocking. It's really too bad RMS does not provide this somehow, for 'parameter' or 'lookup table' style files. Or even for say a customer information database. If an application is only tasked to fetch a phone number then it will often be an acceptable risk to get the old or new version if it happens to be updated right there & then. But you would not want it to return total garbage or start spinning due to an other record being inserted or updated. Right?

Note... In BASIC, you might think that specifying "access read, allow read" is good enough. NOT so. BASIC gives you FAC=MSE sharing for free, whether you like it or not, to support a (future) "CONNECT" open clause.
Again you would need a useropen to disable (all) locking:
%INCLUDE "$FABDEF" %FROM %LIBRARY "SYS$LIBRARY:BASIC$STARLET"
:
FAB::FAB$B_SHR = 0
:
SYS$OPEN(FAB)
:

Jess,
The subject indicated that Jose is aware of NQL. If you can get to NQL, then IMHO that is much preferable over the (nice) HACK to transform NLK+RRL to NQL _if_ the right process context is active.


Here are some number examples using my RMS_STATS tool to get RMS locking data as alternative to MONI RMS/ITEM=LOCK. Test program below.

Test file generation:

$ perl -e "printf qq(%06dtest\n),$_ for (0 .. 9999)" > x.seq
$ conv/fdl="fil; fil yes; org ind; key 0; seg0_l 10" x.seq x.idx

First test... just read the file:
$ conv/share/fdl=nl: x.idx nl: ! Use _NQL
Local buffer: 186 187 20186
Data record: 0 0

Now with test program using 10,000 pseudo random gets.

Local buffer: 9855 9856 50145
Data record: 10000 10000

Now TUNE the file (very coarsely!)

$conv/fdl="fil; fil yes; org ind; are 0; buc 63;key 0; seg0_l 10" x.seq x.idx

Convert:
Local buffer: 9 10 20009
Data record: 0 0

Test:
Local buffer: 3265 3266 36735
Data record: 10000 10000

Test + SHR=0 (and FAC=GET)
Local buffer: 0 0
Data record: 0 0

Test + NQL=1
Local buffer: 7 8 39993
Data record: 0 0


With Global Buffer:

Local buffer: 0 1 0
Global buffer: 14 7 0
Global section: 2 1 4
Data record: 0 0 0


Your mileage can and WILL vary!
Hope this help some already,
Send Email for professional help.
Regards
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting

FUNCTION LONG MY_USEROPEN (FABDEF FAB, RABDEF RAB, LONG CHANNEL)
OPTION TYPE = EXPLICIT, CONSTANT TYPE = INTEGER
%INCLUDE "$FABDEF" %FROM %LIBRARY "SYS$LIBRARY:BASIC$STARLET"
%INCLUDE "$RABDEF" %FROM %LIBRARY "SYS$LIBRARY:BASIC$STARLET"
%INCLUDE "$RMSDEF" %FROM %LIBRARY "SYS$LIBRARY:BASIC$STARLET"
EXTERNAL LONG FUNCTION SYS$OPEN, SYS$CREATE, SYS$CONNECT
DECLARE LONG MY_STATUS

! FAB::FAB$B_SHR = FAB$M_UPI OR FAB$M_UPD
RAB::RAB$W_ROP_2 = RAB$M_NQL
MY_STATUS = SYS$OPEN(FAB)
MY_STATUS = SYS$CONNECT(RAB) IF (MY_STATUS AND 1) = 1
MY_USEROPEN = MY_STATUS
END FUNCTION

option type = explicit, size = integer long, constant type = integer

call lib$init_timer()
declare integer i
open "x.idx" for input as file #1, indexed, useropen MY_USEROPEN, allow modify, access read, map test, recordtype any
map (test) string test=50
for i = 1 to 10000 step 1
get #1, key #0 eq format$( int(10000*rnd), "<0>#####")
next i
end





Steve Reece_3
Trusted Contributor

Re: About use of RAB$V_NQL No Query Locking

Jose,

If all processes are read only on the files then the SET RMS/QUERY=DISABLE/SYSTEM may be quite effective. However, if you're writing then it may not be quite so good.

An application that I was involved with did shedloads of locking on read-only files. Modifying global buffer settings on the indexed files and disabling query locking were quite effective. In that, though, we had processes reading only - the updates were done overnight when there were no live processes on the systems. This meant that there was no writing during the day at all.

If you have a support contract with HP then they may be able to assist in the diagnosis of your locking issues and help with the solution too.

Bear in mind too that XFC isn't cluster coherent - if two nodes are using the same file then XFC stops caching the file until there's only one node accessing the file again. This means that your IO will probably need to go back to the physical disk device (whether SCSI or Fibrechannel connected) which can also have a significant impact upon application performance. PerfectCache is better, but at the expense (I suspect) of more lock activity. PC doesn't let you use FastPath either which can impact performance too on multiprocessor systems.

Steve
Hein van den Heuvel
Honored Contributor

Re: About use of RAB$V_NQL No Query Locking

Thanks Steve for sharing the hands on experiences!
But I do have some notes.

Steve wrote>> If all processes are read only on the files then the SET RMS/QUERY=DISABLE/SYSTEM may be quite effective.

- It still only effects Record locks.
- NQL works equally well whether others are locking/writing or not.
- The SET RMS/QUERY thingie is ONLY useful for application which already tell RMS to use RRL (Read Regardless) as well as NLK (do Not Lock Keep). BASIC does not give you that without useropen. COBOL gives that with READ REGARDLESS when opened with APPLY LOCK HOLDING.

>> However, if you're writing then it may not be quite so good.

- That's a little confusing. It can never hurt. It does not apply to a program updating/deleting as that would need a lock.

Steve>> Modifying global buffer settings on the indexed files and disabling query locking were quite effective.

Yes!

>> In that, though, we had processes reading only - the updates were done overnight when there were no live processes on the systems.

Irrelevant... unless the application goes 100% readonly for all processes, disallowing writers on the first open.

>> If you have a support contract with HP then they may be able to assist in the diagnosis of your locking issues and help with the solution too.

They may, and you should ask ni case they do. But IMHO it is an HP - or otherwise - Consulting question, not a support issues.

>> Bear in mind too that XFC isn't cluster coherent

It _IS_ cluster coherent, but it implements that by often not being effective.

>> - if two nodes are using the same file then XFC stops caching the file until there's only one node accessing the file again.

NOT correct. The VCC stopped. The XFC actually tolerates concurrent write-allowed opens. It just dos not tolerate concurrent write operations from multiple nodes. It has a timing mechanism to re-try caching after the cache was flushed due to a write on an other node.

>> PerfectCache is better, but at the expense (I suspect) of more lock activity. PC doesn't let you use FastPath either which can impact performance too on multiprocessor systems.

Correct. IOexpress from networkingdynamics is alos an LBN cache.

If anyone has recent (less than 10 years old :-) benchmark / comparision material for XFC vs an LBN cache, then I'd love to hear, either here or privately ( non-disclosure as needed ).

Best regards,
Hein van den Heuvel
Jess Goodman
Esteemed Contributor

Re: About use of RAB$V_NQL No Query Locking

Hein said:

"If you can get to NQL, then IMHO that is much preferable over the (nice) HACK to transform NLK+RRL to NQL _if_ the right process context is active."

Setting the RAB$M_NQL bit in RAB$W_ROP_2 could certainly be called a more direct method, but the method in my example has the advantage of being backwardly compatible - the code can be compiled with older versions of Fortran/FORSYSDEF and run on older versions of VMS that know nothing about the RAB$W_ROP_2 field.

I would also disagree that setting RAB$M_NLK and RAB$M_RRL, along with using $ SET RMS_DEFAULT/QUERY_LOCKING=DISABLE is a "hack". In the VMS documentation on no-query locking it is the very first method
listed.
I have one, but it's personal.
Jose Antonio Rocha
Occasional Advisor

Re: About use of RAB$V_NQL No Query Locking

Robert

The amount of locks is about 320,000 per second, It is very high.

The cluster are 5 Alpha, 4 ES80 with 8 CPU and 32 GB of memory and 1 GS1260 with 16 CPU and 54 GB, OpenVMS 7.3-2

Thanks for your quicky answer.

Jose
Hein van den Heuvel
Honored Contributor

Re: About use of RAB$V_NQL No Query Locking

>> The amount of locks is about 320,000 per second, It is very high.

Not if the system is doing 100,000 transactions per second

Ok, the lock rate is high. So what
Maybe the system has nothing better to do?
Are the end users happy?

:-).

Seriously...


You have to figure out WHICH locks
- ANAL/SYSTEM , LCK and RMS extentions may help)

And uou have to put it in perspective as to what the application is doing.
RMS record locks are an easy target, but are they relevant?

First you may want to take a giant step back, perhaps working with a fresh, independent, consulting resources to figure out what is the real problem (if any) that needs to be addressed.

Best regards,

Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting
Jose Antonio Rocha
Occasional Advisor

Re: About use of RAB$V_NQL No Query Locking

Hello to all

The information that you put here are very valuable and clarify me many thinks about OpenVMS LOCKS, the real situation is that during the night run a lot of batch processes, many of then them only read the files sequentially to generate reports and files to send to another aplications outside Alpha plataforms and others processes modifying data on the same files, but it run slow I think could be by the high generated locks.

The goal is try to avoid to call lock manager by the aplications that only reads sequentilly the files.

I send a picture with the amount of locks during 24 hours and a MONITOR DLOCK from the GS1260 machine.

Thanks in advance

Jose
Hein van den Heuvel
Honored Contributor

Re: About use of RAB$V_NQL No Query Locking

>> The information that you put here are very valuable and clarify me many thinks about OpenVMS LOCKS

Good!

>> the real situation is that during the night run a lot of batch processes, many of then them only read the files sequentially to generate reports and files to send to another aplications outside Alpha plataforms

Ok, so maybe you want to try completely different approaches.

- Use (Host-Based) shadowing to detach a member with a clone of a file.

- Use BACKUP/IGNORE=INTERLOCK to take a readonly snapshot of the file. If you have the memory, then stick the copy into a RAMdisk for transfer + delete?

- my (customers) preferred solution is to use the commercial Vselect tool (EGH company, John Santos and friends) to take a sequential file extract, with or without selection criteria. Vselect can read indexed file faster than RMS can, and uses no locks.

- write your own, or commission someone to create, an extract tool. That's not too hard for specific cases, but more tricky for a solid, commercial solution. (Check out DIX)

- tune the file for bigger data bucket. The reduces ENQ/DEQ proportinally, exchanging them for cheaper Converts to/from NL.

>> and others processes modifying data on the same files, but it run slow I think could be by the high generated locks.

Or because the XFC cache does not get a chance to help?

>> I send a picture with the amount of locks during 24 hours and a MONITOR DLOCK from the GS1260 machine.

Good stuff, but I'd prefer to see more quantitative words around 'it run slow'.
Because that seems to be the real problem you are trying to solve and you have not convinced me yet that locks are the cause for that.

Cheers!
Hein.
Jose Antonio Rocha
Occasional Advisor

Re: About use of RAB$V_NQL No Query Locking

Hein

Thanks for your contribution, I only want that you review the attached information, where in my modest opinion the LOCK is the principal cause of the low performance. Could you tell me if agree with this conclusion?

Thanks a lot for your attention.

Jose
Hein van den Heuvel
Honored Contributor

Re: About use of RAB$V_NQL No Query Locking

>> Thanks for your contribution, I only want that you review the attached information, where in my modest opinion the LOCK is the principal cause of the low performance.

I don't 'see' any low performance.
What is the (end user) measurement for that?
Is the performance good for most of the day, but acceptable during the nightly (transfer?) jobs?

The system is busy and has High MPsync at the time to took the snapshots, but the full day picture shows a relatively healthy system during most of the day. Lots of usermode, little kernel/mpsync overhead... except around midnight.

I'd recommend switching to the dedicated lock manager to reduce overhead and turn the cpu locking MPsyncs into low priority kernel mode waits.

Given the amount of memory, I would certainly consider how you could deploy read-only file clones.

And if extracts and selects are seen as not fast enough, and critical for the end user perception / SLA's then I would looks at alternatives for that, such as the commercial tool mentioned.

At this point this topic start to become too much like 'work' for me, and you exceeded your 'first 15 minutes are free' quota from me. Maybe someone else will step up, or contact me offline.

Best regards,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting



Jose Antonio Rocha
Occasional Advisor

Re: About use of RAB$V_NQL No Query Locking

Thanks a lot to all people who answers to this question, I will continue investigating about this issue at the customer site.

Regards

Jose
Hoff
Honored Contributor

Re: About use of RAB$V_NQL No Query Locking

> Given the amount of memory, I would certainly consider how you could deploy read-only file clones.

Cloning can be an option, as can be sharding the data. There are other approaches toward increasing performance and parallelism.

One of the general approaches here involves the DECset PCA or such tool, and determining where the application(s) are spending the most time. OpenVMS locking (particularly on recent OpenVMS versions) is pretty speedy. Excessive locking can point to some underlying application or file-level issue.

I've also seen the NQL knob used for an on-line archive; to work around some other application that's holding a lock for too long. For similar reasons around why BACKUP /IGNORE = INTERLOCK can produce questionable results, the use of NQL here can be problematic, too.
Hein van den Heuvel
Honored Contributor

Re: About use of RAB$V_NQL No Query Locking

I mentioned VSELECT from EGH as a commercial solution to extract indexed file records faster than RMS can, without locks.

Ther eis a freeware solution bin Fekko Stubbe called DIX.

The extract would roughly look like (good luck with the Dutch comments!):

$ DIX FILE/FAST file -
/DES=$LINE - !een descriptor met een veld char*nnnnnn
/OUTPUT=x.x -
/WIDTH=nn - !groot genoeg om het record te bevatten, ander gaat dix wrappen
/DISPLAY=(NOALL,DATA) - !alleen de data, niet de offset/veldnaam
/FORMAT=PASSALL - !Unprintables zo laten als het is (anders worden het .tjes)
zoekterm [zoekterm]

zoekterm = RSE = Record selection expression. Tons of options. See HELP.

http://www.oooovms.dyndns.org/dix/

Hein.