topic Re: RWCLU in Operating System - OpenVMS

RWCLU

Wim Van den Wyngaert — Thu, 24 Mar 2005 02:34:25 GMT

Yesterday we had a development cluster (with 6 system disk less stations in it) that had processes in RWCLU. All login trials went into RWCLU. All active processes too (not sure of the word all). I still had an open sesssion and could do a show system and a search. The search worked but eventually blocked too.

This took over 15 minutes before I restarted the cluster (I didn't take a system dump, yes I was stupid).

The restart was done for 1 server node but it got a fatal bugcheck in the lock manager.
Then I restarted both nodes. The stations didn't reboot.

I checked the operator log file and found network interruptions but not between the 2 server nodes.

I know RWCLU is notmal an indication of lock remastering. Performance advisor didn't find any anomalies.

Anyone an idea of what happened ?

VMS 7.3 not fully patched on Alphaserver 4100.

Wim

Re: RWCLU

Volker Halle — Thu, 24 Mar 2005 02:47:20 GMT

Wim,

RWCLU is also used during cluster state transitions.

What do you mean with 'network interruptions' ? Connection lost ?

Would you want to post the CLUE file from the lock manager crash ?

Volker.

Re: RWCLU

Wim Van den Wyngaert — Thu, 24 Mar 2005 02:58:48 GMT

Volker,

The operator log file only mentioned "node station lost connection to station or server".

Wim

Re: RWCLU

Volker Halle — Thu, 24 Mar 2005 03:26:35 GMT

Wim,

so you had network problems...

The following OPCOM message will indicate, that NODEA did NOT receive a cluster hello (multicast) message from NODEB since about 8-10 seconds:

Node NODEA (csid 00010074) lost connection to node NODEB

Each node in the cluster sends a multicast-message to all other cluster-members (using a MC address based on the cluster group number) every 3 seconds.

If the problem is intermittent, once the next hello message is received from that node, the following OPCOM message is shown:

Node NODEA (csid 00010074) re-established connection to node NODEB

If no hello message is received from NODEB for more than RECNXINTERVAL seconds, the following message is shown:

Node NODEA (csid 00010074) timed-out lost connection to node NODEB

and NODEB is removed from the cluster by NODEA.

While in this state, any process trying to communicate with the lock manager on a remote node may be put in RWCLU.

Volker.

Re: RWCLU

Volker Halle — Thu, 24 Mar 2005 03:40:53 GMT

Wim,

the LOCKMGRERR crash of node SALPV2 indicates that a bad/corrupted LOCK message has been received from node ALM12. That node should have crashed as well (with LOCKMGRERR) - as indicated by R0=0000223C %SYSTEM-F-NODELEAVE.

This type of crash can happen, when messages are being corrupted (either in the sending or the receiving node OR on the 'wire' i.e. LAN).

Together with your network problems (connection lost), this is another indication of a possible network problem.

Volker.

Re: RWCLU

Wim Van den Wyngaert — Thu, 24 Mar 2005 03:58:09 GMT

Volker,

You're hot. ALM12 did reboot. But it is the only station that rebooted.

At 15:57:30 there were PEA device errors on SALPV1 (the server). In operator log there were "lost connection" messages.
At 15:59:30 my monitoring found processes in RWCLU (a batch job and a RSH to start with).
At 16:00:30 DSM gave circuit timeout messages
At 16:05: I saw the problem.
At 16:12:30 I rebooted the server.

To survive network outages recnxinterval is on 900 seconds, so 15 minutes. I guess that I rebooted the system just to fast. The problem would have solved itself a little bit later.

What I don't understand :
1) search is also locking local files. Why did it work until a certain level ? Cache ?
2) why I never get this with network outages during the night ? There are batch jobs all the time ! Outages too short ?

Wim

Re: RWCLU

Volker Halle — Thu, 24 Mar 2005 04:34:54 GMT

Wim,

I'm assuming that you'll get the RWCLU state, if the lock request somehow involves an operation with a remote node, to which the local node has lost connection.

This would highly depend on which resources your processes touch at that time.

Volker.

Re: RWCLU

Wim Van den Wyngaert — Thu, 24 Mar 2005 08:30:02 GMT

Only the 2 cluster servers have a non-zero LOCKDIRWT. And the network between them was not interrupted. So, why did I get RWCLU on login ? A lock remastering taking almost 15 minutes ?

Re: RWCLU

Ian Miller. — Thu, 24 Mar 2005 11:59:26 GMT

if only two nodes have nonzero lockdirwt then they will be the lock directory nodes. Other nodes talk to them to find out who is the master node for a particular node. Parhaps, due to your network problems, the lock directory lookup operation was what the processes in RWCLU where waiting for.

Re: RWCLU

Wim Van den Wyngaert — Thu, 24 Mar 2005 13:28:05 GMT

Today the same thing happened.
But I didn't reboot and it freed itself without intervention.
A number of cluster stations where powered off.
The cluster was locked for 15 minutes, as expected.

Re: RWCLU

Volker Halle — Thu, 24 Mar 2005 13:44:27 GMT

Wim,

yes, if you power off a satellite without proper shutdown, connections to this node will be lost and it will take RECNXINTERVAL seconds until the node is timed out and removed from the cluster.

Once the node is removed from the cluster, everything (e.g. processes put in RWCLU) will continue.

It's still not 100% clear to me, which kind of LOCK/RESOURCE operation in this scenario causes a process to be put in RWCLU.

Volker.

Re: RWCLU

Volker Halle — Fri, 25 Mar 2005 04:48:58 GMT

Wim,

one way for getting into the RWCLU state (RSN$_CLUSTRAN) is when a cluster state transtition has been started and locks on the local node are being stalled (LCK$GL_STALLREQS .ne. 0). But a cluster state transition is not started until the connection to the remote node times out (which is not yet true in your case).

If the connection is lost (but not yet timed-out) and a process is involved in a lock operation, which needs to send a lock request to the remote node (to which the connection has been lost), the process is put in RWSCS.

So the only remaining scenario I can think of (for entering RWCLU), is that a remastering operation may happen (between your 2 servers), which also involves a lock on the remote node.

Volker.

Re: RWCLU

Wim Van den Wyngaert — Tue, 29 Mar 2005 04:42:20 GMT

Lockdirwt is on 0 on all stations, 1 on the servers.

I would expect that the lock manager stayed on the 2 servers but now I notice that ana/sys show lock/sum results in moves of the lm TO the stations and this very frequently (about every minute a move is done).

???

Wim

Re: RWCLU

Volker Halle — Tue, 29 Mar 2005 04:50:31 GMT

Wim,

MONI RLOCK (or SDA> SHOW LOCK/SUMM) will list the 3 reasons of lock tree outbound movements:

Tree moved due to higher Activity
Tree moved due to higher LOCKDIRWT
Tree moved due to Single Node Locks

What does it say on your servers and satellites ?

Volker.

Re: RWCLU

Wim Van den Wyngaert — Tue, 29 Mar 2005 04:55:40 GMT

Servers : 70% due to higher activity, 0% due to higher lockdirwt, 30% due to single node locks.

Stations : 30% due to higher activity, 3% due to higher lockdirwt, 40% due to single node locks. On another station I found 70% due to lockdirwt.

Wim

Re: RWCLU

Volker Halle — Tue, 29 Mar 2005 06:11:34 GMT

Wim,

lock remastering involves individual RESOURCE TREES (i.e. a root resource and it's sub-resources) - depending on lock activity in this tree on the different nodes in a cluster.

Depending on your version of OpenVMS Alpha, there will be a SYS$SHARE:LCK$SDA SDA extension with various interesting commands (SDA> LCK provides some basic help).

SDA> LCK SHOW ACTIVE will show resource tress with lock activity

SDA> LCK STAT/NOALL/TREE/TOPTREE=n will display the n most active lock trees

Now as you see the amount of lock remastering in your cluster, one of the possible scenarios for RWCLU (remastering a tree which includes a lock on a machine, which has lost connection to the cluster), becomes more plausible.

If you really want to find out about the reason of a process going into RWCLU, force a crash with a process in that state and then we'll find out...

Volker.

Re: RWCLU

Wim Van den Wyngaert — Tue, 29 Mar 2005 09:57:54 GMT

Still not clear after a lot of reading.

1) Why do stations receive management of a resource while lockdirwt is 0 ?

2) Is there no way to see what resource is exactly remastered instead of statistics ? I'm afraid it is something ordinary such as the sysuaf.

3) How can I see how many bandwith is eaten by the remastering ? As I understand it, the packets exchanged can be very big and I only can see the number of packets (=messages).

3) How many remasterings per minute is normal ? I see up to 30 remasterings per minute on my GS160 (running Sybase and DSM, DSM in cluster config).

And Volker, I can not ask for a crash. I have to wait ...

Wim

Re: RWCLU

Volker Halle — Tue, 29 Mar 2005 12:56:12 GMT

Wim,

1) sole interest. If locks on this resource tree only exist on this station

2) doesn't SDA> LCK SHOW ACTIVE at least show the most active resource trees to give you an idea about the resources involved.

3) Lock re-mastering is a trade-off between on-going remote lock messages versus moving a resource tree once to the most active node and thereby trying to increase local locking.
Heavy lock remastering will increase Interrupt Stack Time.

4) You can limit the maximum size of a resource tree being moved by setting PE1 = n
PE1=-1 will disabled remastering.

Volker.

Re: RWCLU

Wim Van den Wyngaert — Wed, 30 Mar 2005 01:21:14 GMT

Volker,

I don't agree.

On 1) : I found 1200 trees moved to a station in 6 days. How can a lock move a tree TO the station if it is the only one locking the resource ?

On 2) : HP implemented lots of show commands (LCK...) but not the most important one : monitoring of remastering in detail.
SDA>show remastering
08:01:01.12 resource xxx moved from node aaa (bbb requests in 8sec) to node ccc(ddd requests in 8sec). Moved yyy K in zzz sec.

On 3) : HP shows the number of messages. What importance does it have it the size is between 0 and 64 K ? Shouldn't the number of MB be shown ?

On 4) : this could be suicide. I should have a statistic of all resources with their size and number of remasterings before I could set this PE1. Better would have been a set file command to modify 1 resource setting.

5) Machines are getting quicker all the time. Why is are the parameters for remastering hardcoded ? This is very un-VMS.

Wim

Re: RWCLU

Volker Halle — Fri, 01 Apr 2005 09:10:58 GMT

Wim,

re: 1) if a resource tree is first used on multiple nodes and then only generates activity on one stations, it should move there, shouldn't it ?

re: others - I'll try to ask these questions during the OpenVMS Bootcamp in June.

Volker.