Synchronization techniques with /reentrancy

[Travis Craig]

This is to continue the discussion related to /reentrancy that we had before and is a copy of the last entry I made there.

Thinking about the types of synchronization/protection, I see three types listed among the reentrancy options:

1. Disabling AST's.
2. DECthreads (pthreads) locking.
3. __TESTBITSSI builtin (test bit set and set, interlocked).

I understand the use of the first one.

For the option that uses the pthreads locking, it says it never disables AST's. I wonder how that can work because it seems possible that a thread would hold the lock and be interrupted by an AST that tries to get the lock, resulting in a deadlock. (Unless no AST routine calls the C RTL.)

The third option can be used with or without disabling AST's. The documents say to use it with disabling of AST's if any AST routines can call the C RTL. I'm assuming that it is a spinlock, though, which seems like a problem in a multi-threaded process. It seems like a performance problem if the threads have time slices and a deadlock problem otherwise. The example I see has thread A in the critical section when thread B tries to enter. Thread B spins on the lock until A exits the section. On a single processor, B could spin forever. Or, with time slicing, it would spin until its time runs out. Then A would run until either its time runs out. They would alternate back and forth like that until A exits the section. The next time B runs, it will enter the section. Without time slices, B could spin forever and A would never exit the section. Do I have something wrong here (other than having a multithreaded process that has AST routines doing too much)?

--Travis

[Robert Gezelter]

Travis,

There is a fourth possibility: the use of AST state as a "Switch to System Stack" analog (the name is from the old RSX and early OpenVMS days). If you are in non-AST state, and you fire an AST (e.g., using SYS$DCLAST), the AST will execute before you get back control. It provides a simplistic locking mechanism, but it is sufficient in many cases.

I may be misunderstanding your question, but the use of TESTBITSSI would seem to be a rare failure to gain access situation. Generally, critical sections should be only occupied for an extremely short period, and should not include a LOCK possibility. Spinlocks are best used for very short term locking, matters of, oh, say a relative handful of instructions.

It is hard to talk about the details without being familiar with the internal workings of the code. I have seen many cases where code was either incorrectly crafted, or, alternatively very carefully crafter to avoid synchronization problems. On some occasions, it has been easy to fail to appreciate the care exercised (In one case of which I am aware, the synchronization problem was missed for several years, until somebody noticed the problem while diagramming the algorithm on a blackboard).

- Bob Gezelter, http://www.rlgsc.com

[Travis Craig]

Bob,

Thanks for your reply and I'm sorry I keep being dragged off of my work on this question. I haven't had time to follow up on one of your suggestions (looking at the RTL source code), but here are my thoughts so far.

I agree that declaring an AST is a way to keep what would otherwise be mainline code from being interrupted by another AST. I don't see any indication that they use this technique in the RTL, though, just the three that I mentioned. I suspect they use disabling of AST's because it works whether the RTL code is called from the mainline or AST level. If their code declares an AST from AST level, won't it cause that work to be done too late, after the RTL routine has returned to the caller and the caller has returned to mainline? Also, while declaring an AST can prevent interference from another AST, it won't protect against another pthread, so they still need a pthread lock too.

As for the TESTBITSSI, I agree that it will rarely find the lock bit already set. When it does though, in the cases I mentioned, the act of spinning will cause the lock holder to wait a long time (time slice) or forever before executing any more instructions. It won't matter at that point how short its critical section is.

--Travis

[John Gillings]

Travis,

It's always possible to get threads into deadlocked states - the trick is to avoid them!

Be careful using __TESTBITSSI spinlocks - you need to equalise priorities before attempting to acquire the spin lock. Otherwise a high priority process attempting to acquire a lock held by a low priority process will spin forever. All it takes is one misguided SET PROCESS/PRIO command and you're deadlocked!

Your points about ASTs are correct. That's why it's always recommended that you don't mix threads and ASTs. At AST level you have to be very careful to make sure you're not locking anything that could already be locked by the non-AST thread.

Things get even more complex in inner mode where you have spinlocks and IPL to contend with.