Use of /reentrancy qualifier for C, CXX, and Fortran

[Travis Craig]

I'm looking at the /reentrancy qualifier for compiling with C, CXX, or Fortran. I find the descriptions of the qualifier confusing and hope somebody has figured it all out.

Basically, I have multithreaded processes that use AST's and could be calling run-time library routines from most anywhere. I would like the typical run-time library routine to protect me from both AST's and multiple threads. If it is doing something that can't be interrupted by an AST, it should disable AST's during the critical code. It should do the same if it can be interrupted by an AST but cannot withstand having the AST call into the run-time library or into the same RTL routine. It should take the same approach concerning multiple threads. If it is doing something that cannot withstand having another thread call into the library or into the same routine, then it should get an interthread lock to prevent reentrance.

It appears that each /reentrancy qualifier allows either AST-safety or thread-safety, but not both. Am I reading it wrong?

--Travis

[Robert Gezelter · Accepted solution]

Travis,

I don't have the C/C++ runtime manual with me at the moment, so I will suggest that you check them carefully (and, as your profile motto says, verify) exactly what happens with the /REENTRANCY switch.

If, the switch prefixes the library entry point names, that is one thing. Beyond that, there are more subtle issues. I do not have the time at this instant, but examples of the problems that can occur:

- Even with the synchronization, files should be used from a consistent state (AST, thread, or regular level) mixing references, while it may not cause the RTL to have problems, can cause strange output.

- The synchronization as described in the HELP text is purely for the safety of the RTL routines, it does not imply anything about your program.

If you are writing your own code, then there are two comments that I have found useful:
- Avoid the use of static variables in C/C++.
- I also recommend against the use of enabling/disabling ASTs as a technique. It is far safer to use the DCLAST service to throw an AST where interlocking is required, than enabling and disabling ASTs (and having a user level example of the classic O/S bug of disabling interrupts and taking a code path that fails to re-enable interrupts).

Some of my Encompass/DECUS presentations may be helpful, see "Events and Treads" (http://www.rlgsc.com/hpets/2002/1228.html ) and "Introduction to AST Programming" (http://www.rlgsc.com/cets/2000/435.html )

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com

[John Gillings]

Travis,

> It appears that each /reentrancy qualifier allows either AST-safety or thread-safety, but not both. Am I reading it wrong?

Yes, that's right. A general principle is you shouldn't mix ASTs and threaded code. They're two different, and sometimes incompatible, ways to do essentially the same thing. Pick the one that most suits your application and use it.

It gets a bit tricky when you want to use two indendent subsystems, one that uses ASTs, the other using threads. However, as long as the threading is "internal" to the subsystem, it's unlikely to cause any difficulties. You just need to make sure you don't create multiple AST threads that use the same DECthreads subsystem or DECthreads that use the same AST dependent subsystem.

In general, RTL routines are AST reentrant unless documented otherwise. For example, SMG is specifically documented as NOT AST reentrant - all that means is you should not allow an SMG routine to be interrupted by an AST routine that calls another SMG routine.

The real problem with AST or thread interrupts is where there are two threads which read/modify/write the same static storage location. In languages like C local variables are kept on the stack, and so are "safe". In FORTRAN local variables are static and therefore UNSAFE. You can play with locks or $SETAST to control access to local variables but in FORTRAN compiled with /REENTRANCY=NONE argument lists are static. So, if routine A calls routine B, passing an argument list (W,X,Y,Z). Suppose A is building the argument list and has setup W and X when it's interrupted by AST routine C. If C then calls A, which then sets up *different* values for the argument list to call B. AST level B and A complete, returning to A, which continues setting up the argument list, BUT the arguments W and X have changed.

I've seen exactly this scenario a few times. It's extremely difficult to debug because it's not reproducible. It's also difficult to code around because FORTRAN is entitled to build its argument lists any way it likes, so you can't be certain what block of code to protect. Compiling with /RECURSIVE/REENTRANCY=AST (BOTH are requried!) will cause FORTRAN to build argument lists on the stack, thereby plugging this hole.

If I wanted to write thread and AST safe code, I probably would NOT pick FORTRAN as it has many features that make implicit assumptions that storage is static. So does C to some extent (the dreaded errno!).

Remember that reentrancy is a DATA issue, not a CODE issue. You need to make sure that anything that performs a read/modify/write is properly interlocked. Examine your listing files to make sure you know where everything is allocated. Stuff on the stack is OK.

Heavy weapons like $SETAST tend to unnecessarily impact performance and should be avoided.

[Ian Miller.]

apart from the excellent points so far the $SETAST system service can take a surprising amount of time to execute so don't call it frequently. Ensure access is syncronised to data as bad things can happen otherwise. Compiler behaviour can change with compiler or vms or RTL upgrades.

There is a noshare modifier in C which is worth reading about.

[Travis Craig]

Thanks for all the replies. I guess the answer is that I can't use the compiler /reentrancy qualifier to get the run-time libraries to protect me from AST's and pthreads at the same time.

A little more background for those who are interested. Our oldest code did most of its event-handling work at AST level to use the implicit protection against preemption. Then we made a big change to a work queue system to let us send all events to mainline level, where they are dispatched in sequence, with no preemption. That was to adapt our software to UNIX/Linux and Windows. On the Alpha, we used the interlocking queue instructions to implement the work queue and the corresponding free queue.

Then people began to multithread some of our apps. Many of our own library routines were modified to use a pthread mutex (on OpenVMS and UNIX) to protect their data structures in the multithreaded environment. Unfortunately, we still have some of our oldest OpenVMS code that does more at AST level than simply queue the event to the mainline level. As soon as it called one of the protected routines, it could deadlock by waiting for a mutex that was being held by the same thread at mainline level (as I recall). In any case, I changed the lock code to disable AST's before getting the mutex and to reenable them after releasing it. It does require that the same function that locks the mutex must unlock it, risking the problem that Bob mentioned.

The nice thing about out lock, though, is that it protects against both AST's and pthreads, as I wish the C RTL would do. I wish that because that same AST code that does non-trivial stuff can call C RTL routines as well as the protected ones in our own library. With OpenVMS use ending (at least on our systems), we don't spend any time redesigning the old code.

I'm still going to have the think some about our Fortran code. I don't know whether any of it is involved in AST's anymore or not, but I'll give those qualifiers some more thought.

I was worried about the performance effects of using SETAST a lot, but the one time I tested the effect, I didn't see a problem. In fact, it seemed that things ran faster. I wonder whether I made a mistake in that test.

--Travis

[Travis Craig]

And I forgot to mention that the /reentrancy qualifier doesn't seem to prefix function names. Instead, it causes a call from the main routine to decc$set_reentrancy, presumably passing it an argument that tells which type of synchronization/protection to use.

Thinking about the types of synchronization/protection, I see three types listed among the reentrancy options:

1. Disabling AST's.
2. DECthreads (pthreads) locking.
3. __TESTBITSSI builtin (test bit set and set, interlocked).

I understand the use of the first one.

For the option that uses the pthreads locking, it says it never disables AST's. I wonder how that can work because it seems possible that a thread would hold the lock and be interrupted by an AST that tries to get the lock, resulting in a deadlock. (Unless no AST routine calls the C RTL.)

The third option can be used with or without disabling AST's. The documents say to use it with disabling of AST's if any AST routines can call the C RTL. I'm assuming that it is a spinlock, though, which seems like a problem in a multi-threaded process. It seems like a performance problem if the threads have time slices and a deadlock problem otherwise. The example I see has thread A in the critical section when thread B tries to enter. Thread B spins on the lock until A exits the section. On a single processor, B could spin forever. Or, with time slicing, it would spin until its time runs out. Then A would run until either its time runs out. They would alternate back and forth like that until A exits the section. The next time B runs, it will enter the section. Without time slices, B could spin forever and A would never exit the section. Do I have something wrong here (other than having a multithreaded process that has AST routines doing too much)?

--Travis

[Travis Craig]

I'm going to start a new thread with my previous entry, because I'm afraid I shut down discussion here by having the rabbit come out of the hat.