Operating System - OpenVMS
1748322 Members
4108 Online
108762 Solutions
New Discussion

Re: Posix Threads and ASTs

 
Berryman
Occasional Contributor

Posix Threads and ASTs

I'd like to request some clarification on how ASTs work within a threaded application.  For the following questions, please assume User mode ASTs, a multiprocessor system with MULTITHREAD set greater than one, and an application linked /THREADS_ENABLE.

 

1. In one place in the documentation, it states that AST queues have been moved from the PCB to Kernel Thread Blocks (KTB) so that each kernel thread can receive ASTs independently.  In another place, it states that there are both per-thread ASTs and process-wide ASTs and that process-wide ASTs are delivered only to the initial thread.  Finally, there is a blue note stating that "in all OpenVMS releases to date, all ASTs are delivered process-wide".

 

To me, the first statement implies that it is possible for there to be multiple, user-mode ASTs executing simultaneously.  The remaining statements seem to imply that, while things are moving in that direction, for right now all ASTs are queued to a single thread and still only executed serially.  My question is this: is it possible, either now or in the planned future, for there to be multiple ASTs executing simultaneously?

 

2. Does $SETAST disable AST delivery for the entire process or only for the calling thread (to me, the documentation is not clear)?  In my program there are a number of variables manipulated only by AST routines and there is one place in the code that must guarantee that no other AST operates while it is running because it must manipulate those variables in a non-atomic fashion.  I can either execute that code with a $DCLAST, assuming only one AST can be active at a time,  or disable ASTs if ASTs can operate in parallel.  Which do I need to do?

 

3. As currently written, the main() routine initializes and starts the appropriate number of threads then becomes a regular "worker bee" thread itself.  If ASTs are queued only (or mainly) to the initial thread, which I understand to mean the code just mentioned, would it make more sense for main() to end with the equivalent of "while (1) { sys$hiber(); }" and just let that thread handle ASTs?

 

4. Is there any way for the code to determine or control if an AST is going to be thread-specific or process-wide?

 

Thanks for any help or insights you can give,

 

Mark Berryman

10 REPLIES 10
Hoff
Honored Contributor

Re: Posix Threads and ASTs

If you have not already read this:  http://h71000.www7.hp.com/wizard/wiz_6099.html

Berryman
Occasional Contributor

Re: Posix Threads and ASTs

Thank you.  That was an informative article.  Assuming I have understood it correctly, the following are all true statements:

 

1. Regardless of the number of CPUs in a system or the number of threads operating in parallel, an application can assume that only one of its ASTs will be active at a time.

2. However, if multiple Kernel threads are enabled (there are multiple CPUs and MULTITHREAD is set greater than one) both mainline code and an AST can be simultaneously active.

3. If there is data that is manipulated only within an AST no other synchronization is required.  This means that said data can be manipulated with regular instructions, and interlocked instructions, such as BBSSI, are not needed.  (Although, in this particular application, the AST routines insert items in queues and the instructions used to do so are either INSQTIQ or INSQUEQ.  If non-interlocked versions of these commands were available on Alpha/IA64, they could be used.)

4. ASTs must severely limit what system calls they make and they must not make calls to thread routines.  In my case, they basically check status, put work on a queue, and requeue the I/O.  They might call SYS$QIO, SYS$CANCEL, SYS$GETMSG, SYS$FAOL_64 or LIB$PUT_OUTPUT and, within the CRTL, malloc/free and close.  Any thing else is placed on a queue for mainline processing.

 

Does this seem about right?

 

Hoff
Honored Contributor

Re: Posix Threads and ASTs

Seems right.

 

I'd consider using KP Threads and not POSIX threads, too.  KP Threads are comprised of the EXE$KP_START, EXE$KP_STALL_GENERAL, EXE$KP_RESTART, EXE$KP_END and friends.  (Yeah, user-mode EXE calls.)

 

This given that ASTs, INSQTIQ and INSQUEQ aren't particularly portable, either.

John Gillings
Honored Contributor

Re: Posix Threads and ASTs

Looks about right to me. Threads and ASTs do somewhat the same thing, but very differently. Both can be tricky to get right. Mixing them multiplies the complexity.

 

1) is correct

2) seems correct, and is slightly alarming! be careful.

3) is correct. Note that you can access the same data from non-AST code using $SETAST to disable and restore ASTs around the critical region (but I'm not sure if blocking ASTs in one thread applies to the whole process - worth an experiment to find out).

4) I don't know about "severely limit", but yes, you need careful consideration of anything you call from AST level. For example, you'd generally avoid any waiting variant of a system service, or I/O. Best practice is to keep AST routines as short as possible. I'd tend to avoid LIB$PUT_OUTPUT, except for debugging.

 

Sometimes it makes sense to "invert" an application and basically do everything at AST level, but that means you can't take advantage of multiple CPUs, as you can with kernel threads.

A crucible of informative mistakes
Berryman
Occasional Contributor

Re: Posix Threads and ASTs

As I read them, the KP routines require voluntary stalls and also would require my program to implement the parallelism and cpu scheduling.  Considering the number of system services I am using that do I/O behind the scenes, e.g. LIB$FID_TO_NAME, I don't think they would be practical for this particular application.

Hoff
Honored Contributor

Re: Posix Threads and ASTs

I'm not a big proponent of the POSIX threads (pthreads) design, nor of the VMS implementation.

 

The pthreads layer is comparatively arcane and historically somewhat fragile, the VMS APIs haven't tracked the associated POSIX standards and the defacto standards and thus aren't current (which can make porting threaded code even more "fun").   It works.  Except when it doesn't.  And when it doesn't work, it's often really hard to figure out what really happened.

 

(Old joke: there are two classes of bugs.  Application bugs. System bugs. And threading bugs.)

 

The application design resulting from using pthreads is comparatively harder to implement and maintain than some of the more modern threading schemes, as the programmer ends up littering the codebase with the threading constructs and the inevitable debugging of threading-related bugs involves chasing the threaded routines all over the source pools and the modules.  Go find this thread over there, either elsewhere in this module or else-module, and without any debugger and editor assists for finding this stuff and following it all around.

 

If this code doesn't need to be portable, I'd likely look to use a combination of KP threads and hardware queues to implement my own threading (hanging pending threads off of queues, and with one or more of these thread queues configured and active as required), and would probably avoid pthreads.  This to mimic Apple's Grand Central Dispatch (GCD) threading design, which is just way simpler to deal with than pthreads.  Yeah, it's a thread scheduler.  But one that's not as opaque as pthreads.  

 

(Having the gcc or llvm/clang blocks (closures) available here would be handy here too - they're effectively an in-line AST - though that's not an option with the VMS C compiler.  But I digress.)

 

Put another way, yes, KP Threads is ugly, but going directly at these lower-level APIs removes one whole layer of complexity from the application design.

 

Here's an overview on the topic: http://h71000.www7.hp.com/openvms/integrity/openvms-kp-services.pdf and including a list of the various pieces and parts that are using KP threads.  (Does anybody know of any pieces of VMS that are using the pthreads C APIs?  I don't recall encountering any examples.)

Re: Posix Threads and ASTs

Hi Mark,
Below are the answers to your queries.
From Post 1

-----------

Re 1: Currently there are only process-wide ASTs. With MULTITHREAD > 1 (and linking with /THREADS=UPCALLS), these ASTs are delivered only to the initial thread. Hence there can be no two ASTs executing in parallel. However, an AST can simultaneously execute along with another thread (non-AST). Multiple ASTs can never execute in parallel (either now or in future).
Re 2: The behaviour of $SETAST changes depending on whether upcalls are enabled or not. Since you mentioned that upcalls are enabled, $SETAST will disable the AST delivery for the entire process. ASTs will be delivered only if *ALL* the threads allow it.
Re 3: The main thread can continue to do whatever it wants. Whenever there is an AST, it will be executed in this thread. Once the AST is done with its execution, it will continue with what code it was running before the AST came in. ASTs are delivered to the intial thread since that thread has huge stack space, and running an AST in this thread can rarely result in a stack overflow.
Re 4: Currently, it is only process-wide. Thread-specific ASTs will are not supported in any versions so far (and are not in plan for future versions)
From Post 3

-----------

Re 1: This is correct
Re 2: This is correct
Re 3: Assuming the data is manipulated *ONLY* within AST, then this is correct.
Re 4: ASTs in a threaded application cannot call system calls (or CRTL APIs      that do an I/O). Please refer to http://openvms.compaq.com/doc/84final/5763/5763profile_006.html#multithread_restrict_sec      for more details on this.

 

Best Regards, 

Sandeep

OpenVMS Engineering

Berryman
Occasional Contributor

Re: Posix Threads and ASTs

Sandeep,

 

Re 4: The manual you reference is the C run-time library reference manual.  Thus, when it refers to not using any of the I/O routines I understood it to mean none of the C RTL I/O routines.  I currently call close in one of the ASTs but I can correct that.

 

However, I read your message as saying that no system service that does I/O is allowed either.  These ASTs were written to handle all I/O going to/from the network.  Thus, they check the IOSB status and then issue a SYS$QIO to queue up another read or write (or SYS$CANCEL in the case of an error).  Are you saying that these system calls are not allowed either?  If so, then I will have to follows Hoff's suggestion and switch to KP threads.

 

For debugging, I am also using LIB$PUT_OUTPUT from within AST routines.  However, I can remove that routine if calling it from an AST has potential issues.

 

Thank you,

Mark Berryman

Hoff
Honored Contributor

Re: Posix Threads and ASTs

The tr_print tracng stuff is lighter-weight than printf or lib$put_output; it dispatches your tracing information into a system-maintained ring buffer, and accessed via the SDA> TR tools.  FWIW.   (I have some tr_print stuff posted at the web site.)

 

Or you can roll your own integrated logging, if the application is large enough. 

 

This part is key: " If you must use ASTs and threads, you will want to perform the  absolute minimum processing within the AST routines, passing off  all processing to threads via application-specific work request  packets and interlocked queues or other re-entrant technique."  

 

Get out of the AST as soon as you can, and offload everything to a thread.  I wouldn't recommend I/O here.

 

In addition to the usual and expected usage of naturally aligned or interlocked data accesses when hitting memory that can be referenced in and shared across a mixture of threads and ASTs, also avoid cases where adjacent variables within the same granularity of reference; adjacent variables that will be used among threads and ASTs and colocated within the same memory management granularity.  (Errors involving this lattermost case can be Really Hard to find, too.)