About migrating application to POSIX THREADS

[Bob CI]

Hello everybody. I´m new in this forum and i'm very suprised with the knowlegment´s level of this site!!!

We have a application that makes a intensive use of ASTS. There is 80 $QIO pending in 80 decnet mailboxes for 80 processes in machine.

The performance are good, but we want to migrate to POSIX THREADS.

We probe a process with 80 threads listening every thread, every channel, but the performance was the worst. We gave up this way.

In a initial aproximation, we introduce the sincronous part in a new thread. This thread deal with packets collect by AST and when there isn´t work, makes a cond_wait. AST, save packets in memory and instead of wake makes a cond_signal when is necesary.

In my openvms readings, i read that AST code run in the context of first default thread. Perfect, I want to gain with SMP, with one AST runnig and second thread running in paralell.

But we have a problem. Althougth I establised SCHED_OTHER and priority 8 for the default thread, it seems that the default thread ( primer thread ), when is running the AST CODE ( this first thread doesn´t do anything, only perpetuous cond_wait ) his SCHED and priority are FIFO and 33 ( i saw this with debug ), and i can´t fight with it, because the second thread has a sched FIFO and priority is 31, the maxinum that system let me for the thread.

Everybody can make a suggestion ?

Thanks in advance....

[Hein van den Heuvel]

>> new in this forum and i'm very suprised with the knowlegment´s level of this site!!!

Roberto, Welcome to the HP ITRC Forum for OpenVMS.
Nice job in the 'butter up' depertment.

Just a few question, which will hopefully help to get some clarifications.

>> We have a application that makes a intensive use of ASTS. There is 80 $QIO pending in 80 decnet mailboxes for 80 processes in machine.

So help me understand the purpose and point of the ASTs? If each process is just waiting, then it might was well work synchroneously.

Don't get me wrong, I really like AST's.
I think they are a very powerful way to construct and application, possibly better than threads. But it does not sound like they are really exploited here.

>> We probe a process with 80 threads listening every thread,

Have you also tried a single process with 80 AST's outstanding?

How many CPU's are available?
The power of (kernel) threads is the potential to have more than 1 active CPU for a single process.

How much CPU is needed? More than 1 CPU accross the 80 streams? If it is less than 1 CPU, then a single process juggling 80 threads through ASTs may be optimal.
Or maybe a few processes, each dealing with say 20 - 30 ports.

Or, an old time favourite, a single 'listener' or execution controller with unlimited active waits, which then hands out work parcels to the first free slave task (ACMS paradigm).


>> priority are FIFO and 33 ( i saw this with debug ), and i can´t fight with it, because the second thread has a sched FIFO and priority is 31,

I'm afraid I never studied this area, but priorty 33 vs 31 are no problem as long as there is more than 1 processor available for the process. Is there?


Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Jon Pinkley]

Roberto,

Another multiple choice question. What is the purpose of migrating to Posix Threads?

I can think of several possible reasons that you may be considering the migration.

1. Portability. You have plans to migrate off VMS, and are starting to rewrite code without VMS specific features.

2. Familiarity. The current support organization is not familiar with VMS and its features, and is familiar with Posix threads.

3. Performance. Hope for better performance due to multiprocessing, and the belief that Posix threads are more scalable in an SMP environment.

4. Management directive to use "standards".

Is it true that the 80 $QIO all have the same AST completion routine? Is this becoming the point of congestion? If you do a

$ monitor modes/cpu

is one CPU busy while the others are nearly idle? If your concern is performance, but no single CPU is limiting your performance, what are your performance goals?

Also, please tell us about your hardware and software configuration (is this VAX, Alpha, IA64, VMS 5.5-2 7.3-2 8.3 ???) Can any of this change?

I realize this answers none of your questions, but it will help us make a better assessment of your problem and goals.

Jon

[Bob CI]

Thanks you for your quick answer. I didn´t explain very well. The process works like a server that manages 80 parallel decnet connections with another 80 process in the same machine. I use AST, because i don´t want to polling 80 decnet channels in the process, due to high latencies between messages from the same process client. This is the reason why i use ASTS. I have 80 parallel $QIO READVBLK in 80 decnet channels and the AST are delivered when messages arrives. It is a unique process that manages the 80 parallel conexion with other 80 clients process in the same machine.

The system has 8 CPUS, and i link with upcalls and kernel_threads enabled.

I´m trying to find a way to make the same, but with better performance with pthreads.

I tried with 80 threads, but the performance is very bad, because all threads are very related one another, because all save in the same chain of memory several messages. The latencies are high too. The sincronous part is the consumer thread and write the messages into disk.

And now, I trying another possibility, respect the AST code into pthread process. I use only 2 threads, 1 (default) for AST, and another for sincronous part of the process ( when AST do WAKE and mainline code do the work of write messages into disk ). In this case AST don´t make WAKE, do cond_signal, and the sincronous part do cond_wait instead of HIBER in the former case.

Thanks.....

[Bob CI]

We are in OpenVMS 8.3.

[Bob CI]

We wants to migrate because support HP recomended us migrate to POSIX THREADS for better perfomance....

[Jon Pinkley]

RE: "We wants to migrate because support HP recomended us migrate to POSIX THREADS for better perfomance...."

Was this OpenVMS engineering, or some local Unix support engineer?

?

[Bob CI]

Thanks you, Jon for your answer. In my previous version, when i have a very big chain in memory, i had to WAKE, off ASTS, deal with the excess of messages, and achive to decrease the chain in memory. I want, with threads, that everybody works, AST, and sincronous part in parallel.

Thanks....

[Bob CI]

Jon, was OpenVMS engineering......

[Hein van den Heuvel]

Thanks for the update. That helps us understand.

What is the expected main advantage of having all tasks execute in one process?

Shared memory which can not be implemented as true shared memory? Probably not.

A big potential advantage I can think of is a shared file which each worker has to write to. In the multy process case it would have to be opened shared, with each process its own (rms) buffers.
With multythreading you can open it just once, possibly exclusive, and effectively share the buffers between the streams. They could be RMS streams connecting multiple RABs to the same FAB to be able to have individual worker have their own RMS context, or they might just all use the same handle (FAB+RAB).
To take advantage of this, you would not only have to switch to threads, but change the code to use the same file handles. Was that in place for the test?

This is but one example, but without changes like that you are pretty much only hoping/expecting that the OpenVMS (process) scheduler is noticably slower than the pthread scheduler. For that to be noticable at all, never mind whether it is faster, there would need to be relatively little processing and a lot of context switching.

Also, my original comment on the need for ASTs stands.
Again, I love doign it this way, but there needs to be a reasons. With a 1 on 1 AST -to process mapping one has to wonder why each process does not just call a common subroutine set to set up the listening.
In doing so, you may be able to avoid the context switch after the AST being taken, and the worker process being woken up, and avoid a message from having to be passed.

For now, the value of the listener process is not clear yet, and getting rid of it may be a more interesting way to improve the application.

Finally, me being an RMS consultant, you indicated one of the true work activities is to add data to a file.
While it is valueable to try to get to start doign that as efficiently as possible, has there been a review whether the work itself is being done efficiently? There may be much more 'bang for the buck' to be found in the back-end processing than the front-end scheduling and message passing.
Shared file? Indexed? Buffers tuned?


Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Bob CI]

Actually, all the work is in the same process and the perfomance is very high.

Teorically, with threads, i achive a paralelism that i haven´t now. But always thinking use 1 process for all. Why not achive better results if i have a process multithread in which 1 AST y several threads can running in separate processors.

Thanks...

[Hein van den Heuvel]

>> Why not achive better results if i have a process multithread in which 1 AST y several threads can running in separate processors.

Because quite possibly, deeper down, the underlying work is serialized anyway?

"do the work of write messages into disk"
Do they all have their own file, or is there some synchronization happening?

Cheers,
Hein.

[Jean-François Piéronne]

Why not use 8 processes each process managing 10 connections using AST, and some locks/shared memory if you need communication between these processes?

JFP

[Jon Pinkley]

RE:"Teorically, with threads, i achive a paralelism that i havenà ´t now. But always thinking use 1 process for all. Why not achive better results if i have a process multithread in which 1 AST y several threads can running in separate processors."

I think it really depends on how much work is done for each message recieved. If it takes more work to "delegate" the work to another process than to process the message, then the performance won't improve with SMP. "Too many cooks in the kitchen".

On the other hand, if each message requires a lot of processing, then multiple thread or multiple processes makes more sense.

Or divide the work among multiple processes, each handling a subset of the decnet connecttions. This is like building more kitchens, and each kitchen handling a different banquet hall.

Jon

[Bob CI]

With the asyncronous part, i haven´t problems because it is very fast. The process with AST is be able to collect a lot of messages in memory. Yes, i have the problem of the same file to write. This is the reason why i use a unique process for delivery AST. I don´t need more speed. What i wanted, was avoid to stop AST for the sincronous work in the same process. This was the reason to covert to threads y achive that the asyncronous and syncronous part work together. But, in the multithread process, actually the second thread ( takes messages and writes down to disk doesn´t work fine. Only when the AST finish ( process doesn´t receive more messages ), the second thread begin to work. It seems a SCHED FIFO with AST running in a thread with highest priority and althought i put a low priority to default thread, the behaviour is the same!!!!

[Bob CI]

Because the second thread doesn´t work, and ís is impossible to give it max priority above the default thread that deal with ASTS, i´ll try to separate mi real process in two process. One of them, for the management of ASTS for message collection, and other separate process for write down to disk ( there are 3 distinct files in which write ).

What is the better mechanism that you recomended for communicating both processes :

Global section, mailbox, etc...

There are other options ???

Global section is the faster opcion, i think, but the memory structures to sharing aren´t simple ( it can´t be simple ). This structures are double link chains, one chain for packets, and every packet points to chain with all messages of this packet. I think that is not practicable with mailboxes.

Only we can remove a packet entry from the first list in the second process, when all messages from the packet are written in files ( not all messages ar written in all files ). This last process would be multithread for paralell writing in 3 files.

Can you give me options, or other suggestions ?

Thanks in advance....

[Hein van den Heuvel]

>> Because the second thread doesnà ´t work, and à ­s is impossible to give it max priority above the default thread that deal with ASTS,

Something is broken or misunderstood still.
If multithreading is properly enabled, then the second, lower priority thread should compete for processor resources and get some from de OpenVMS scheduler if available.
Heisenberg? The closer you watch some aspects of the process (debugger) the less precise you see others?


>> ià ´ll try to separate mi real process in two process. One of them, for the management of ASTS for message collection,

As you do, consider a pool of servers, rather then a 1 on 1.

>> and other separate process for write down to disk ( there are 3 distinct files in which write ).

Maybe a server per file, that way they don't fight amongst themself.

>> What is the better mechanism that you recomended for communicating both processes :

>> Global section, mailbox, etc...

Yes.

What is the rate? 20/sec? 200/sec?
Mailboxes work fine.

Global sections work better, specifically when lots of data needs to be communicated, with not yet determined processes (pool) but they are a lot more work to setup.
The construct with a double-linked list in a global section, with $HIBER in the slave when a last element is picked up and $WAKE in the master when inserting into an empty queue is powerful.

Some (many!) folks use an RMS (indexed!) file to have a pertinent record of things to be done.

>> There are other options ???

Check out the $ICC set of system services, like $ICC_TRANSCEIVE.
Those are made for this!


>> This last process would be multithread for paralell writing in 3 files.
>> Can you give me options, or other suggestions ?

How long can/is the writing taking?
Can that be improved with greater benefit?

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Robert Gezelter]

Roberto,

Having worked extensively with ASTs over the years, I can probably say that it is unlikely that the application, as described, will benefit from the work of converting to POSIX threads, unless you are routinely exceeding 100% of a single CPU.

In my experience, if you are DECnet messages queue, it is because of a wait mode system service somewhere in the path. I have written programs that have gone at over 10K ASTs per minute (on an early generation AlphaStation), and the bottleneck was the DECterm trace generation, not the AST processing of my code.

Often, POSIX threads, particularly, preemptable threads are more work than they are worth. Virtually every case that I see is better done differently. As I noted in a DECUS session several years ago, the pyramid of difficulty is: serial, FIFO non-preemtable threads, parallel threads; with the difficulty dramatically increasing at an ever increasing pace. (At the Fall 1996 DECUS symposium, I presented "Threads vs. Events" on essentially this question, the slides are available at: http://www.rlgsc.com/decus/usf96/index.html ).

One project that I recovered was over several months late due to the work in multi-threading. The reason was "efficiency". Installing the shared image, and using several copies of the "demonstrator" application had a utilization of approximately 2% of a MVII (.9 VUP). I will admit that I fixed the "efficiency" problem by removing the infinite loop at the bottom of the program.

I can also recommend taking a look "Introduction to AST Programming", from the 2000 symposium (slides at http://www.rlgsc.com/cets/2000/435.html ).

It may be appropriate to retain an outside review of the code and how it is implemented, it would likely be far less expensive than rewriting the code (The project that I referenced earlier only required a few days of my time.) [Disclosure: My firm, as do several others who contribute to this forum, provide such services].

- Bob Gezelter, http://www.rlgsc.com

[Hoff]

There's no way I'd make this move, not without a whole lot more evidence that the port would work, and that it would have advantages.

Whomever told you that you'd get better performance is technically correct for those operations that can occur in parallel.

Amdahl's law, and all.

I'd also take a careful look at the existing and expected upcoming bottlenecks, too. Whether CPU cycles, or file system or disk bandwidth, or how much I/O you can through DECnet or your NIC, etc.

None of this is free, after all.

There's also the coding effort. ASTs don't map directly to threads. The application designs and design requirements differ.

A number of applications take full and direct advantage of the blocking implicit in an active AST; only one AST can be active at a time, for instance. This is the other side of the cited improved performance of threads.

With ASTs in a single-process application, you have to worry about two threads: the mainline and the AST. Threads require added care, coordination and control. Having these additional threads active can be fodder for the formation of heisebugs, and other critters; if you were only recently using a design based on polling and on blocking I/O, threading is the deep end of the programming pool.

If the current performance is acceptable, you almost certainly have better things to do with your time than to move to DECthreads.

I've coded and maintained distributed DECnet processes serving significantly more network I/O channels, and an AST design was on a MicroVAX II was entirely adequate for that application.

As for other options, one of the fastest interprocess communications paths available on OpenVMS is a file with global buffers enabled. Yes, a file. Why? Because RMS deals with the clustering, coordination and cache control and coherency. By the time you deal with all that, well, you've done a whole lot of coding and debugging to recreate RMS. Sure, there are light-weight solutions, but as you find the limits of these, you can well find yourself with a growing application code base.

There are also various middleware packages around that can provide communications.

Synchronization:
http://64.223.189.234/node/638

ASTs:
http://64.223.189.234/node/617

Shared memory:
http://64.223.189.234/node/407

Parallelism:
http://64.223.189.234/node/523

And mixing ASTs and threads:
http://h71000.www7.hp.com/wizard/wiz_6099.html

As does Robert G., I get paid to write and to port this sort of code around. That said, I'd not advise nor suggest this migration. I'd want substantially more supporting information here; some reassurance that this (substantial) programming (redesign) effort will have benefits.

Stephen Hoffman
HoffmanLabs LLC

[Jon Pinkley]

I think the following advice is appropriate.

-------

Everything should be made as simple as possible, but not simpler.

- unknown origin, often attributed to Albert Einstein

[Jon Pinkley]

Here's an interesting document that you should scan, at least the following sections: General Design, Server Behaviour, Multi-Threaded, AST Behaviour, Server "Tasks", Memory Management, and Output Buffering sections.

http://wasd.vsm.com.au/ht_root/src/httpd/readmore.txt

It's the Nuts and Bolts document describing the WASD HTTP Server. It is a highly concurrent, multi-threaded (with AST's, not a threading library) HTTP server.

The WASD VMS Hypertext Services - Technical Overview http://wasd.vsm.com.au/ht_root/doc/htd/ has a chapter on performance (21). This is the first paragraph

"The server has a single-process, multi-threaded, asynchronous I/O design. On a single-processor system this is the most efficient approach. On a multi-processor system it is limited by the single process context (with scripts executing within their own context). For I/O constrained processing (the most common in general Web environments) the AST-driven approach is quite efficient."

I think it would be worth your time to look at that before you decide to rewrite using pthreads.

Good Luck,

Jon

[Ian Miller.]

multiple posix threads in a process are co-ordinated by a by a mutex. IIRC there can be only one of them calling a system service at any time. So your bottleneck may be contention for this muxtex. The PTHREAD or MTX SDA extensions may help to determine this (although they are not documented but have some built in help)

[Robert Gezelter]

Jon,

Indeed. When I speak and teach about implementing concurrency, I always speak about a "pyramid of complexity". Ascending each step of the pyramid requires a positive statement. For example:

- Going from simple sequential code to event driven code requires that the sequence of events be indeterminate (e.g., timeouts and network connections, or multiple network connections). A simple payroll computation program would never present that condition.

- Going from non-preemptable threading (e.g., AST, the equivalent in POSIX threads) requires that the processing of each event exceed the computational capacity of a single CPU OR introduces unacceptably long latencies.

In thirty years of working with event driven code and threads, I can only count a relative handful of cases that met the second criteria needed to go to pre-emptable threads. In most cases where pre-emptable threads were perceived to be "the solution", the true underlying problem was some form of synchronous processing that became [unnecessarily] embedded in the event processing.

WASD is a good example of how high performance is quite achievable.

- Bob Gezelter, http://www.rlgsc.com

[Bob CI]

Thanks to everybody for so complete answers. I need some time to analize these informations. In this moment, i´m developing new codes with a multiprocess schema ( 2 servers with the same capabilities, every server del with the half of all clients ), but, at the moment, doesn´t achieve better results that before with a unique process. Because i need to sinchronizing the concurrent access at one of the application file, i use a lock for this operation. Asinchronous part works better, and is be able to deal with more messages ( every process accumulate more messages in global terms that with a unique process), but i´m afraid i have the bottleneck with the application files. And the cost of sinchronizing is high.


Thanks for everybody and i´ll tell you my advances.....

[Hein van den Heuvel]

>> 2 servers with the same capabilities, every server del with the half of all clients ), but, at the moment, doesnà ´t achieve better results that before with a unique process

Well Duh! No kidding huh? !?

You would have the same (file) locking as with 1 process. I fact it may well be worse as RMS now may have to 'ping' the most recent block from one process to the next and back aided byt the XFC (after writing to the disk)

>> afraid i have the bottleneck with the application files.

Well Duh!

That would be true for 95% of all your applications. If if was not true the application you work on, then you would have known this and would have been able to 'proof' it with argumentation.

>> And the cost of sinchronizing is high.

Duh!

I see stuff like applications taking out a lock, reading a record which holds a next value, update, write new record using that next new value, release the lock. All they needed to do was to just keep the next-new-key value in the lock value block, update teh underlyng storage ever 10 minutes or 100 updates, and release the lock for the next user.

Good luck!
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting