About migrating application to POSIX THREADS

[Bob CI]

Hello everybody. I´m new in this forum and i'm very suprised with the knowlegment´s level of this site!!!

We have a application that makes a intensive use of ASTS. There is 80 $QIO pending in 80 decnet mailboxes for 80 processes in machine.

The performance are good, but we want to migrate to POSIX THREADS.

We probe a process with 80 threads listening every thread, every channel, but the performance was the worst. We gave up this way.

In a initial aproximation, we introduce the sincronous part in a new thread. This thread deal with packets collect by AST and when there isn´t work, makes a cond_wait. AST, save packets in memory and instead of wake makes a cond_signal when is necesary.

In my openvms readings, i read that AST code run in the context of first default thread. Perfect, I want to gain with SMP, with one AST runnig and second thread running in paralell.

But we have a problem. Althougth I establised SCHED_OTHER and priority 8 for the default thread, it seems that the default thread ( primer thread ), when is running the AST CODE ( this first thread doesn´t do anything, only perpetuous cond_wait ) his SCHED and priority are FIFO and 33 ( i saw this with debug ), and i can´t fight with it, because the second thread has a sched FIFO and priority is 31, the maxinum that system let me for the thread.

Everybody can make a suggestion ?

Thanks in advance....

[Hein van den Heuvel]

>> new in this forum and i'm very suprised with the knowlegment´s level of this site!!!

Roberto, Welcome to the HP ITRC Forum for OpenVMS.
Nice job in the 'butter up' depertment.

Just a few question, which will hopefully help to get some clarifications.

>> We have a application that makes a intensive use of ASTS. There is 80 $QIO pending in 80 decnet mailboxes for 80 processes in machine.

So help me understand the purpose and point of the ASTs? If each process is just waiting, then it might was well work synchroneously.

Don't get me wrong, I really like AST's.
I think they are a very powerful way to construct and application, possibly better than threads. But it does not sound like they are really exploited here.

>> We probe a process with 80 threads listening every thread,

Have you also tried a single process with 80 AST's outstanding?

How many CPU's are available?
The power of (kernel) threads is the potential to have more than 1 active CPU for a single process.

How much CPU is needed? More than 1 CPU accross the 80 streams? If it is less than 1 CPU, then a single process juggling 80 threads through ASTs may be optimal.
Or maybe a few processes, each dealing with say 20 - 30 ports.

Or, an old time favourite, a single 'listener' or execution controller with unlimited active waits, which then hands out work parcels to the first free slave task (ACMS paradigm).


>> priority are FIFO and 33 ( i saw this with debug ), and i can´t fight with it, because the second thread has a sched FIFO and priority is 31,

I'm afraid I never studied this area, but priorty 33 vs 31 are no problem as long as there is more than 1 processor available for the process. Is there?


Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Jon Pinkley]

Roberto,

Another multiple choice question. What is the purpose of migrating to Posix Threads?

I can think of several possible reasons that you may be considering the migration.

1. Portability. You have plans to migrate off VMS, and are starting to rewrite code without VMS specific features.

2. Familiarity. The current support organization is not familiar with VMS and its features, and is familiar with Posix threads.

3. Performance. Hope for better performance due to multiprocessing, and the belief that Posix threads are more scalable in an SMP environment.

4. Management directive to use "standards".

Is it true that the 80 $QIO all have the same AST completion routine? Is this becoming the point of congestion? If you do a

$ monitor modes/cpu

is one CPU busy while the others are nearly idle? If your concern is performance, but no single CPU is limiting your performance, what are your performance goals?

Also, please tell us about your hardware and software configuration (is this VAX, Alpha, IA64, VMS 5.5-2 7.3-2 8.3 ???) Can any of this change?

I realize this answers none of your questions, but it will help us make a better assessment of your problem and goals.

Jon

[Bob CI]

Thanks you for your quick answer. I didn´t explain very well. The process works like a server that manages 80 parallel decnet connections with another 80 process in the same machine. I use AST, because i don´t want to polling 80 decnet channels in the process, due to high latencies between messages from the same process client. This is the reason why i use ASTS. I have 80 parallel $QIO READVBLK in 80 decnet channels and the AST are delivered when messages arrives. It is a unique process that manages the 80 parallel conexion with other 80 clients process in the same machine.

The system has 8 CPUS, and i link with upcalls and kernel_threads enabled.

I´m trying to find a way to make the same, but with better performance with pthreads.

I tried with 80 threads, but the performance is very bad, because all threads are very related one another, because all save in the same chain of memory several messages. The latencies are high too. The sincronous part is the consumer thread and write the messages into disk.

And now, I trying another possibility, respect the AST code into pthread process. I use only 2 threads, 1 (default) for AST, and another for sincronous part of the process ( when AST do WAKE and mainline code do the work of write messages into disk ). In this case AST don´t make WAKE, do cond_signal, and the sincronous part do cond_wait instead of HIBER in the former case.

Thanks.....

[Bob CI]

We are in OpenVMS 8.3.

[Bob CI]

We wants to migrate because support HP recomended us migrate to POSIX THREADS for better perfomance....

[Jon Pinkley]

RE: "We wants to migrate because support HP recomended us migrate to POSIX THREADS for better perfomance...."

Was this OpenVMS engineering, or some local Unix support engineer?

?

[Bob CI]

Thanks you, Jon for your answer. In my previous version, when i have a very big chain in memory, i had to WAKE, off ASTS, deal with the excess of messages, and achive to decrease the chain in memory. I want, with threads, that everybody works, AST, and sincronous part in parallel.

Thanks....

[Bob CI]

Jon, was OpenVMS engineering......

[Hein van den Heuvel]

Thanks for the update. That helps us understand.

What is the expected main advantage of having all tasks execute in one process?

Shared memory which can not be implemented as true shared memory? Probably not.

A big potential advantage I can think of is a shared file which each worker has to write to. In the multy process case it would have to be opened shared, with each process its own (rms) buffers.
With multythreading you can open it just once, possibly exclusive, and effectively share the buffers between the streams. They could be RMS streams connecting multiple RABs to the same FAB to be able to have individual worker have their own RMS context, or they might just all use the same handle (FAB+RAB).
To take advantage of this, you would not only have to switch to threads, but change the code to use the same file handles. Was that in place for the test?

This is but one example, but without changes like that you are pretty much only hoping/expecting that the OpenVMS (process) scheduler is noticably slower than the pthread scheduler. For that to be noticable at all, never mind whether it is faster, there would need to be relatively little processing and a lot of context switching.

Also, my original comment on the need for ASTs stands.
Again, I love doign it this way, but there needs to be a reasons. With a 1 on 1 AST -to process mapping one has to wonder why each process does not just call a common subroutine set to set up the listening.
In doing so, you may be able to avoid the context switch after the AST being taken, and the worker process being woken up, and avoid a message from having to be passed.

For now, the value of the listener process is not clear yet, and getting rid of it may be a more interesting way to improve the application.

Finally, me being an RMS consultant, you indicated one of the true work activities is to add data to a file.
While it is valueable to try to get to start doign that as efficiently as possible, has there been a review whether the work itself is being done efficiently? There may be much more 'bang for the buck' to be found in the back-end processing than the front-end scheduling and message passing.
Shared file? Indexed? Buffers tuned?


Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting