Operating System - OpenVMS
1748195 Members
3493 Online
108759 Solutions
New Discussion юеВ

Re: About migrating application to POSIX THREADS

 
Bob CI
Advisor

Re: About migrating application to POSIX THREADS

Actually, all the work is in the same process and the perfomance is very high.

Teorically, with threads, i achive a paralelism that i haven├В┬┤t now. But always thinking use 1 process for all. Why not achive better results if i have a process multithread in which 1 AST y several threads can running in separate processors.

Thanks...
Hein van den Heuvel
Honored Contributor

Re: About migrating application to POSIX THREADS

>> Why not achive better results if i have a process multithread in which 1 AST y several threads can running in separate processors.

Because quite possibly, deeper down, the underlying work is serialized anyway?

"do the work of write messages into disk"
Do they all have their own file, or is there some synchronization happening?

Cheers,
Hein.
Jean-Fran├зois Pi├йronne
Trusted Contributor

Re: About migrating application to POSIX THREADS

Why not use 8 processes each process managing 10 connections using AST, and some locks/shared memory if you need communication between these processes?

JFP
Jon Pinkley
Honored Contributor

Re: About migrating application to POSIX THREADS

RE:"Teorically, with threads, i achive a paralelism that i haven├Г ├В┬┤t now. But always thinking use 1 process for all. Why not achive better results if i have a process multithread in which 1 AST y several threads can running in separate processors."

I think it really depends on how much work is done for each message recieved. If it takes more work to "delegate" the work to another process than to process the message, then the performance won't improve with SMP. "Too many cooks in the kitchen".

On the other hand, if each message requires a lot of processing, then multiple thread or multiple processes makes more sense.

Or divide the work among multiple processes, each handling a subset of the decnet connecttions. This is like building more kitchens, and each kitchen handling a different banquet hall.

Jon
it depends
Bob CI
Advisor

Re: About migrating application to POSIX THREADS

With the asyncronous part, i haven├В┬┤t problems because it is very fast. The process with AST is be able to collect a lot of messages in memory. Yes, i have the problem of the same file to write. This is the reason why i use a unique process for delivery AST. I don├В┬┤t need more speed. What i wanted, was avoid to stop AST for the sincronous work in the same process. This was the reason to covert to threads y achive that the asyncronous and syncronous part work together. But, in the multithread process, actually the second thread ( takes messages and writes down to disk doesn├В┬┤t work fine. Only when the AST finish ( process doesn├В┬┤t receive more messages ), the second thread begin to work. It seems a SCHED FIFO with AST running in a thread with highest priority and althought i put a low priority to default thread, the behaviour is the same!!!!

Bob CI
Advisor

Re: About migrating application to POSIX THREADS

Because the second thread doesn├В┬┤t work, and ├Г┬нs is impossible to give it max priority above the default thread that deal with ASTS, i├В┬┤ll try to separate mi real process in two process. One of them, for the management of ASTS for message collection, and other separate process for write down to disk ( there are 3 distinct files in which write ).

What is the better mechanism that you recomended for communicating both processes :

Global section, mailbox, etc...

There are other options ???

Global section is the faster opcion, i think, but the memory structures to sharing aren├В┬┤t simple ( it can├В┬┤t be simple ). This structures are double link chains, one chain for packets, and every packet points to chain with all messages of this packet. I think that is not practicable with mailboxes.

Only we can remove a packet entry from the first list in the second process, when all messages from the packet are written in files ( not all messages ar written in all files ). This last process would be multithread for paralell writing in 3 files.

Can you give me options, or other suggestions ?

Thanks in advance....



Hein van den Heuvel
Honored Contributor

Re: About migrating application to POSIX THREADS

>> Because the second thread doesn├Г ├В┬┤t work, and ├Г ├В┬нs is impossible to give it max priority above the default thread that deal with ASTS,

Something is broken or misunderstood still.
If multithreading is properly enabled, then the second, lower priority thread should compete for processor resources and get some from de OpenVMS scheduler if available.
Heisenberg? The closer you watch some aspects of the process (debugger) the less precise you see others?


>> i├Г ├В┬┤ll try to separate mi real process in two process. One of them, for the management of ASTS for message collection,

As you do, consider a pool of servers, rather then a 1 on 1.

>> and other separate process for write down to disk ( there are 3 distinct files in which write ).

Maybe a server per file, that way they don't fight amongst themself.

>> What is the better mechanism that you recomended for communicating both processes :

>> Global section, mailbox, etc...

Yes.

What is the rate? 20/sec? 200/sec?
Mailboxes work fine.

Global sections work better, specifically when lots of data needs to be communicated, with not yet determined processes (pool) but they are a lot more work to setup.
The construct with a double-linked list in a global section, with $HIBER in the slave when a last element is picked up and $WAKE in the master when inserting into an empty queue is powerful.

Some (many!) folks use an RMS (indexed!) file to have a pertinent record of things to be done.

>> There are other options ???

Check out the $ICC set of system services, like $ICC_TRANSCEIVE.
Those are made for this!


>> This last process would be multithread for paralell writing in 3 files.
>> Can you give me options, or other suggestions ?

How long can/is the writing taking?
Can that be improved with greater benefit?

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

Robert Gezelter
Honored Contributor

Re: About migrating application to POSIX THREADS

Roberto,

Having worked extensively with ASTs over the years, I can probably say that it is unlikely that the application, as described, will benefit from the work of converting to POSIX threads, unless you are routinely exceeding 100% of a single CPU.

In my experience, if you are DECnet messages queue, it is because of a wait mode system service somewhere in the path. I have written programs that have gone at over 10K ASTs per minute (on an early generation AlphaStation), and the bottleneck was the DECterm trace generation, not the AST processing of my code.

Often, POSIX threads, particularly, preemptable threads are more work than they are worth. Virtually every case that I see is better done differently. As I noted in a DECUS session several years ago, the pyramid of difficulty is: serial, FIFO non-preemtable threads, parallel threads; with the difficulty dramatically increasing at an ever increasing pace. (At the Fall 1996 DECUS symposium, I presented "Threads vs. Events" on essentially this question, the slides are available at: http://www.rlgsc.com/decus/usf96/index.html ).

One project that I recovered was over several months late due to the work in multi-threading. The reason was "efficiency". Installing the shared image, and using several copies of the "demonstrator" application had a utilization of approximately 2% of a MVII (.9 VUP). I will admit that I fixed the "efficiency" problem by removing the infinite loop at the bottom of the program.

I can also recommend taking a look "Introduction to AST Programming", from the 2000 symposium (slides at http://www.rlgsc.com/cets/2000/435.html ).

It may be appropriate to retain an outside review of the code and how it is implemented, it would likely be far less expensive than rewriting the code (The project that I referenced earlier only required a few days of my time.) [Disclosure: My firm, as do several others who contribute to this forum, provide such services].

- Bob Gezelter, http://www.rlgsc.com
Hoff
Honored Contributor

Re: About migrating application to POSIX THREADS

There's no way I'd make this move, not without a whole lot more evidence that the port would work, and that it would have advantages.

Whomever told you that you'd get better performance is technically correct for those operations that can occur in parallel.

Amdahl's law, and all.

I'd also take a careful look at the existing and expected upcoming bottlenecks, too. Whether CPU cycles, or file system or disk bandwidth, or how much I/O you can through DECnet or your NIC, etc.

None of this is free, after all.

There's also the coding effort. ASTs don't map directly to threads. The application designs and design requirements differ.

A number of applications take full and direct advantage of the blocking implicit in an active AST; only one AST can be active at a time, for instance. This is the other side of the cited improved performance of threads.

With ASTs in a single-process application, you have to worry about two threads: the mainline and the AST. Threads require added care, coordination and control. Having these additional threads active can be fodder for the formation of heisebugs, and other critters; if you were only recently using a design based on polling and on blocking I/O, threading is the deep end of the programming pool.

If the current performance is acceptable, you almost certainly have better things to do with your time than to move to DECthreads.

I've coded and maintained distributed DECnet processes serving significantly more network I/O channels, and an AST design was on a MicroVAX II was entirely adequate for that application.

As for other options, one of the fastest interprocess communications paths available on OpenVMS is a file with global buffers enabled. Yes, a file. Why? Because RMS deals with the clustering, coordination and cache control and coherency. By the time you deal with all that, well, you've done a whole lot of coding and debugging to recreate RMS. Sure, there are light-weight solutions, but as you find the limits of these, you can well find yourself with a growing application code base.

There are also various middleware packages around that can provide communications.

Synchronization:
http://64.223.189.234/node/638

ASTs:
http://64.223.189.234/node/617

Shared memory:
http://64.223.189.234/node/407

Parallelism:
http://64.223.189.234/node/523

And mixing ASTs and threads:
http://h71000.www7.hp.com/wizard/wiz_6099.html

As does Robert G., I get paid to write and to port this sort of code around. That said, I'd not advise nor suggest this migration. I'd want substantially more supporting information here; some reassurance that this (substantial) programming (redesign) effort will have benefits.

Stephen Hoffman
HoffmanLabs LLC
Jon Pinkley
Honored Contributor

Re: About migrating application to POSIX THREADS

I think the following advice is appropriate.

-------

Everything should be made as simple as possible, but not simpler.

- unknown origin, often attributed to Albert Einstein
it depends