The HP-UX Operating Systems Team Wants Your Feedback

[HP-UX OS TEAM]

Hello HP-UX Community Members,

We are the HP-UX OS team engineers in Cupertino, California. Currently we are investigating how customers use the Pthreads library in their work environment. We want to gather feedback from you that will help us design future versions of the Pthread library to better fit your needs. You can be a part of this exciting process by responding to any or all of the 3 groups of questions posted in the HP Forum. We do not intend to use your information to create a mailing list. Likewise, we will not pass your names on to marketing or anything of that sort. None of your private information (like email, address, name, company, etc) will be shared with anyone. Our intentions are only to improve Pthreads, not to annoy you with unwanted solicitors.

We greatly appreciate your input into this investigation and we look forward to hearing your feedback on future product enhancement ideas.

Thank you,

THE HP-UX OS TEAM

PLEASE NOTE: This is NOT a defect logging mechanism. We will not be providing solutions to those types of questions. The purpose of this is solely to collect helpful and valuable information about future product development.

Pthreads Questions

1)HP-UX supports the POSIX.1c pthread library from 11.0 onwards. Are you linking your multi-threaded apps with the POSIX.1c pthread library (/usr/lib/libpthread.sl) or the earlier one provided by DCE (/usr/lib/libcma.sl) ? How many threads on an average do you create in your application? Would you like to migrate to the MxN thread model to be released soon by HP?

2)Do you use any kind of tools to analyze/tune your multi-threaded app performance? If so, what are they? What kind of tools would you like to have? Would you like to tune certain parameters in the pthread library to improve your app?s performance? Would you be interested in achieving performance improvement with some non-standard features?

3)What kind of synchronization and IPC mechanisms do you use in your application?
1)pthread mutexes
2)pthread condition variables
3)pthread read-write locks
4)System V message queues/ semaphores / shared memory
5)POSIX semaphores
6)Memory semaphores (OSF AES semaphores)

[Ravi_8 · Accepted solution]

1. yes we are not linking our appliction with posix provided by DCE, we usually create 32 threads , i don't have any idea about MxN threads.
2. we don't use any tools to analyze/tune the multi-threaded application performance.i would like to have some tools to achive this.
3.In our application we use system V message queues,shared memory and semaphores

[Wodisch]

Hello Pthreads-team,

1) for most multi-threaded apps I am using DCE anyway,
so I have to be portable to other DCE-implementations
but it would be nice to have some kind of "-Defines"
to switch behaviour.
What are "MxN" threads?
2) Self-written checking, have not investigated the
"Wildebeest", yet - does it allow analyzis of DCE-
threads anyway?
3) For the DCE apps of course mutexes and condition
variables, only rarely POSIX semaphores...

Where do you publish the "results", i.e. what you will
implement in the future (HP-UX12?)???

HTH,
Wodisch

[Ralf Hildebrandt]

It would be totally sufficient if such software like eg. Apache-2.0 or a multithreading BIND-9.x will build and work successfully.

This is - after all - where HP fails to offer adequate up-to-date software themselves.

[John Bolene]

1)using DCE

I know nothing about MxN.

2)Don't know of any tools to use.
Tell us about non-standard features.

3)mutexes / semaphores / shared memory

[Victor BERRIDGE]

Hi,
1) /usr/lib/libpthread.sl
I also know nothing about MxN

2)No...

3) -> System V...

All the best
Victor

[Steve Woloszyn]

1. Yes - already migrated to pthread (MxN).
2. No - tools are not required.
3. Thread, Mutex, Semaphores are used.

[Karl Staas_1]

We don't sell any multi-threaded applications but we do sell a single-threaded shared library that gets used by several customer's multi-threaded applications. The problem they typcially have is that we build on our 10.20 system (which is kept there because a) its a 715 and b) we need to keep a very conservative least common denominator) with its POSIX 1003.4 pthreads. That doesn't work well with many of these multi-threaded applications.

On AIX, OSF, and Solaris, where there's a more recent POSIX snapshot, we can build our library so its thread aware and our customers can use us in their multi-threaded applications.

So ... I'd have to say that I'd like to see the update to pthreads.
--
thanks for asking,
Karl

[Kiran N. Mehta]

I'll chose to answer the three questions w.r.t. my previous employment (a database engine):

1) The DB server- and client-side libraries were linked with /usr/lib/libpthread on HP11.00 (and with libcma on 10.20). The server could sometimes spawn thousands of threads (one per client). The MxN scheduling model would certainly be desirable.

2) I didn't end up using ttv (much though I would have liked to); a character interface to ttv would have encouraged me to use it, as the slow connection wouldn't have posed as grave a bottleneck to it...

I would like to have the following analysis tools:
+ A runtime tool that logs, detects & reports the lack of consistency in the order in which threads acquire different mutexes--something that, if I understand correctly, will result in a deadlock in a time-critical fashion...
+ A log (structured per condition-variable) that will help identify in terms of thread-identity the sequence in which the threads blocked on a CV acquired the associated mutex, in response to a pthread_cond_broadcast

I would like the following level of tunability:
+ If an appln. is single threaded but links with libpthread only to avail of the thread-local-storage scope which is integral to the API the appln. must use, there shouldn't be any significant overhead; there is a libc patch that allows the such an appln to be de-linked from libpthread and what I am saying is such a relink with a patched libc should not be necessary--it should happen transparently by the same libpthread...
+ Some benchmarking information and guidance about the judicious use of _M_ARENA_OPTS

3) The appln used pthread mutexes & condition variables and an assembly-level module that implemented interfaces similar to the libc interfaces _check_lock( ), _safe_fetch on AIX4.3.3 (why doesn't HP export the same??); for IPC, we used shared memory & message queues.

Further, I would like the following enhancements:
+ An error/signal when the 64KB thread-stack overflows
+ The dynamic loader should be able to load libraries that use thread-local-storage.

Thanks.

[Satish Y]

1) Using DCE. Mostly 2 threads, rarely 3 threads.
2) We don't have any tools... like to have some. We have scripts to monitor.
3) We use threads, shared memory and semaphores.

Cheers...
Satish.

[yetagain]

We use libpthread with about 50 threads per process with HP/UX 11.0. There are several processes per machine. About 10 of the threads are highly active with about 5,000 pthread_mutex_lock per second per thread. pthread_mutex_lock and unlock are the highest CPU using calls. About half of the pthread_mutex calls are generated from malloc or delete. The other half are explicit calls or from RWMutexLock::acquire/release.

To make our application faster, I am trying to replace memory allocation with thread_specific storage to avoid malloc/delete. Another think I would like to do is some faster way (maybe semaphore(?) to do explicit lock on counter. Many places we do pthread_mutex_lock, ctr++;, pthread_mutex__unlock; is there some faster way to do this without the pthread_mutex_lock/_spin_lock, _spin_unlock, pthread_mutex_lock?

Is there something like the libcres.a for libpthread?

I often suspect that some internal lock on heap or similar is bottleneck but I can't get any tracing tool to work at high volumes (and the HP CxPerf does not trace _spin_lock/_spin_unlock). Is there some way to tell if the _spin_lock time is increasing as transaction rate increases?

[John Bolene]

1) Using DCE lib. 12 threads, sometimes up to 15. Not sure what the MxN thread model is?
2) no monitoring is done, not sure how, but it would be wonderful to do it
3) We use threads, mutex, shared memory and semaphores.

[Ricardo Bassoi]

1)using DCE
2)No...
3)mutexes / semaphores / shared memory

[Christopher Caldwell]

1)HP-UX supports the POSIX.1c pthread library from 11.0 onwards. Are you linking your multi-threaded apps with the POSIX.1c pthread library (/usr/lib/libpthread.sl) or the earlier one provided by DCE (/usr/lib/libcma.sl) ?

libpthread on newer apps; user threads on older apps.

How many threads on an average do you create in your application?

Web servers high (256+).
Other apps, low.

Would you like to migrate to the MxN thread model to be released soon by HP?
Not sure.

2)Do you use any kind of tools to analyze/tune your multi-threaded app performance?
Nope.

If so, what are they?
N/A

What kind of tools would you like to have?
We use Glance for perf mon - but Glance doesn't really seem to "show" us much about what's happening in threaded apps.

Would you like to tune certain parameters in the pthread library to improve your app?s performance?
Yup.

Would you be interested in achieving performance improvement with some non-standard features?
As long as non-standard is CLEARLY marked and can be conditionally defined/implemented for portability.

3)What kind of synchronization and IPC mechanisms do you use in your application?
1)pthread mutexes
Yup.
2)pthread condition variables
Dunno.

3)pthread read-write locks
Dunno.

4)System V message queues/ semaphores / shared memory
Yup.

5)POSIX semaphores
Dunno.

6)Memory semaphores (OSF AES semaphores)
Dunno.

[Ralph Grothe]

As I haven't done any significant IPC programming yet I probably am not entitled to participate here.

I can only say that I wished to test a threading Perl version as well as a threading Apache webserver on any of our HP-UX boxes.

Because some apps (e.g. such as databases) seem to thread I experienced very annoying and time consuming installation orgies of CPANs DBD::Oracle driver for the Perl DBI.
Only after a longer correspondence with the DBI mailinglist have I been able to get it linked against the Oracle libs and installed to have my Perl scripts working again.
The whole issue seemed to be unique to HP-UX whereas other Unices had no problems in that respect.

Please forgive my unqualified utterance for I don't know what I'm talking about.

[unixdaddy]

1) Generally I am using DCE anyway,so I can use it on other DCE- implementations.
2) I've written various checking utils to investigate what is happening
3) mutexes and posix semaphores

[Dan Carter_1]

Well, okay so my response is a little late but I hope it will help. I'm a system test/integrator with an application company and work very closely with our lead engineers, provide escalation support for our customers, and interface with the HP kernel team for problem resolution.

Pthread Questions:
We use them extensively, creating between 90 and 200 threads. We stick with the new and not old DCE. JVM in mixed mode with HotSpot.

tuning/Optimizing:
I use jmeter, glance plus and the various system utilities like tusc and tune according to the HP Java developer guide and specific patch recommendations. More work on native threads and the pthread library would be great, especially with mutex issues. Right now I'm evaluating the lastest round of patches from Jan 2002. Of course we're also looking for optimizing for garbage collection, heap sizes, permanent memory and the like.

Synchonizations:
Just about every one that is listed we use as we link with an Oracle Enterprise server. I find that the introduction of third-party enterprise communications (another enterprise software back end) is where serious performance tuning and troubleshooting happens and gets ugly very fast.