Performance problem with syslog in multithreaded application

[Nils-Olof Wilske]

Hi,

I recently discovered that syslog calls in a multithreaded application is a real performance killer. I guess there is a mutex in the syslog function that blocks the threads, but there must be something more than that.
If I replace the syslog function with a function that does a "mutex.lock; printf; mutex.unlock" sequence there is no noticable performance loss.
I get the same results on HP-UX 11.00 and HP-UX 11.23 (Itanium).

Any ideas?

[Laurent Menase]

Hi,

It may be interesting to get a 'tusc' of the application with the options
-E -v -l -T "" -f -p -o result.out
It can give an idea where the time is passed.

[Nils-Olof Wilske]

As requested I have attached the output of
'tusc -E -v -l -T "" -f -p'
for the process. (gzipped)
A short explanation:
The process serves clients (8 in this case) connecting via TCP/IP. A thread is created for each client.
The thread sends a query to Oracle, and returns the result to the client. The thread also makes 6 rapid syslog calls.
The response time for a client is proportional to the number of clients! Without the syslog calls it is more or less independent of the number of clients.

[Laurent Menase]

Apparantly it sched_yield a lot to get the mutex.
I suppose it is the cause of the perf kill

you can try to call not portable
pthread_mutex_setyieldfreq_np( PTHREAD_MUTEX_YIELDNEVER_NP);

[Nils-Olof Wilske]

Thanks for your suggestion. I can see a slight improvement after calling pthread_mutex_setyieldfreq_np(PTHREAD_MUTEX_YIELDNEVER_NP).

But it is not even close to the response times without the syslog calls.
What is syslog doing here? Maybe I should use another syslog client implementation?

[Stephen Keane]

Are you calling openlog() before you spawn your threads, or afterwards (i.e. once per thread)?

[Nils-Olof Wilske]

>> Are you calling openlog() before you spawn your threads,
>> or afterwards (i.e. once per thread)?

openlog() is called once in the main thread before starting any other threads.