topic Re: Performance problem with syslog in multithreaded application in Operating System - HP-UX

Performance problem with syslog in multithreaded application

Nils-Olof Wilske — Mon, 16 May 2005 03:28:17 GMT

Hi,

I recently discovered that syslog calls in a multithreaded application is a real performance killer. I guess there is a mutex in the syslog function that blocks the threads, but there must be something more than that.
If I replace the syslog function with a function that does a "mutex.lock; printf; mutex.unlock" sequence there is no noticable performance loss.
I get the same results on HP-UX 11.00 and HP-UX 11.23 (Itanium).

Any ideas?

Re: Performance problem with syslog in multithreaded application

Laurent Menase — Mon, 16 May 2005 03:38:39 GMT

Hi,

It may be interesting to get a 'tusc' of the application with the options
-E -v -l -T "" -f -p -o result.out
It can give an idea where the time is passed.

Re: Performance problem with syslog in multithreaded application

Nils-Olof Wilske — Mon, 16 May 2005 04:18:10 GMT

As requested I have attached the output of
'tusc -E -v -l -T "" -f -p'
for the process. (gzipped)
A short explanation:
The process serves clients (8 in this case) connecting via TCP/IP. A thread is created for each client.
The thread sends a query to Oracle, and returns the result to the client. The thread also makes 6 rapid syslog calls.
The response time for a client is proportional to the number of clients! Without the syslog calls it is more or less independent of the number of clients.

Re: Performance problem with syslog in multithreaded application

Laurent Menase — Mon, 16 May 2005 06:00:09 GMT

Apparantly it sched_yield a lot to get the mutex.
I suppose it is the cause of the perf kill

you can try to call not portable
pthread_mutex_setyieldfreq_np( PTHREAD_MUTEX_YIELDNEVER_NP);

Re: Performance problem with syslog in multithreaded application

Nils-Olof Wilske — Mon, 16 May 2005 07:51:37 GMT

Thanks for your suggestion. I can see a slight improvement after calling pthread_mutex_setyieldfreq_np(PTHREAD_MUTEX_YIELDNEVER_NP).

But it is not even close to the response times without the syslog calls.
What is syslog doing here? Maybe I should use another syslog client implementation?

Re: Performance problem with syslog in multithreaded application

Stephen Keane — Mon, 16 May 2005 11:01:24 GMT

Are you calling openlog() before you spawn your threads, or afterwards (i.e. once per thread)?

Re: Performance problem with syslog in multithreaded application

Nils-Olof Wilske — Tue, 17 May 2005 01:27:41 GMT

>> Are you calling openlog() before you spawn your threads,
>> or afterwards (i.e. once per thread)?

openlog() is called once in the main thread before starting any other threads.