syslogd

[Jamal Salim]

We use syslog service to create our application log files. This offers us a centralised mechanism to direct our application logging from different processes to the same file and allows us to implement error hierarchy easily. We recently bought L series machines to upgrade our environmet. We are encountering system performance issues when we run the application on the new machine. We have tracked it down to syslog based logging. If we disable the syslog, and instead use direct logging to the same file, everything appears to run as expected, but with syslog the performance is 4 times slower.

ARE THERE ANY KERNEL PARAMETERS THAT WE NEED TO PLAY WITH IN ORDER TO TWEAK THE PERFORMANCE OR IS THERE ANY OTHER WAY OF RESOLVING THIS PROBLEM.

[A. Clay Stephenson]

Hi Jamal,

Assuming you have a reasonably tuned machine there are no kernel parameters that are going to help. You might try starting syslogd with the -r option (allow duplicate entries) this reduces the overhead quite a bit.

It sounds as though you are doing a large amount of logging. I'm not surprised that syslog is 4x slower but one of the things that syslogd is doing that simply appending to a file is not is preventing interleaving of messages.

If the probability of interleaving output is low then I can suggest one method that might be the best of both worlds. Simply let your applications append to a text file and have a daemon to then log the data to syslog. The daemon would have to keep up with its last checkpointed line number and simply periodically start from that point to the end of file. When done, reset the line number to the current EOF. This way your applications could write at full speed and you with still have centralized messages.
A more complicated version would either be multi-threaded to allow several connections at once using a named pipe, or use semaphores so that only one application could write at a time, or use ipc messages for the same thing.

Food for thought, Clay

[Jamal Salim]

Hello Clay:
Thanks a whole bunch for all your great ideas. I would have them tired tomorrow, however I do like to add here that the same design is working fine on a K class machine with comparable capacity in terms of processing power and memory. The new machine that we aretrying to set up has no user load what so ever, whereas our production environment typically runs with 250 to 300 concurrent users without any slowing down of logging.

[A. Clay Stephenson]

The tunable I would look at in that case is dbc_max_pct. By default it is set to 50% and that is way too high. Usually setting dbc_min_pct to about 5% and dbc_min_pct to 15% is a very good starting point. But if your K-box runs that much better, I am beginning to suspect that you have a runaway process or too little memory. Use top or glance to look for processes and memory usage.

I would also trim your syslog and see if that makes a performance difference.

Regards, Clay

[Jamal Salim]

Thanks again for your prompt response.

I know for sure that we don't have a run-away process. We have been monitoring system usage and we didn't find any of the usages high enough to have caused concerns. About the other parameters, I am not sure what percentages they are set at. Certainly I will try the recommended values and see if that would make any difference. I will let you know.