Dealing with big number of system calls

[Vassily Gorbounov_1]

Hi, all
I have a third-party application which was written not optimally. Actually it
make a big number of system calls to read data.
And the symptoms are as follows:
Ten users use this application in active mode (when it calculates something) -
then approximately 12 processes in run queue (2 CPU system).
Glance shows approx. 38000 sys calls/sec (GBL_SYSCALL_RATE parameter)
In the same time disk activity shows 10000 logical reads/sec (4 MB/sec) for the
most usable disk with only 80-90 physical reads (200 KB/sec).

And my question is: can something be done in this situation, except for the new
CPU buying?
Do filesystem settings help it?
And the second question - are there any obstacles for adding not 2 but 1
processor, that is - is a presence of 3 CPUs less correct than 4 CPUs or not?

Thank You.

[Evan Day_1]

Unfortunately, I don't think there's anything you can do about the number of
system calls. You could try spreading the filesystem across multiple disks -
if there is more than one filesystem being accessed, put each filesystem on a
different disk (or spread them as much as the number of disks allows). You
could also try striping (placing each logical volume in stripes across multiple
disks).

As far as adding a 3rd CPU, there is nothing wrong with it. The # of CPUs
isn't restricted to an even number or power of 2 or anything like that. I've
got a box with 3 CPUs and one with 5 and they both run just fine.

HTH
-Evan

[Kimberly Ann]

test

[Hein van den Heuvel]

38,000 sys calls/sec may be reasonable, with 10,000 reads / sec, and some timer and lock calls probably. 4mb/sec is not much. small IOs? What tool did you use to see that?
80 - 90 physical IO/second is more likely to be more that 200KB/second but still low.

What are the usr:sys:idle ratios?
Little idle left, so you are cpu bound?

Short of fixing the 3rd party application, more CPUs are possibly needed. There are no tuning knobs for the application? Bigger IOs? larger internal cache?

It can certainly be appropriate to have an odd number of CPUs. But if you are going to order something and open the box and take the downtime then adding a pair and really making a difference in potential performance might be more effective. If your software has a per cpu license (Oracle?) then you coudl add it in and not use it (for now).

Also be sure to add more memory as you add CPU power. That is likely to be needed.
Now adding memory often needs to be done in pairs or even quads to fil 'banks' and get balanced interleaving. For CPUs that is not the case in designs I am familiar with.

You did not mention the box, some systems come with 2 CPUs per board/cell/chips giving you no choice.

Good luck,
Hein van den Heuvel
HvdH Performance Consulting.

[Hein van den Heuvel]

[Sorry if this becomes a duplicate reply.
I did a submit, but do not see my or other replies. The index shows 2 replies. Odd.]

38,000 sys calls/sec may be reasonable, with 10,000 reads / sec, and some timer and lock calls probably. 4mb/sec is not much. small IOs? What tool did you use to see that?
80 - 90 physical IO/second is more likely to be more that 200KB/second but still low.

What are the usr:sys:idle ratios?
Little idle left, so you are cpu bound?

Short of fixing the 3rd party application, more CPUs are possibly needed. There are no tuning knobs for the application? Bigger IOs? larger internal cache?

It can certainly be appropriate to have an odd number of CPUs. But if you are going to order something and open the box and take the downtime then adding a pair and really making a difference in potential performance might be more effective. If your software has a per cpu license (Oracle?) then you coudl add it in and not use it (for now).

Also be sure to add more memory as you add CPU power. That is likely to be needed.
Now adding memory often needs to be done in pairs or even quads to fil 'banks' and get balanced interleaving. For CPUs that is not the case in designs I am familiar with.

You did not mention the box, some systems come with 2 CPUs per board/cell/chips giving you no choice.

Good luck,
Hein van den Heuvel
HvdH Performance Consulting.

[A. Clay Stephenson]

Normally when one sees a large number of read() system calls that means that the application is not taking advantage of buffering and because you are probably i/o bound, adding CPU's either in one's or two's is probably going to make minimal difference. What is really needed is a rewrite of part of the application.

[Sandman!]

What system calls are the predominant ones?? Since your phys I/O is far too small compared to the logical I/O you need to isolate the system calls that are being executed majority of the time.

~cheers

[Bill Hassell]

The only thing that can be done is to fix the application. HP-UX is doing what it was told to do. The high system call rate (and low physical disk I/O rate) indicates that the application is probably reading very small records and the vast majority of the records are already in memory via the buffer cache. So this is a good thing compared to trying to read small records directly from disk.

More processors will help. The quantity that you can add is very dependent on what computer you have. However, with 12 in the run queue, a better number might be to add 4 or even 6 processors. The idea here is process the cache in parallel to help with the poor programming.

[anonymousert]

Systems calls are a necessary part of any operating system, but they can be a performance bottleneck. In this blog post, we'll take a look at how to deal with a large number of system calls, and how to optimize them for performance. System calls are a necessary part of any operating system, but they can be a performance bottleneck. In this blog post, we'll take a look at how to deal with a large number of system calls, and how to optimize them for performance.

System calls are used to request services from the kernel and are typically made by user-space programs. Each system call requires a certain amount of time to execute, which can add up when there are many of them. This can lead to performance bottlenecks, as the kernel can become overwhelmed with requests.

There are a few ways to deal with this issue. One is to optimize the system calls themselves. This can be done by reducing the number of parameters that are passed to the kernel, or by using faster algorithms.

Another way to deal with this issue is to reduce the number of system calls that are made. This can be done by batching requests together, or by using a library that provides an alternative to system calls.

Finally, it's also possible to use a virtual machine, which can execute system calls in a separate process. This can be helpful if the system calls are particularly slow, or if they're causing performance bottlenecks. System call optimization is an important part of performance tuning and can make a big difference in the performance of your system. By reducing the number of system calls, or by optimizing them for speed, you can improve the performance of your system.