General
cancel
Showing results for 
Search instead for 
Did you mean: 

Re: Dealing with big number of system calls

 
Occasional Advisor

Dealing with big number of system calls

Hi, all
I have a third-party application which was written not optimally. Actually it
make a big number of system calls to read data.
And the symptoms are as follows:
Ten users use this application in active mode (when it calculates something) -
then approximately 12 processes in run queue (2 CPU system).
Glance shows approx. 38000 sys calls/sec (GBL_SYSCALL_RATE parameter)
In the same time disk activity shows 10000 logical reads/sec (4 MB/sec) for the
most usable disk with only 80-90 physical reads (200 KB/sec).

And my question is: can something be done in this situation, except for the new
CPU buying?
Do filesystem settings help it?
And the second question - are there any obstacles for adding not 2 but 1
processor, that is - is a presence of 3 CPUs less correct than 4 CPUs or not?

Thank You.
7 REPLIES 7
Frequent Advisor

Re: Dealing with big number of system calls

Unfortunately, I don't think there's anything you can do about the number of
system calls. You could try spreading the filesystem across multiple disks -
if there is more than one filesystem being accessed, put each filesystem on a
different disk (or spread them as much as the number of disks allows). You
could also try striping (placing each logical volume in stripes across multiple
disks).

As far as adding a 3rd CPU, there is nothing wrong with it. The # of CPUs
isn't restricted to an even number or power of 2 or anything like that. I've
got a box with 3 CPUs and one with 5 and they both run just fine.

HTH
-Evan
Honored Contributor

Re: Dealing with big number of system calls

test
Honored Contributor

Re: Dealing with big number of system calls

38,000 sys calls/sec may be reasonable, with 10,000 reads / sec, and some timer and lock calls probably. 4mb/sec is not much. small IOs? What tool did you use to see that?
80 - 90 physical IO/second is more likely to be more that 200KB/second but still low.

What are the usr:sys:idle ratios?
Little idle left, so you are cpu bound?

Short of fixing the 3rd party application, more CPUs are possibly needed. There are no tuning knobs for the application? Bigger IOs? larger internal cache?

It can certainly be appropriate to have an odd number of CPUs. But if you are going to order something and open the box and take the downtime then adding a pair and really making a difference in potential performance might be more effective. If your software has a per cpu license (Oracle?) then you coudl add it in and not use it (for now).

Also be sure to add more memory as you add CPU power. That is likely to be needed.
Now adding memory often needs to be done in pairs or even quads to fil 'banks' and get balanced interleaving. For CPUs that is not the case in designs I am familiar with.

You did not mention the box, some systems come with 2 CPUs per board/cell/chips giving you no choice.

Good luck,
Hein van den Heuvel
HvdH Performance Consulting.


Honored Contributor

Re: Dealing with big number of system calls

[Sorry if this becomes a duplicate reply.
I did a submit, but do not see my or other replies. The index shows 2 replies. Odd.]

38,000 sys calls/sec may be reasonable, with 10,000 reads / sec, and some timer and lock calls probably. 4mb/sec is not much. small IOs? What tool did you use to see that?
80 - 90 physical IO/second is more likely to be more that 200KB/second but still low.

What are the usr:sys:idle ratios?
Little idle left, so you are cpu bound?

Short of fixing the 3rd party application, more CPUs are possibly needed. There are no tuning knobs for the application? Bigger IOs? larger internal cache?

It can certainly be appropriate to have an odd number of CPUs. But if you are going to order something and open the box and take the downtime then adding a pair and really making a difference in potential performance might be more effective. If your software has a per cpu license (Oracle?) then you coudl add it in and not use it (for now).

Also be sure to add more memory as you add CPU power. That is likely to be needed.
Now adding memory often needs to be done in pairs or even quads to fil 'banks' and get balanced interleaving. For CPUs that is not the case in designs I am familiar with.

You did not mention the box, some systems come with 2 CPUs per board/cell/chips giving you no choice.

Good luck,
Hein van den Heuvel
HvdH Performance Consulting.


Acclaimed Contributor

Re: Dealing with big number of system calls

Normally when one sees a large number of read() system calls that means that the application is not taking advantage of buffering and because you are probably i/o bound, adding CPU's either in one's or two's is probably going to make minimal difference. What is really needed is a rewrite of part of the application.
If it ain't broke, I can fix that.
Honored Contributor

Re: Dealing with big number of system calls

What system calls are the predominant ones?? Since your phys I/O is far too small compared to the logical I/O you need to isolate the system calls that are being executed majority of the time.

~cheers
Honored Contributor

Re: Dealing with big number of system calls

The only thing that can be done is to fix the application. HP-UX is doing what it was told to do. The high system call rate (and low physical disk I/O rate) indicates that the application is probably reading very small records and the vast majority of the records are already in memory via the buffer cache. So this is a good thing compared to trying to read small records directly from disk.

More processors will help. The quantity that you can add is very dependent on what computer you have. However, with 12 in the run queue, a better number might be to add 4 or even 6 processors. The idea here is process the cache in parallel to help with the poor programming.


Bill Hassell, sysadmin