cpu spends 50%+ of the time on nice

[HPP]

Hi,
I am having CPU bottelneck issue on L-Class (HP-UX 11.00) with 2 CPU, 2 GB memory. Running Oracle 8i Database, with 6 oracle session active.
When i ran gpm i see CPU is spending 50% of the time on nice, overall AVG CPU is at 97 to 100%. memory usage less than 65%, disk usage is less then 25% and swap usage is less than 25%. Also CPU usage for user proc is at 25%. Through gpm, i found out that system call rate for SIGPROCMASK is at greater than 50000.
Any idea why this is happening?

Just for comparision:
---------------------
We have got one more HP-UX 11.00 server on N-Class, 2 CPU, 4 GB memory and running Oracle 9i server. On this we have more than 15 session active out 50 sessions logged in. On this system NICE is at 0% and system call rate for SIGPROCMASK is less than 300.

Thanks in advance.

[harry d brown jr · Accepted solution]

Padmanabha,

Can you post your kernel parameters from your L-class? Did you happen to use the "template" to build your kernel parameters? Make sure timeslice is 10 and not 1!

live free or die
harry

[Sandip Ghosh]

According to your posting we could find that your memory usage is less than 65% and the swap usage is 25%. Swap should not be used before exhausting the memory. Have you checked about the kernel parameters of L Class with the N Class? I think your dbc_max_pct is configured at default 50%. If that is the case reduce your dbc_max_pct to 15% and look at the performance. Also please post the following things

Kernel Parameters of L class
swapinfo -tm
sar -b 2 20
sar -v 2 5
sar -q 2 10

Sandip

[HPP]

Harry, Thanks for the reply. I dont remember how i built the kernel. timeslice is set to 10. I have attached a file which contains the kernel paramter set in l-class.

[HPP]

Harry,
I have attached a file, which has the output of sysdef command.

[HPP]

Sandip Gosh,
Thanks for the reply.
I have already posted the kernel parameter for l-class. Here is the additional info you asked for:

Output of "sar -v 2 5"
-----------------------
13:45:21 text-sz ov proc-sz ov inod-sz ov file-sz ov
13:45:23 N/A N/A 217/8212 0 1874/9404 0 1516/15205 0
13:45:25 N/A N/A 214/8212 0 1871/9404 0 1509/15205 0
13:45:27 N/A N/A 214/8212 0 1867/9404 0 1501/15205 0
13:45:29 N/A N/A 216/8212 0 1864/9404 0 1499/15205 0
13:45:31 N/A N/A 220/8212 0 1862/9404 0 1507/15205 0

output of "sar -q 2 10"
-----------------------

13:45:35 runq-sz %runocc swpq-sz %swpocc
13:45:37 2.5 50 0.0 0
13:45:39 1.2 100 0.0 0
13:45:41 1.7 75 0.0 0
13:45:43 2.0 76 0.0 0
13:45:45 2.0 100 0.0 0
13:45:47 2.0 50 0.0 0
13:45:49 2.3 75 0.0 0
13:45:51 1.2 100 0.0 0
13:45:53 1.7 75 0.0 0
13:45:55 1.0 100 0.0 0

Average 1.7 80 0.0 0


Output of "sar -b 2 20"
------------------------
13:54:48 bread/s lread/s %rcache bwrit/s lwrit/s %wcache pread/s pwrit/s
13:54:50 20 2355 99 354 621 43 0 0
13:54:52 22 2490 99 12 411 97 0 0
13:54:54 14 2666 99 8 590 99 0 0
13:54:56 15 3172 100 111 472 77 0 0
13:54:58 4 3163 100 398 875 55 0 0
13:55:00 0 2400 100 562 878 36 0 0
13:55:02 5 2194 100 246 1092 77 0 0
13:55:04 13 3664 100 26 792 97 0 0
13:55:06 12 3238 100 12 620 98 0 0
13:55:08 13 2670 99 330 446 26 0 0
13:55:10 18 3456 99 2 828 100 0 0
13:55:12 10 3478 100 24 438 95 0 0
13:55:14 55 2544 98 158 570 72 0 0
13:55:16 18 2263 99 176 420 58 0 0
13:55:18 3 2914 100 2 653 100 0 0
13:55:20 0 1449 100 31 236 87 0 0
13:55:22 26 3197 99 30 1150 97 0 0
13:55:24 26 2422 99 38 712 95 0 0
13:55:26 42 3685 99 7 1281 99 0 0
13:55:28 49 2774 98 208 474 56 0 0

Average 18 2809 99 137 678 80 0 0

Oputput of "swapinfo -tm"
-------------------------
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 0 4096 0% 0 - 1 /dev/vg00/lvol2
reserve - 905 -905
memory 1518 873 645 58%
total 5614 1778 3836 32% - 0 -

Thanks

[harry d brown jr]

Padmanabha,

I'd make these changes:

Drive dbc_max_pct down to 10% from 25%.

Change bufpages & nbuf to 0.

And I would change maxdsiz & maxdsiz_64bit to 0x40000000 (1,073,741,824), maxssiz & maxssiz_64bit to 0x800000 (8,388,608).

What processes are getting "nice" time?

live free or die
harry

[Krishna Prasad]

Your dbc_max_pct does seem a little hight to me. It is at 25. I perfer 5% max when using Oracle and then I give Oracle a bigger SGA.

However, this doesn't explain why nice is so high. Nice is 0% on my systems, because we don't feel the need to run any process at a higher or lower process then normal, Somewhere you have process that are getting started using nice to change the run level different from the system default.

I suggest to look at all your start-up scripts to see how the majority of your programs are getting started. You can also run top and look for any process different then 20 in the NI field (5th field when using top). Then do a ps -ef | grep PID ( which you can see from top in the 3rd field.) Then once you have the program that is running using a different nice value you will need to find out how they get started.

[Sandip Ghosh]

From your posting of
1. swapinfo -tm
I could find that there is no swaping at all. So you don't have to think about that.

2. sar -b 2 20
It shows that the buffer what you have set is meeting your requirements. It may extra as per your requirements. From the sysdef posting I could find out that in the kernel you have set the bufpages and nbuf. Is there any specific reason/requirement to set the nbuf/bufpages to a fixed value? If not set the bufpages and nbuf as 0 and dbc_max_pct to 15% and see the performance. But dont forget to keep a hard copy of the kernel setting what you are having right now.

3. sar -q 2 10
It is showing the average runq size, which is not that bad.

4. sar -v 2 5
It shows the condition of the file tables.According to your posting I could find that, the no. of Inodes required are very less for the system. The present usage is around 1900 and in your system it has been set as 9404. You can reduce it to 3000 to 4000. Sometimes this oversized inode tbles load the CPU. I don't know whether your system is most active this time or not. If it file-sz is the peak usage time data then you can reduce it to 3000. Otherwise watch at the peak time and set it to peak time value +1000.

5. From the system file it apears that your shmmax is set very high for 2 GB system. Try to make it to 512MB for your system and see the performance.


Hope you can see some difference after changing those parameters.