Strange use of resources

[Tonatiuh]

Oracle Application Server 10g R1 / BI and Forms instance / Forms and Reports application. In Red Hat Enterprise Linux 3.0.
-
I have released to production my app and it works properly. But when workload of server grow up, it shows a very strange behavior.

The (linux) "top" command, shows that more thatn 60% of the resources (specially CPU) are used by "system" and a very little percentage is used by "user". The server become extremly slow when it happen.

Any idea about the problem?

[Ivan Ferreira]

More information is needed. You can post your vmstat output, iostat output or sar output.

Make sure you don't have files other than oracle's on the OGFS shared filesystem (if you use it).

High system time usually means that the system is busy doing I/O related tasks, like paging. The vmstat may help you to determine the paging activity.

[Tonatiuh]

[oracle@ias97 oracle]$ iostat 2 5
Linux 2.4.21-4.EL (ias97.lc.org) 07/11/2005

avg-cpu: %user %nice %sys %idle
1.58 0.00 1.38 97.04

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 2.29 3.68 41.53 921182 10405714

avg-cpu: %user %nice %sys %idle
0.00 0.00 0.00 100.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 27.14 0.00 502.51 0 1000

avg-cpu: %user %nice %sys %idle
10.50 0.00 44.00 45.50

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 0.00 0.00 0.00 0 0

avg-cpu: %user %nice %sys %idle
0.00 0.00 0.00 100.00

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 25.50 0.00 376.00 0 752

avg-cpu: %user %nice %sys %idle
0.51 0.00 0.00 99.49

Device: tps Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn
dev8-0 0.00 0.00 0.00 0 0

--------


[oracle@ias97 oracle]$ vmstat 2 5
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy id wa
1 0 0 449048 117536 502384 0 0 2 21 120 88 2 1 97 0
1 0 0 450112 117536 502388 0 0 0 0 250 655 46 54 0 0
2 0 0 451196 117536 502424 0 0 0 142 222 693 45 55 0 0
0 0 0 453604 117536 502428 0 0 0 0 350 917 30 67 2 0
13 0 0 454696 117536 502428 0 0 0 184 197 622 2 1 97 0

[Stuart Browne]

I think we're also going to need a bit more information on what sort of system you're using this on:

CPU's
Memory
disk subsystem (SCSI/IDE, RAID? How many disks and how fast are they? What controller?)

Also, what else is running on the system at the same time as OAS?

How many users accessing it when this starts happening? 100? 200?

[Tonatiuh]

1) CPU's: 1 Intel P4
2) Memory: 2 GB
3) disk subsystem: IDE (known rpm)
4) What controller: IDE Controler
5) what else is running on the system at the same time as OAS?: Nothing else. Just Op. Syst. and OAS.
6) How many users accessing it when this starts happening? 100? 200?: Problem presents from 35 users and up (not constantly, just for moments).

[Stuart Browne]

System using an inordinate amount of resources? Well, it's using an IDE IO subsystem. This isn't unusual.

If anybody does anything with a reasonable amount of IO (running a report query which thashes the disk a bit), it's going to seriously hinder everybody else.

The throughput out an IDE channel is, well, miniscule. As IDE requires the CPU to do a fair amount of the processing, this means that whenever there is any decent amount of disk activity, the machine's performance will drop like a stone!

30+ users doing SQL queries against an IDE disk before you get serious degredation, you're getting amazing performance already.

Now, a bit to why it's thrashing so badly. In the stats you showed us, there was no swap activity, which means that you've got enough memory, and haven't over-tuned the system. This is good. So all that's left is for you to look through your logs and see if large queries coincide with the performance drops.

[Tonatiuh]

No queries involved in this. This is only an Application Server, not a database server. The database server is another server.

Same application in older version or Application server was working several years in another (windows) server with less resources (smaller CPU, less memory, etc.). So that, the capacity of the server should be enough for this application, I guess.

[Stuart Browne]

Well, find out what was happening at the time of these performance spikes. There will be some correlation.

For instance, if the SQL server dumped a whole packet load of data on the OAS, and the Application had to do something to it (sort it, change it, whatever), then it could be a cause.

[Ivan Ferreira]

The iostat you posted is not helpfull but the vmstat show a few lines with cpu 0 and high system time.

The vmstat command does not show bi and bo, so you are not paging. So the problem should not be memory related.

Another thing is that you don't have I/O wait on the vmstat output (wa), so I/O should not be the problem (disk/network).

You may need to obtain oracle statistics for I/O to confirm this.

You should verify the queries, indexes, etc.

[Tonatiuh]

You are right Ivan, no memory or I/O problems are found.

Queries, indexes, etc. could use resources but about the database servers. This is not a database server.

The only one thing that resides here is the application, and all the database operations are performed in anoter server.

[Ivan Ferreira]

The high amount of System time would
then be just another symptom of the real problem which, lies in the
application logic 99.99% of the time.

[Tonatiuh]

Yes Ivan, there is very obvious that there is a problem, and it could be in the Oracle software, but there is nothing where to begin.

I want to know any kind of method, free tool or something to search and analyze the processes that are using "system" resources, and with this info I could have a better idea about the source (maybe in Oracle software) of the problem.