CPU Load Higher

[Ahmad Munawwar]

Hello,

I recently experiencing the CPU utilization too high that to some extend causing my application went down. I just wanted to know what would be your comment after looking this following print-out:-

CPU average idle time take from /var/adm/sa/sar file.

00:00:04 %usr %sys %wio %idle

00:15:03 42 24 31 3

00:30:03 57 23 18 2

00:45:03 61 23 14 1

01:00:03 45 24 28 4

01:15:04 43 23 31 4

01:30:03 25 21 43 11

01:45:01 17 21 57 6

02:00:03 19 21 55 4

02:15:06 18 21 58 3

02:30:02 34 22 41 4


Average 36 22 37 4


TOP command:-

last pid: 29206; load averages: 2.66, 2.25, 3.43 01:40:59
127 processes: 125 sleeping, 1 running, 1 on cpu
CPU states: 3.2% idle, 7.2% user, 11.9% kernel, 77.7% iowait, 0.0% swap
Memory: 4096M real, 59M free, 4432M swap in use, 2768M swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
20944 oracle 11 59 -20 1916M 1850M sleep 640:01 4.31% oracle
20668 oracle 10 59 -20 29M 8312K sleep 516:49 2.79% oidldapd
19000 root 66 59 -20 94M 24M sleep 101:15 1.12% java
20857 oracle 11 60 -20 1853M 1801M sleep 401:14 0.76% oracle
20847 oracle 11 59 -20 1881M 1848M sleep 629:20 0.74% oracle
11095 oracle 258 59 -20 1750M 1709M sleep 31:05 0.63% oracle
20954 oracle 11 59 -20 1880M 1848M sleep 622:45 0.61% oracle
25328 oracle 1 59 -20 1744M 1710M sleep 17.9H 0.56% oracle
20904 oracle 11 60 -20 1879M 1839M sleep 608:21 0.55% oracle
20830 oracle 1 60 -20 1902M 1792M sleep 398:02 0.54% oracle
20792 oracle 11 59 -20 1880M 1848M sleep 641:39 0.53% oracle
25337 oracle 1 59 -20 1744M 1708M sleep 18.7H 0.53% oracle
28690 root 1 59 0 2696K 1736K cpu/2 0:00 0.42% top
20874 oracle 1 59 -20 1901M 1818M sleep 408:48 0.41% oracle
663 root 1 0 0 1104K 808K sleep 535:35 0.40% lomset


last pid: 14026; load averages: 1.55, 1.91, 2.28 02:11:28
126 processes: 121 sleeping, 2 running, 1 zombie, 2 on cpu
CPU states: 0.9% idle, 9.7% user, 15.2% kernel, 74.2% iowait, 0.0% swap
Memory: 4096M real, 57M free, 4432M swap in use, 2769M swap free

PID USERNAME THR PRI NICE SIZE RES STATE TIME CPU COMMAND
20792 oracle 11 60 -20 1880M 1804M sleep 642:07 0.77% oracle
25337 oracle 1 59 -20 1744M 1708M sleep 18.7H 0.70% oracle
25328 oracle 1 59 -20 1744M 1710M sleep 17.9H 0.67% oracle
20823 oracle 1 60 -20 1808M 1761M sleep 190:08 0.63% oracle
20944 oracle 11 60 -20 1916M 1813M sleep 640:30 0.60% oracle
20857 oracle 11 60 -20 1853M 1810M sleep 401:44 0.56% oracle
20830 oracle 1 60 -20 1902M 1811M sleep 398:25 0.55% oracle
20904 oracle 11 60 -20 1879M 1820M sleep 608:51 0.54% oracle
663 root 1 20 0 1104K 808K sleep 535:51 0.49% lomset
20923 oracle 11 60 -20 1866M 1817M sleep 400:46 0.46% oracle
20670 oracle 10 59 -20 28M 7808K sleep 332:08 0.45% oidldapd
20779 oracle 11 59 -20 1839M 1794M sleep 317:34 0.44% oracle
20881 oracle 11 59 -20 1898M 1796M sleep 320:23 0.42% oracle
20946 oracle 11 60 -20 1848M 1813M sleep 399:31 0.34% oracle
20813 oracle 11 59 -20 1846M 1794M sleep 318:59 0.34% oracle

To resolve this, the applicationwas restarted. The above the only data captured during the events.

Would great if I you can share the opinion, and what would be the approach if the recur in the future.

Regards,
Munawwar

[Steven E. Protter]

You might want to collect more data over a period of time for better analaysis. See the script tools I'm attaching.

I note that there are a lot of sleeping processes listed.

I see problems with iowait. If you collect more data, you may find a disk bottleneck you can resolve.

Seems that your applications are waiting for disk i/o.

SEP

[RAC_1]

Do you have glance installed?? Start with glance. Is it just the CPU or anything else that also is causing the problem.

Just looking at top output gives feeling that oracle processes are causing problem.

glance -g
glance -a
glance -m

Also let us know about memory, network, swap and cpu usages from glance.

the following command will give the top 10 processes (cpu utilization)

UNIX95= ps -ef -o "pid, ppid, ruser, cpu" | sort -nrk4

Anil

[Ahmad Munawwar]

Hi,

Unfortunately, this machine is SUN Solaris. No Glance application running..emmm.

But attached is the vxstat result. If that help to understand the situation.

br,
Munawar

[RAC_1]

Into the wong forum. Not a sun guy. Just by looking at the vxstat gives feeling that the following volume is being hit hard. Seems a lot of reads and writes.

vol Oracle-DS2 29185252 13403874 2436301049 345687885 6.6 2.4

whatever is there on that, you may want to distribute the data over multiple volumes.

Anil

[Prashant Zanwar_4]

You are in wrong forum, but you can try..

/usr/ucb/ps aux | more

also

mpstat 3 30

and

prstat ...try these..

Thanks
Prashant

[Geoff Wild]

Did the application actually go down?
Is there anything in the application logs?
Or did you manually do it?

Any error(s) in /var/adm/messages? or /var/log/syslog?

To me, your cpu's seem fine...

What about:
vmstat -1 10

Any sr or de?

It does look more like a loaded disk...

Do you run Perfview in your environment?


Rgds..Geoff

[Ahmad Munawwar]

Hi Prashant,

It seem at that time I have a problem with ldap... there were a lot of established connection on port 38900 (see attachment)

and here is the process.

/usr/ucb/ps -auxwwww 20678
USER PID %CPU %MEM SZ RSS TT S START TIME COMMAND
oracle 20678 1.7 0.229256 7544 ? R Aug 26 436:04 oidldapd connect=mmsoid01 dispatcher=1 worker=4 control=20646 debug=16384 logchange=TRUE configset=1 instance=2 key=1339369838 host=ds-01

Hi Wild Pharaoh... I will try to get the vmstat.. if i still got it during the problem... get back to you.

My only Forum.. I used to be supporting HP-UX box before..till now but this SUN box..came recently.

br,
Munawwar

[Ahmad Munawwar]

Sorry...the attachment

[Prashant Zanwar_4]

you can find open files or the process tree even by using

/usr/proc/bin/pfile

/usr/proc/bin/ptree etc commands

just see if you get to know some anything suspicious and also...

ps -e -o cpu -o command you can try this and sort to find which is actually causing the problem. I dont have solaris running here, but all help is there in ps man page which you can customize for your use and find out the problem

Hope it helps
prashant

[Prashant Zanwar_4]

You can post your question at sunmanagers.org

Send sunmanagers mailing list submissions to
sunmanagers@sunmanagers.org

To subscribe or unsubscribe via the World Wide Web, visit
http://www.sunmanagers.org/mailman/listinfo/sunmanagers
or, via email, send a message with subject or body 'help' to
sunmanagers-request@sunmanagers.org

You can reach the person managing the list at
sunmanagers-owner@sunmanagers.org

Thanks
Prashant

[Ahmad Munawwar]

Hello Guys,

Thanks a lot for the invaluable info. Hope to talk to you again in near future.

br,
Munawwar

[Hein van den Heuvel]

Hmmm,

Seems that system is just low on memory during the problem time:

Memory: 4096M real, 59M free

The relatively high systems time versus user time is consistent with that, as is high IO load.

Is there a rogue memory user (those connections?) or is it overcommitted (the Oracle SGA appears to be 1800M).

fwiw,
Hein.

[Ahmad Munawwar]

Hi Hien,

They used CORBA application to do some data provisioning to a oracle database (adding, deleting , modifying etc) suddenly they complain the provisioning was too slow and found out that the CPU idle very low as above.

Attached is the sysstat during the event hope that give more indication of the cause.

[Sridhar Bhaskarla]

Hi Munawwar,

From your first sar output, you can see %wio more than 35%. It's an indication of probable disk bottleneck. May be there were some wild full table scans. Check with your DBAs on what they could see in the database at that time.

Run your 'sar' again when you think the application is running fine and compare the %wio with the previous one.

-Sri

[Ahmad Munawwar]

Hi,

Here is the sar file taken from the normal situation.

Which document that I ca refer to understand and be able to make a conclusion on the sar data?

br,
Munawwar

[Tim D Fulford]

Hi, I know I'm comming in late, but a few things that may be worth mentioning...

Ignoring that it is a SUN box, just looking at principles...

1 - The system is a 1 CPU system running an oracle instance. Th CPU is up to 80% utlised
2 - The disks seem to be a bit of bottlenack (as mentioned earlier)

By removing the disks bottleneck, you are likely to put the next bottleneck back on-to the CPU, as you are already using up to 80% (15 minute sample implies 100% at some points) I'd say you ave at least two bottlenecks to sort.

On the measurement side of things, OV PerformanceAgent (nee MeasureWare) is available for SUN, and this forum does answer questons on that subject!!!

I think there are really only two courses of action.
1 - buy more and re-arrange the data on the disk & upgrade or install more CPU
2 - imprive the way the sql/psql runs on the system such that it requires less CPU & disks activity.

The above may seem a little usless, but tuning the system will only get you so far. I think you are seriously pushed for resource and need to respond accordingly, either by getting more resource or by useing less. BTW the second option generaly does not happen, as getting someone to say "well I guess I could have done it better, I'll improve ypur code for free" happens rarely.

Regards

Tim