Server Performance Analysis

[PROSANJIT MALLICK]

Hi everybody,

I am having problem in pinpointing the exact reason of slow response of my
Database Server.
Below are the output of top and Informix command 'onstat'
It shows System utilisation 50% and User utilisation as 25%. Does this means
that my system is using up all my system resources. What is the exact relation between the user usage and system usage ? How can I bring down the System Utilisation ?

System: himalaya Sat Aug 19 11:53:30 2000
Load averages: 3.49, 2.81, 2.64
116 processes: 104 sleeping, 12 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 6.81 16.8% 0.0% 12.1% 71.1% 0.0% 0.0% 0.0% 0.0%
1 1.39 28.9% 0.0% 67.3% 3.8% 0.0% 0.0% 0.0% 0.0%
2 2.61 37.2% 0.0% 59.2% 3.6% 0.0% 0.0% 0.0% 0.0%
3 3.13 26.1% 0.0% 68.9% 5.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 3.49 27.3% 0.0% 51.9% 20.8% 0.0% 0.0% 0.0% 0.0%

Memory: 36564K (32696K) real, 101476K (96472K) virtual, 13620K free Page# 1/13

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 rroot 2193 informix 154 20 8060K 780K sleep 216:04 89.61 89.45 oninit
1 rroot 2192 informix 180 20 8060K 780K run 203:07 87.82 87.67 oninit
3 rroot 2190 informix 180 20 8060K 792K run 215:29 81.15 81.00 oninit
0 rroot 2194 informix 154 20 8060K 816K sleep 118:54 29.24 29.19 oninit
2 rroot 2046 root 20 20 9544K 8320K sleep 11:21 2.01 2.01 cmcld
2 rroot 1184 root 154 20 404K 92K sleep 5:47 1.03 1.03 inetd
0 rroot 2 root 128 20 0K 0K sleep 3:27 0.85 0.84 vhand
0 rroot 34 root 152 20 0K 0K run 0:41 0.78 0.78 vxfsd
0 rroot 357 root 154 20 32K 40K sleep 4:11 0.75 0.74 syncer

Informix Dynamic Server Version 7.31.FC4 -- On-Line -- Up 11:22:03 -- 3220672
Kbytes

Segment Summary:
id key addr size ovhd class blkused blkfree
97286 1384728577 c0000000001dd000 299040768 5624 R 36495 9

7 1384728578 c000000011f0d000 1610612736 25208 V 196608 0

8 1384728579 c000000071f14000 1073741824 17016 V 129920 1152
1033 1384728580 c0000000b1f14000 104857600 2232 V 11637 1163
10 1384728581 c0000000b8314000 104857600 2232 V 7760 5040
1035 1384728582 c0000000be714000 104857600 2232 V 1360 11440
Total: - - 3297968128 - - 383780 18
804

(* segment locked in memory)

Please help..
Thanks in advance.

[Rita C Workman]

System CPU means just that....resources being used by system to complete a process. User would be user application, etc....And based on those percentages it would "seem" way out of 'kilter'. But it looks like you supplied the info from TOP. And remember that TOP is just a snapshot of that moment. So the percentages may for that particular moment be ok....
What you want is to find what everything is doing, while it's running it...and than keep an overall performance average/stats.
One way is to run Glance or Glance+. Set up your mwa performance options in the parm file; in so doing you can 'seperate' out your Oracle processes; OmniBack; Special Applications; or whatever you need to monitor to see who or what is showing as impacting the system.
There are other monitoring tools...but I think what your needing is to set some sort of monitoring tool in place.
Hope this helps...

[Steve Sauve]

Hi,
As Rita said you really need to get a look at your system over time. One useful way of doing this is the sar command. You can setup scripts to monitor what is going on and dump it to a file. You can then graph/manipulate the numbers in a spreadsheet program. (basically the poor mans Measureware). This is a link to a previous forum question. I have a couple scripts attached to my answer that may be helpful to you.

http://my1.itrc.hp.com/cm/QuestionAnswer/1,1150,0x31a583667c40d4118feb0090279cd0f9,00.html

Hope that helps,
Steve

[Anthony deRito]

Prosanjit, when you see process spending a lot of time in kernel mode, it is important to determine what is actually happening while running your processes and what resources are the processes waiting for. It does not have to be the disk or the CPU.

Check vmstat. Look at the procs colums (the r,b&w colums). This will tell you how many processes are currently in the run queue and how many of them are blocked for resources.

If you notice many processes blocked on resources, use the Process List window of GlancePlus and check out the process wait states. You should be able to see what type of events the process is blocked on. Also, use GlancePlus in the Process Resource screens to find out what types of system calls are being made.

Use swapinfo -tma and check out your swap figures. use the PCT USED column and see if any number are high. Your primary swap devices should all read very low PCT USED numbers. If these are high (>50%) then you are experiencing some page out activities.

If your are using sar -d, do not be concerned about high disk utilization so much as to concentrate on the application and how much time the process is spending in kernel or system mode.

If you suspect that it is I/O releated, compare the service time with the wait time of each request to your disk. Use sar -d for these figures.

Also, if your doing filesystem data transfers, look at how well your utilizing your buffer cache. Use sar -b for this.

Read the man pages for all of the utilities mentioned.

Hope this info helps.

Tony