Server Performance Analysis

[PROSANJIT MALLICK]

Hi everybody,

I am having problem in pinpointing the exact reason of slow response of my
Database Server.
Below are the output of top and Informix command 'onstat'
It shows System utilisation 50% and User utilisation as 25%. Does this means
that my system is using up all my system resources. What is the exact relation between the user usage and system usage ? How can I bring down the System Utilisation ?

System: himalaya Sat Aug 19 11:53:30 2000
Load averages: 3.49, 2.81, 2.64
116 processes: 104 sleeping, 12 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 6.81 16.8% 0.0% 12.1% 71.1% 0.0% 0.0% 0.0% 0.0%
1 1.39 28.9% 0.0% 67.3% 3.8% 0.0% 0.0% 0.0% 0.0%
2 2.61 37.2% 0.0% 59.2% 3.6% 0.0% 0.0% 0.0% 0.0%
3 3.13 26.1% 0.0% 68.9% 5.0% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 3.49 27.3% 0.0% 51.9% 20.8% 0.0% 0.0% 0.0% 0.0%

Memory: 36564K (32696K) real, 101476K (96472K) virtual, 13620K free Page# 1/13

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
2 rroot 2193 informix 154 20 8060K 780K sleep 216:04 89.61 89.45 oninit
1 rroot 2192 informix 180 20 8060K 780K run 203:07 87.82 87.67 oninit
3 rroot 2190 informix 180 20 8060K 792K run 215:29 81.15 81.00 oninit
0 rroot 2194 informix 154 20 8060K 816K sleep 118:54 29.24 29.19 oninit
2 rroot 2046 root 20 20 9544K 8320K sleep 11:21 2.01 2.01 cmcld
2 rroot 1184 root 154 20 404K 92K sleep 5:47 1.03 1.03 inetd
0 rroot 2 root 128 20 0K 0K sleep 3:27 0.85 0.84 vhand
0 rroot 34 root 152 20 0K 0K run 0:41 0.78 0.78 vxfsd
0 rroot 357 root 154 20 32K 40K sleep 4:11 0.75 0.74 syncer

Informix Dynamic Server Version 7.31.FC4 -- On-Line -- Up 11:22:03 -- 3220672
Kbytes

Segment Summary:
id key addr size ovhd class blkused blkfree
97286 1384728577 c0000000001dd000 299040768 5624 R 36495 9

7 1384728578 c000000011f0d000 1610612736 25208 V 196608 0

8 1384728579 c000000071f14000 1073741824 17016 V 129920 1152
1033 1384728580 c0000000b1f14000 104857600 2232 V 11637 1163
10 1384728581 c0000000b8314000 104857600 2232 V 7760 5040
1035 1384728582 c0000000be714000 104857600 2232 V 1360 11440
Total: - - 3297968128 - - 383780 18
804

(* segment locked in memory)

Please help..
Thanks in advance.

[Philip Chan_1]

That system utilization rate is referring to the percentage of CPU spent on system calls. These system calls may be in turn triggered by your user process(es). My suggestion is to first locate what user process(es) are using up the CPU then find out which system call they made. I've seem a case where a database systems kept looping on the 'select' call for nowait I/Os, and that the server process just loop'ed too frequent hence system utilization rate increased to a high percentage as a result. So, watch out for your Informix and some tuning on it may be required.

Last time I used glance to find out what the problem was. If you got glance in your place then just give it a try.

Rgds,
Philip

[Tim Malnati]

You have 20% CPU idle, suggesting that CPU is not your bottleneck. I noticed that you only have 32MB of memory though. This seems VERY low in any sort of database environment.

[CHRIS_ANORUO]

You have a memory bottleneck. With 32MB you don't expect the system to fly. The system is doing a lot of work tying to free and allocate memory to processes thereby 51.9% usage on the report. A good system should report system % as 3/4 of user % (ie 75%user to 25%system )on at a pick period. Try and up grade to 64MB or 128MB.

[Bill Hassell]

Actually, 128 megs would be way too small. This is a 4 processor system. It's going to be very difficult to utilize all the CPU power with such low RAM.

I would suggest 1000 to 2000 megs of RAM for this system. You may need to adjust some Informix and HP-UX kernel parameters to take advantage of the additional RAM. For 4 processor database servers, most sysadmins run 1000 to 4000 megs on 10.20, 6Gb to 12Gb on 11.0 64 bit systems.

[Doug_3]

PROSANJIT,
You appear to have three issues: tools to use for assessing performance, server performance, and informix performance. I am not an expert on systems admin...so

Specific to informix, go to the informix user group www.iiug.org. Do a search under performance for massive amounts of information. To monitor performance most folks use the command line (onstat). However, some use the built in tools and there are a couple 3rd party tools if you search the informix user site.

Sign up on the iiug.org list serve, ask your performance question and post the following (or you can just email the group: informix-list@iiug.org with the following):
- onconfig file
- h/w specs especially your drive and raid setup
- onstat -iov, -p, -F -g seg, -g iov
- is KAIO on or off?
- have you run update statistics at all? or run it high, low, medium?
- is it an oltp or dss database, number of users, etc.

From the info you did post I can see your shared mem is probably not high enough. shmvrtsize affects how many shared mem segments you have and you should only have one to two, it appears you have at least 5, which means that informix is dynamically adding memory when it needs it.

hth
Doug