System Bottleneck: RAM, Disk, or CPU?

[Adam Stahl]

I am a linux admin recently given an HP-UX 11.00 box to administer. There seems to be some type of bottleneck on the system and I can not determine if adding more RAM or CPUs would help. Current specs:
550 MHz CPU (x2)
2 GB RAM

Ouput of vmstat:
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
100 1 0 411959 4395 11 1 4 0 1 0 0 270 89853 21704 51 49 0
97 0 0 337798 4395 9 0 4 1 0 0 0 295 88024 21731 59 41 0
97 0 0 337798 4404 7 0 7 3 1 0 0 298 87292 21607 54 46 0
95 0 0 337649 4663 5 0 5 2 40 0 0 305 86891 22059 54 46 0
95 0 0 337649 4637 6 0 7 4 25 0 0 335 84096 21069 60 40 0

The CPU is obviously pegged because "id" is 0. However some paging is going on so I dont know if adding more RAM would help. Here are the top processes from "top":
CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 19908 egate 152 20 4452K 6056K run 246:12 16.17 16.14 stciqmgrd.exe
0 ? 19934 egate 152 20 2916K 3864K run 237:30 14.51 14.49 stciqmgrd.exe
1 ? 6219 egate 152 20 7780K 8744K run 914:46 13.81 13.79 stciqmgrd.exe
1 ? 6300 egate 152 20 56708K 22468K run 826:01 11.92 11.89 stcdgwtcpiphl7.exe
1 ? 6223 egate 152 20 3044K 3720K run 533:06 11.79 11.77 stciqmgrd.exe

From looking at the vmstat and top output can anyone help me figure out the bottleneck? Again, sorry I am such a newbie at HP-UX

[Court Campbell]

Just from a quick glance I would say that you need either more processors or faster processors. More memory is always good as it is cheap. You might want to post some sar -d output as well. by the way, vmstat, sar, iostat, etc really are not the best measurement tools. I highly recommend the glanceplus pak, or whatever they are calling it these days.

[Tommy_6]

Would you put the output of these commands to a file and attach them to this forum. This will help with the formatting and make it easier to read.

[Adam Stahl]

I've attached a file so the output can be read easier.

[Tommy_6]

By the way, how much swap do you have?

[Adam Stahl]

I have 5120MB of swap.

[Tommy_6]

I also agree with Count, try running GlancePlus, if you have it installed. Make sure you have you display set and run the following:

/opt/perf/bin/gpm

Hope this helps!

[Adam Stahl]

# /opt/perf/bin/gpm


This trial software has expired.

^^^ I guess our company didnt purchase it.

[A. Clay Stephenson]

I suspect that your real problem lies in your application. You probably need to look at the STC website for eGATE support. If memory serves, STC was acquired by Sun but I suspect that your system is being swamped by the application. You are doing a little bit of paging out but from the STC software I've seen in the past, it was very CPU intensive especially if badly configured.

[Adam Stahl]

Thanks A. Clay

I wondered if the app was the problem and not configured correctly. Our developers have contacted SeeBeyond and they are looking into the issue as well. I just wanted to make sure my box was beefy enough to run the app and all of the interfaces.

[Hein van den Heuvel]

Memory is ok for now.
Cpu is the first bottleneck here.
Focus on that application first though!

There is a disproportional CPU usage in system mode.
It corresponds with a high rate of system calls. Now those might relate to good honest work, or thay may indicate a poorly tuned application, with perhaps some components competing instead of collaborating (high context switch rate).

Are those 'stciqmgrd.exe' processes doing real honest business processing work, or are they more monitor/schedule components.

Can you tune the number of them? Allthough you are probably suffering from poor performance and would want to schedule more of them to speed stuff up, the underlying CPU resource is exhuasted, so adding more would just create more overhead.
Can you reduce the number of those processes, untill some idle time starts to appear (add one back). As you reduce, do you see a better user:sys:idle ratio?

If you decide that more CPUs need to be added, then please realize that you probably want to also add more memory. Without detailed other info I would stick to at least 1GB per CPU as is configured.

Good Luck!
Hein van den Heuvel
HvdH Performance Consulting