Only half CPUs being fully utilized

[Kenneth Penland]

On one of our systems our DBAs are complaining about how their database is running extrememly slow. after taking a look at top/glance/vmstat we all see something odd that I cannot explain. any thoughts as to why all 8 cpus appear to be "used" but only 4 are getting hit hard, while the other 4 seem to be practically idle?

attached is a txt output from the top command.

[Torsten.]

It would be nice to know what server model / what CPUs / what OS you are using.

Hyperthreading enabled?

[Kenneth Penland]

rx7640 running 11.23

[Dennis Handly]

Can you check the rest of the top(1) output and see which processes are running on those 4 busy CPUs?

If you aren't using threading, 4 processes can only use 4 CPUs.

[Kenneth Penland]

sorry, it took me a minute to remember the command to get additional information:

Number of CPUs = 8
Clock speed = 1598 MHz
Bus speed = 533 MT/s
CPUID registers
vendor information = "GenuineIntel"
processor serial number = 0x0000000000000000
processor version info = 0x0000000020010104
architecture revision: 0
processor family: 32 Intel(R) Itanium 2 9100 series
processor model: 1 Intel(R) Itanium 2 9100 series
processor revision: 1 Stepping A1
largest CPUID reg: 4
processor capabilities = 0x0000000000000005
implements long branch: 1
implements 16-byte atomic operations: 1
Bus features
implemented = 0xbdf0000020000000
selected = 0x0020000000000000
Exclusive Bus Cache Line Replacement Enabled

Cache info (per core):
L1 Instruction: size = 16 KB, associativity = 4
L1 Data: size = 16 KB, associativity = 4
L2 Instruction: size = 1024 KB, associativity = 8
L2 Data: size = 256 KB, associativity = 8
L3 Unified: size = 9216 KB, associativity = 9

Memory = 16249 MB (15.868164 GB)

Firmware info:
Firmware revision = 9.048
FP SWA driver revision: 1.18
IPMI is supported on this system.
ERROR: Unable to obtain manageability firmware revision info.

Platform info:
model string = "ia64 hp server rx7640"

OS info:
sysname = HP-UX
release = B.11.23
version = U (unlimited-user license)
machine = ia64
@(#) $Revision: vmunix: B11.23_LR FLAVOR=perf Fri Aug 29 22:35:38 PDT 2003 $

[Kenneth Penland]

Okay, here is top with more info...basically the question they are asking is why are all of these just using cpu 1-4 and skipping 0, 5-8?

[Dennis Handly]

>the question they are asking is why are all of these just using CPU 1-4 and skipping 0, 5-7?

Good question.
All of the ora_s*_dlcdbap and ora_s*_dlcdbap processes seem to be on 1-4.

Are you using any workload manager? Or fairshare scheduler?

Or is Oracle using mpctl(2)?

[Kenneth Penland]

I do not believe so....the common consensus between admins here are that there are just some "big processes" that happened to of started on 1-4...and those are plugging the pipe so to speak on those 4 CPUs, while the other CPUs are able to keep up... is there a way to tell if there is some sort of workload manager at work on the box? I dont think there is cause it would have to of been placed on the box, and I know I didnt do it!

[Dennis Handly]

>the common consensus between admins here are that there are just some "big processes" that happened to have started on 1-4 and those are plugging the pipe so to speak on those 4 CPUs, while the other CPUs are able to keep up.

top(1) says they are idle, nothing to "keep up". The ones that are busy are mostly in the kernel (SYS).

>is there a way to tell if there is some sort of workload manager at work on the box?

Since they cost money, it should be obvious. :-)
Do you see anything obvious with:
swlist -l bundle

[Dennis Handly]

I should have asked, do you have glance to see what those processes are doing?
You could also try caliper, it's free.