L1000 system performance mystery

[Rudy Williams]

We have an L1000 with 1 CPU and 1 GB of RAM running Progress. The following was found while analyzing data gathered with "vmstat 5" over the course of 1.5 hours:

-There is no memory bottleneck.
-There is no I/O bottleneck.

The following started at 10:33 AM and continued until the data collection stopped:

-The load average dropped from 4.6 to 4.3
-CPU idle time dropped from 60-100% down to a flat 0 and stayed there
-The CPU was in user mode for 80-90% of the time (up from 0-30%)
The CPU was in system mode for 10-20% of the time (up from 0-5%)
-Paging activity was light with 0 pageouts
-No more than 1 blocked process at any one time (same as before 10:33)

Any idea what can cause the load average to remain relatively unchanged yet the CPU idle time to drop to a flat zero?

[A. Clay Stephenson]

Vmstat is not my weapon of choice for this; about the only column of vmstat that I ever pay attention to is the page-out rate. Glance is a much, much better tool for this and you could drill down into the busy process. You can install the fully-functional Trial Version from any application CD set.

The high fraction of user mode really suggests that this is probably application code that is CPU intensive. Possibly very tight loops or stored procedure executions -- but these are just guesses. What has changed recently in the application code and/or SQL?

One other possibility (since this is database stuff) is very high logical i/o (either in UNIX buffer cache or in the database cache) rates but with low physical i/o rates -- because most (or all) of the data are cached. Glance can pinpoint this sort of thing very quickly.