High Run Queue on HPUX

[Karen Shen]

HPUX 11i. 16way superdome.

All of a sudden, we have been seeing a run queue that is a 1 or 2 count higher than normal. The run queue also stays elevated (.3 - .5) when no users are using the system. Previously, the load would be .5 or .6 under heavy load, now it's 2 or higher.

However, system performance seems fine, and CPU use according to glance is generally low, averaging between 20 - 50% during peak periods. We have plenty of free memory, and system activity has not increased.

The output from sar -q is:

09:55:27 runq-sz %runocc swpq-sz %swpocc
09:55:37 3.0 12 0.0 0
09:55:47 2.2 6 0.0 0
09:55:57 2.1 11 0.0 0
09:56:07 2.6 13 0.0 0
09:56:17 1.8 7 0.0 0
09:56:27 2.8 13 0.0 0
09:56:37 2.7 14 0.0 0
09:56:47 3.1 14 0.0 0
09:56:57 2.6 17 0.0 0

No changes have been made to the system.

Thanks in advance for any help.

[Stefan Farrelly]

Normally an artifically high run queue like this is caused by high i/o usage or an i/o problem (if cpu and memory are fine). What does sar -d show and wio% (just plain sar 1 10)

[Karen Shen]

Thanks for your quick response.

Output from sar -1 10

10:10:55 27 10 51 12
10:10:56 25 6 52 17
10:10:57 26 7 55 12
10:10:58 32 11 38 18
10:10:59 15 6 56 23
10:11:00 14 8 55 23
10:11:01 10 7 57 26
10:11:02 17 9 54 20
10:11:03 20 12 47 22
10:11:04 17 6 59 18

Average 20 8 52 19

Output (average) from sar -d 1 10:

Average c0t0d0 4.09 0.50 66 533 4.98 0.72

Average c0t1d0 5.08 1.89 14 299 8.31 8.36

Average c4t1d0 3.89 1.85 13 291 8.00 8.36

Average c10t0d0 84.95 0.52 102 2316 6.13 23.86

Average c10t0d1 85.84 0.50 119 2735 6.21 19.61

Average c10t0d3 69.19 0.50 63 1136 5.80 23.05

Average c10t0d4 75.17 0.54 85 2251 6.91 28.49

Average c10t1d6 1.00 0.50 1 23 4.65 11.52

Average c10t1d7 0.20 0.50 3 120 4.64 0.75

Average c12t0d0 87.54 0.51 106 2367 5.18 24.26

Average c12t0d1 86.44 0.51 126 2800 6.21 19.61

Average c12t0d3 67.10 0.50 62 1129 5.16 21.92

Average c12t0d4 73.08 0.59 84 2185 6.82 27.45

Average c12t1d6 0.20 0.50 0 5 1.26 14.59

Average c14t0d0 92.12 0.51 147 3054 7.03 19.82

Average c14t0d1 85.24 0.51 156 3500 8.44 14.73

Average c14t0d3 69.29 0.50 75 1380 8.44 19.22

Average c14t0d4 68.59 0.62 108 2696 9.21 21.17

Average c14t1d6 0.60 0.50 0 4 6.47 11.40

Average c14t1d7 0.70 0.50 7 180 5.50 0.79

Average c8t0d0 89.83 0.50 140 3016 7.21 19.01

Average c8t0d1 84.85 0.50 157 3358 7.04 15.15

Average c8t0d3 74.98 0.50 84 1571 6.44 18.86

Average c8t0d4 74.48 0.59 103 2635 7.77 21.63

Average c8t1d1 1.50 0.50 1 19 6.30 10.33

Average c8t1d6 0.20 0.50 0 8 2.50 8.92

Average c8t1d7 0.60 0.50 8 254 4.79 0.94

Average c0t2d0 0.40 0.50 1 5 4.94 5.53

Average c4t2d0 0.30 0.50 0 4 5.02 4.93

Average c4t0d0 0.30 0.50 1 6 3.81 2.94

Average c12t1d7 0.30 0.50 2 75 5.41 1.44

Average c10t1d1 0.80 0.50 0 5 2.01 28.53

Thanks again.

[Stefan Farrelly]

Youre averaging over 50 for wio% - thats a huge number showing you are completely I/O bound and this is causing your high load avg/run queue. Something is smashing your disks very hard indeed. Normal guideline is to keep this wio% avg below 10%, we keep ours this low. Not much point having fast cpu's and lots of memory if you cant utilise them becuause your disk subsystem cant keep up with I/O requests.

[Bharat Katkar]

I would suggest you to monitor the things in this order and rectify it in the same order:
1. Buffers sar -b
2. Memory vmstat/ sar -w
3. Disk I/o sar -d / iostat
4. CPU

[Karen Shen]

Thanks to all for their input.

The funny thing is, the system isn't doing any more IO than it normally does and no changes (supposedly) have been made to the IO configuration behind the scenes. We are running IO via an EMC SAN network.

Thanks again.

[Jeff Schussele]

Hi Karen,

High run queues in and of themselves are not totally indicative of performance problems. Something else needs to be tied to them - something as simple as poor response times or in your case high %WIO.

What *can* indicate performance trouble by itself is high priority queue values because that indicates processes are constantly getting bumped off the CPU before they can get their work done. But again you'll almost always find something else is found wanting (Memory, disk access times) unless it's truly solely CPU bound.

Try to spread the disk I/O load across multiple HBAs - is you have them - or at least multiple disks/LUNs.

Rgds,
Jeff

[Steven E. Protter]

A agree the fun queue issue can result from disk or i/o bottlenecks.

I'm attaching a background sar collection package that might help you pinpoint the trouble.

SEP

[Stefan Farrelly]

The funny thing is, the system isn't doing any more IO than it normally does and no changes (supposedly) have been made to the IO configuration behind the scenes. We are running IO via an EMC SAN network.

I doubt very much its anything to do with your SAN/EMC setup - its all host based. Some process(es) on your HP server are doing a lot more I/O than normal. You need to investigate further to find out who/what. Could be someone running an erroneously large report, a runaway application process, anything. Best thing to do is restart your apps and/or the server and see if wio% returns to normal (a lot lower than 52%!).

[Karen Shen]

Thanks again for your responses.

The funny thing is that the load average has been high fairly consistently for the last few days.

We track user and database activity, and while it's a bit higher than normal, isn't outrageous.

[Karen Shen_1]

And we also noticed that alarm history from Glance contains following:

MI trace buffer loss detected

Any idea what this "MI trace buffer" is and if this has anything to do with the high load?

[Jeff Schussele]

Hi Karen,

I suspect that's part of the midaemon (Measurement Interface daemon) that the system uses for glance, gpm & scope utilities to track performance. IF the system get's under a *heavy* I/O load it will dump some of these buffers to make sure they don't starve the applications that are doing the *productive* work.
You should check the following log
/var/opr/perf/status.mi
for confirmation & man midaemon for details.

HTH,
Jeff

[Ted Buis]

I wouldn't be too concerned about the load parameter if response time is okay, but the fact that it seems that you don't have a real baseline is something that you should correct. Measureware is great for establishing a baseline. The book "HP-UX Tuning and Performance" (now 2nd Edition to cover 11i) by Robert F. Sauers (Amazon.com) provides good overview of tools and methodology. Now is the time to measure your system, before you get to performance problems. If your I/O is really saturated, then any increase in user activity will cause the load and ultimately response time to soar (knee of the curve we used to say).
So I would say get a head start on the solution now, before users start screaming.