RP7410 machine slowing down

[Jeff Hagstrom]

Today my RP7410, running 11.00 slowed to a crawl. There weren't any big process running from our application, just normal everyday processing. It lasted for a 3-4 minutes. What could be the cause and are there any commands to look at what's going on with the box, such as memory or space or heavy processor user? We don't have glance or any other tools that I know about on the machine.

[Sยภเl Kย๓คг]

Run the following commands and post the o/p

#top -h
#sar -d 5 5
#swapinfo -tam

[Jeff Hagstrom]

System: rp7410 Fri Jul 20 15:18:18 2007
Load averages: 2.94, 3.14, 3.13
623 processes: 609 sleeping, 14 running
Cpu states: (avg)
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
2.94 4.0% 0.0% 4.0% 92.1% 0.0% 0.0% 0.0% 0.0%

Memory: 2191380K (959864K) real, 3136332K (1651180K) virtual, 61908K free Page#
1/45

[Jeff Hagstrom]

# sar -d 5 5

HP-UX rp7410 B.11.11 U 9000/800 07/20/07

15:21:50 device %busy avque r+w/s blks/s avwait avserv
15:21:55 c0t6d0 1.80 0.50 3 18 1.79 18.44
c6t0d0 3.59 0.50 11 147 3.87 4.62
15:22:00 c0t6d0 0.80 0.50 2 8 1.18 14.73
c6t0d0 0.40 0.50 5 54 1.88 1.73
15:22:05 c0t6d0 3.41 0.50 7 58 1.74 17.92
c6t0d0 0.40 22.85 23 314 4.87 2.79
15:22:10 c0t6d0 3.40 0.50 7 80 3.07 10.51
c6t0d0 1.20 0.50 16 229 2.88 1.55
15:22:15 c0t6d0 2.40 0.50 5 38 2.74 8.94
c6t0d0 0.20 1.17 8 85 1.43 1.58

Average c0t6d0 2.36 0.50 5 41 2.32 13.65
Average c6t0d0 1.16 8.74 12 166 3.51 2.56
#

[Jeff Hagstrom]

top
sar
swapinfo

[Sยภเl Kย๓คг]

Load averages: 2.97 is too bad.

Looks like "_progre" process is creating the mess,
look at the number of sleep processes in system. 606 processes: 593 sleeping, 13 running

May be an applcation bounce will help

[Jeff Hagstrom]

what is the load avg atrributed to?

[Patrick Wallek]

Load average, is just that, an average.

The load average is the average number of processes waiting on CPU, jobs in the run queue, over the interval it is figured for.

So with uptime, your intervals are 1, 5 and 15 minutes for the 3 values.

I don't know that load average of slightly less than 3 is necessarily bad. That just means that were an average of 3 items in the run queue during that period.

The real question is, what is the load average normally? If you don't know, then you can't say for sure if 3 is good or bad. If you usual load average is 0.25, then maybe 3 is bad.

[A. Clay Stephenson]

Rather than using "The Force" as a diagnostic tool, it makes more sense to gather some metrics. Look on any of your Applications CD sets and you will find a 60-day Trial version of Glance. I would load that. You should also start sar in data collector mode so that it grabs snapshots every 20 minutes or so so that you have some data to compare when things are good, bad, and plumb ugly.

[Bill Hassell]

There is nothing that will help now. The delay occurred in the past when no measurements were being made. You can look at /var/adm/syslog/syslog.log, specifically at the exact time of the slowdown. Also look at last and lastb output to see who may have logged in at that time. At the moment you took the measurements, the machine was 97% idle so nothing is wrong -- right now.

Do you have the diagnostics and EMS configured? Run this command:

echo "selclass qualifier memory;info;wait;infolog" | cstm

As Clay mentions, you need to get Glance running and start logging performance measurements. Next time things slowdown, you need a single script to run all the measurements and log them into a file.