%runocc is 100 in sar -q output

[Vanja]

Hi,

I am seeing this issue on 2 rx1620 servers and 1 rx2620 server. All of these are 1 cpu boxes. They are running HPUX 11.23 with VXVM 4.1 There is really nothing running on these boxes - this is pretty much bare OS.

1) Is this normal behavior?
2) How can I find out what is causing this 100%? I have glance but not sure if that would help.
3) Can/Should I fix this - ie. is there a kernel tunable parameter or patch that I'm missing here?

Any help will be appreciated!
Below is output of sar and top from one of the rx1620 systems.

Thanks!

# sar -q 5 5

HP-UX cif9oh00 B.11.23 U ia64 05/19/06

03:46:41 runq-sz %runocc swpq-sz %swpocc
03:46:46 3.2 100 0.0 0
03:46:51 3.2 100 0.0 0
03:46:56 3.2 100 0.0 0
03:47:01 3.0 100 0.0 0
03:47:06 3.2 100 0.0 0

Average 3.2 100 0.0 0
# sar -q

HP-UX cif9oh00 B.11.23 U ia64 05/19/06

00:00:00 runq-sz %runocc swpq-sz %swpocc
00:10:00 4.1 100 0.0 0
00:20:00 4.1 100 0.0 0
00:30:00 4.1 100 0.0 0
00:40:00 4.1 100 0.0 0
00:50:00 4.1 100 0.0 0
01:00:00 4.3 100 0.0 0
01:10:00 4.7 100 0.0 0
01:20:00 4.4 100 0.0 0
01:30:00 4.5 100 0.0 0
01:40:00 4.4 100 0.0 0
01:50:00 4.4 100 0.0 0
02:00:00 3.3 100 0.0 0
02:10:00 3.2 100 0.0 0
02:20:00 3.2 100 0.0 0
02:30:00 3.1 100 0.0 0
02:40:00 3.2 100 0.0 0
02:50:00 4.5 100 0.0 0
03:00:00 3.7 100 0.0 0
03:10:00 4.5 100 0.0 0
03:20:00 4.5 100 0.0 0
03:30:00 3.6 100 0.0 0
03:40:00 3.1 100 0.0 0

Average 4.0 100 0.0 0
# sar -u

HP-UX cif9oh00 B.11.23 U ia64 05/19/06

00:00:00 %usr %sys %wio %idle
00:10:00 0 1 0 99
00:20:00 0 1 1 99
00:30:00 0 1 0 99
00:40:00 0 1 1 99
00:50:00 0 1 1 99
01:00:00 0 1 1 99
01:10:00 0 1 1 99
01:20:00 0 1 1 98
01:30:00 1 1 1 97
01:40:00 0 1 1 99
01:50:00 0 1 1 98
02:00:00 1 1 1 97
02:10:00 0 1 1 98
02:20:00 0 1 1 99
02:30:00 0 1 1 98
02:40:00 1 1 1 97
02:50:00 0 1 1 98
03:00:00 0 1 1 98
03:10:00 0 1 1 99
03:20:00 1 1 1 97
03:30:00 0 1 0 99
03:40:00 0 1 0 99

Average 0 1 1 98
# more /tmp/topout.out
System: cif9oh00 Fri May 19 03:46:23 2006
Load averages: 0.04, 0.04, 0.04
114 processes: 86 sleeping, 28 running
Cpu states:
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0.04 0.0% 0.0% 0.0% 100.0% 0.0% 0.0% 0.0% 0.0%

Memory: 285204K (162068K) real, 535712K (363980K) virtual, 74948K free Page# 1/
8

TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
? 58 root 152 20 10944K 9728K run 590:37 0.39 0.39 vxfsd
? 1648 root 152 20 29076K 4048K run 3:19 0.26 0.26 rpcd
? 112 root 152 20 936K 832K run 1:48 0.22 0.21 vxiod
? 2069 root 152 20 97092K 16380K run 19:15 0.20 0.20 vxsvc
? 2188 root 152 20 48344K 9100K run 5:11 0.18 0.18 opcctla
? 1906 root 152 20 25228K 3152K run 1:24 0.14 0.14 swagentd
? 8504 root 152 20 37048K 10812K run 4:10 0.12 0.12 opcmsga
? 552 root 152 20 7820K 1940K run 7:36 0.10 0.10 utmpd
? 8510 root 152 20 32460K 2684K run 7:40 0.06 0.06 opceca
? 49 root 152 20 216K 192K run 28:34 0.06 0.06 schedcpu
? 1656 root 152 20 56020K 6040K run 64:31 0.06 0.06 cimserver
? 0 root 127 20 72K 64K sleep 2:22 0.04 0.04 swapper
? 1 root 152 20 1960K 548K run 6:07 0.04 0.04 init
? 9 root 152 20 864K 768K run 8:09 0.04 0.04 nfsktcpd
? 18 root 191 20 144K 128K run 8:55 0.04 0.04 ksyncer_daemon
? 51 root 191 20 144K 128K run 0:03 0.04 0.04 cmcd
#

[RAC_1 · Accepted solution]

First you need to check what is the run queue. Following glance command will tell you about priority run queue on your system. Prepare file glance.syntax with following contents.

print "Global Priority Queue" , gbl_pri_queue

Run glance in adviser mode and check your global priority run queue.

glance -j2 -iterations 3 -adviser_only -syntax .glance.syntax

For a single cpu machine, double digit priroity run queue consistently is not good.

Check what processes are causing high cpu util.

UNIX95= ps -ef -o pcpu,pid,ppid,ruser,args|sort -nk1

Once you check what processes are causing it, you can do further troubleshooting.

[Ninad_1]

Basically the high run queue figures indicate there are a lot of runnable processes and there must be a lot of context switches happening. But I am surprised that the %sys is very less and its showing 99 %idle.
Please check which are the processes which are in run state continuously. As from the output it seems to the the processes related to vxfs and HP open view - opcctla, opceca, opcmsga etc.
Also the swagentd is in run state - is there any software installation local-remote in progress ?

Regards,
Ninad

[Vanja]

Hi,

Thanks for the replies.
I'm still not sure what this implicates since this is as I said bare bones OS on rx1620 or rx2620.

The global priority queue is 0 on all systems.

I'm attaching the glance output and ps output for a rx2620. This server doesn't even have the OV agent loaded.

Any insight is appreciated.

# glance -j2 -iterations 3 -adviser_only -syntax /tmp/glance.syntax

Welcome to GlancePlus

Global Priority Queue 0.0
Global Priority Queue 0.0
Global Priority Queue 0.0

#
# UNIX95= ps -ef -o pcpu,pid,ppid,ruser,args|sort -nk1
%CPU PID PPID RUSER COMMAND
0.00 8 0 root ioconfigd
0.00 11 0 root lvmkd
0.00 12 0 root lvmkd
0.00 13 0 root lvmkd
0.00 14 0 root lvmkd
0.00 15 0 root lvmkd
0.00 16 0 root lvmkd
0.00 17 0 root lvmschedd
0.00 19 0 root pagetable_init_daemon
0.00 21 0 root strmem
0.00 22 0 root strweld
0.00 23 0 root strfreebd
0.00 25 0 root kmemdaemon
0.00 26 0 root ttisr
0.00 48 0 root eventdaemon
0.00 52 0 root kend
0.00 83 0 root rng
0.00 576 1 root sh /usr/sbin/dgcfgdaemon
0.00 591 576 root vxnotify
0.00 615 573 root vxnotify -f -w 15
0.00 860 574 root vxnotify
0.00 976 1 root /usr/sbin/ptydaemon
0.00 1000 1 root /usr/lbin/nktl_daemon 0 0 0 0 0 1 -2
0.00 1010 1 root /usr/lbin/ntl_reader 0 1 1 1 1000 2 /var/adm/nettl /v
ar/adm/conslog.opts
0.00 1172 1 root /opt/ssh/sbin/sshd
0.00 1194 1 root /usr/sbin/rpcbind
0.00 1214 1 root /usr/sbin/biod 16
0.00 1215 1 root /usr/sbin/biod 16
0.00 1216 1 root /usr/sbin/biod 16
0.00 1217 1 root /usr/sbin/biod 16
0.00 1218 1 root /usr/sbin/biod 16
0.00 1219 1 root /usr/sbin/biod 16
0.00 1220 1 root /usr/sbin/biod 16
0.00 1221 1 root /usr/sbin/biod 16
0.00 1222 1 root /usr/sbin/biod 16
0.00 1223 1 root /usr/sbin/biod 16
0.00 1224 1 root /usr/sbin/biod 16
0.00 1225 1 root /usr/sbin/biod 16
0.00 1226 1 root /usr/sbin/biod 16
0.00 1227 1 root /usr/sbin/biod 16
0.00 1228 1 root /usr/sbin/biod 16
0.00 1229 1 root /usr/sbin/biod 16
0.00 1238 1 root /usr/sbin/rpc.statd
0.00 1244 1 root /usr/sbin/rpc.lockd
0.00 1578 1 root /usr/sbin/snmpdm
0.00 1593 1 root /usr/sbin/hp_unixagt
0.00 1633 1 root /usr/sbin/trapdestagt
0.00 1660 1 root /usr/sbin/fddi4subagt
0.00 1879 1 root /usr/sbin/cron
0.00 1902 1 root /usr/sbin/envd
0.00 1912 1 root /usr/sbin/hubd
0.00 1950 1 root /opt/perf/bin/ttd
0.00 2057 1 root /etc/opt/resmon/lbin/emsagent
0.00 2198 1 root /usr/sbin/rpc.mountd
0.00 2205 1 root /usr/sbin/nfsd 16
0.00 2206 1 root /usr/sbin/nfsd 16
0.00 2210 2206 root /usr/sbin/nfsd 16
0.00 2212 2206 root /usr/sbin/nfsd 16
0.00 2216 2206 root /usr/sbin/nfsd 16
0.00 2218 2206 root /usr/sbin/nfsd 16
0.00 2222 2206 root /usr/sbin/nfsd 16
0.00 2223 2206 root /usr/sbin/nfsd 16
0.00 2225 2206 root /usr/sbin/nfsd 16
0.00 2228 2206 root /usr/sbin/nfsd 16
0.00 2231 2206 root /usr/sbin/nfsd 16
0.00 2233 2206 root /usr/sbin/nfsd 16
0.00 2235 2206 root /usr/sbin/nfsd 16
0.00 2236 2206 root /usr/sbin/nfsd 16
0.00 2237 2206 root /usr/sbin/nfsd 16
0.00 2238 2206 root /usr/sbin/nfsd 16
0.00 2239 2206 root /usr/sbin/nfsd 16
0.00 2283 1 root /sbin/sh /usr/dt/bin/dtrc
0.00 2296 1 root /usr/sbin/getty console console
0.00 2298 1 root /sbin/sfd
0.00 11692 1270 root telnetd
0.00 11785 11693 root ksh -o vi
0.00 11789 11785 root ps -ef -o pcpu,pid,ppid,ruser,args
0.00 11790 11785 root sort -nk1
0.02 0 0 root swapper
0.02 2 0 root vhand
0.02 3 0 root statdaemon
0.02 4 0 root unhashdaemon
0.02 9 0 root nfsktcpd
0.02 10 0 root autofskd
0.02 20 0 root supsched
0.02 24 0 root progressdaemon
0.02 27 0 root ipmid
0.02 41 0 root dmperrd
0.02 42 0 root dmprestored
0.02 50 0 root pagezerod
0.02 117 0 root vxiod
0.02 277 1 root /usr/sbin/vxconfigd -k -m enable
0.02 550 1 root /usr/sbin/syncer
0.02 573 1 root /sbin/sh - /etc/vx/bin/vxrelocd root
0.02 574 1 root /sbin/sh - /etc/vx/bin/vxconfigbackupd
0.02 946 1 root ipmon -sD
0.02 972 1 root /usr/sbin/syslogd -D
0.02 1011 1010 root /usr/sbin/netfmt -C -F -f /var/adm/nettl.LOG000 -c /v
ar/adm/conslog.opts
0.02 1270 1 root /usr/sbin/inetd
0.02 1604 1 root /usr/sbin/ipv6agt
0.02 1614 1 root /usr/sbin/mib2agt
0.02 1863 1 root /usr/sbin/pwgrd
0.02 1898 1 root /usr/sbin/stm/uut/bin/sys/diagmond
0.02 1986 1 root /opt/perf/bin/midaemon
0.02 2017 1 root /opt/perf/bin/scopeux
0.02 2297 1 root /sbin/krsd -i
0.02 2302 1 root /etc/opt/resmon/lbin/p_client
0.02 2303 1 root /opt/wbem/sbin/cimserverd
0.02 2325 1 root /usr/lbin/kcmond
0.02 2350 2283 root /usr/dt/bin/dtlogin
0.02 2353 2307 root /opt/wbem/sbin/cimservera
0.02 2457 1898 root diaglogd
0.02 11693 11692 root -sh
0.02 11715 1 root /usr/bin/X11/hpterm -fn 6x13 -sb -ls -display 135.210
.98.172:0.0 -name hpterm (dst9dh01.lab.cif.att.com via REX
0.02 11716 11715 root -sh
0.04 1 0 root init
0.04 18 0 root ksyncer_daemon
0.04 51 0 root cmcd
0.05 1987 1 root /opt/perf/bin/perflbd
0.06 49 0 root schedcpu
0.06 1250 1 root /usr/sbin/automountd
0.06 1566 1 root sendmail: accepting connections
0.06 2307 1 root /opt/wbem/sbin/cimserver
0.08 554 1 root /usr/sbin/utmpd
0.08 2004 1 root /usr/sbin/swagentd -r
0.12 3028 3027 root /opt/perf/bin/alarmgen -svr 3027 -t alarmgen /var/opt
/perf/datafiles//
0.16 3026 1987 root /opt/perf/bin/rep_server -t SCOPE /var/opt/perf/dataf
iles/logglob
0.16 3027 1987 root /opt/perf/bin/agdbserver -t alarmgen /var/opt/perf/da
tafiles/
0.18 1676 1 root /opt/dce/sbin/rpcd
0.21 2179 1 root vxsvc -r /opt/VRTSob/config/Registry -e
0.29 1999 1 root /opt/prm/bin/prm3d
0.78 58 0 root vxfsd
#

[Chan 007]

Hi,

I guess you need to update your patches,

see below

Path Name: /hp-ux_patches/s700_800/11.X/PHKL_33694

Symptoms:
PHKL_33694:
( SR:8606401348 CR:JAGaf61297 )
New Itanium processors may have poor performance due to
differences in functions for processor resources.

Defect Description:
PHKL_33694:
( SR:8606401348 CR:JAGaf61297 )
Hardware enablement. This update is needed to enable
maximum performance on new Itanium processors.

Resolution:
Added call to new function in the wait loop of busy_wait to
yield the processor resource to other processes.


Link

http://www1.itrc.hp.com/service/patch/releasePage.do?BC=main|releaseIndexPage|&releaseId=0603-11.23

Chan

[Steven E. Protter]

Shalom,

I reccommend you patch your system up to the March 2006 QPK for 11i v2.

I further recommend you fully tune your system, especially kernel parameters.

SEP

[Vanja]

Hi,

Thanks for the suggestions.
I applied the single patch as well as the march 2006 11.23 patches in a sandbox environment that is exhibiting the same behavior, and this still doesn't solve the issue.

Steven - Do you recommend any specific kernel tune parameter to fix this issue - even though this is just a base 11.23 image with vxvm4.1 on rx1620?

Here's the current info on the system with runocc still at 100%.

# sar -q 5 5

HP-UX dst9dh01 B.11.23 U ia64 05/19/06

09:10:12 runq-sz %runocc swpq-sz %swpocc
09:10:17 1.2 100 0.0 0
09:10:22 1.4 100 0.0 0
09:10:27 1.4 100 0.0 0
09:10:32 1.4 100 0.0 0
09:10:37 1.4 100 0.0 0

Average 1.4 100 0.0 0
# swlist -l product |grep 33694
PHKL_33694 1.0 adding yield code to busy_wait
# swlist|grep 2006
FEATURE11i B.11.23.0603.040 Feature Enablement Patches for
HP-UX 11i v2, March 2006
HWEnable11i B.11.23.0603.040a Hardware Enablement Patches fo
r HP-UX 11i v2, March 2006
QPKAPPS B.11.23.0603.040a Applications Quality Pack Bund
le for HP-UX 11i v2, March 2006
QPKBASE B.11.23.0603.040a Base Quality Pack Bundle for H
P-UX 11i v2, March 2006
#

Any more suggestions?

Thanks,

[Chan 007]

Hi,

Try this one but not sure that exaclty running on your system

1. Disable NFS, if not used. Then check.

Else better log a software call with your local HP to check this system.

Chan

[Vanja]

Hi,

Tried that - didn't work either.

I have a case open since Thursday night but no resolution yet.

If anybody has any other ideas let me know.

Thanks!