CPU Utilization High

[Daniel Cardani]

I am running OVO 7.x on HP 11.11 and getting tons of CPU utilization (100.00%) exceeds configuration threshold (95.00%). When I run TOP the two process that I see are samd and swci. These process did not run this high until I installed OVO. I do have loaded the PHKL_27091 patch leaded.
System: lilian Fri Mar 7 11:08:36 2003
Load averages: 2.35, 2.38, 2.59
202 processes: 165 sleeping, 37 running
Cpu states:
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
2.35 72.3% 0.0% 27.7% 0.0% 0.0% 0.0% 0.0% 0.0%
Memory: 293928K (225384K) real, 628060K (428604K) virtual, 12092K free Page# 1/14
TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
? 1846 root 152 20 16784K 1244K run 476:41 43.65 43.57 samd
? 1062 root 152 20 13268K 1216K run 469:49 42.69 42.62 swci
? 1809 root 152 20 27792K 16008K run 30:02 5.54 5.53 opctrapi
? 1519 bin 152 20 56144K 9288K run 3:42 1.08 1.07 ovrequestd
? 1695 root 152 20 32380K 15128K run 2:05 0.80 0.80 ovalarmsrv
? 0 root 127 20 32K 0K sleep 7:10 0.63 0.63 swapper
? 1699 root 152 20 30088K 5596K run 0:39 0.53 0.53 ovoareqsdr
? 1722 root 152 20 23036K 1704K run 1:40 0.51 0.51 opcmsgm
? 1516 root 154 20 19528K 4884K sleep 3:18 0.46 0.46 pmd
? 31 root 152 20 5056K 5056K run 0:53 0.40 0.40 vxfsd
pts/tb 5498 root 154 24 23332K 8776K sleep 0:11 0.32 0.31 swremove
? 1725 root 152 20 28988K 4400K run 0:21 0.31 0.31 opcdispm
? 1961 root 154 24 20572K 5296K sleep 3:17 0.27 0.27 ovw
#

[Steve Steel · Accepted solution]

Hi


Make sure you have the latest DCE on your machine.

check what you have with swlist


Steve Steel

[Michael Steele_2]

Evaluate your server:

a) sar -u 5 5
(* If %idle = 0 then CPU bottleneck. If true and %wio > 15 then disk/tape bottleneck *)

b) sar -d 5 5
(* if %busy > 50% NOTE: Most then possible disk bottleneck. NOTE: Some disks %busy > 20%. *)
(* For any disk where %busy > 50% check if avwait > avserv. If true then I/O bottleneck, else, memory bottleneck. *)
(* If avserv > 20 ms then possible disk bottleneck *)

c) vmstat 5 5
(* if po > 0 then paging is occurring *)

d) sar -v 5 5
(* if numerator is near or equal to denominator for any then file, inode or proc tables are full *)

e) swapinfo -tam
(* if total > 85% then add 2ndary swap *)

Also verify runaway processes. Best to reboot for this.

[Daniel Cardani]

Here is what I have for DCE. I am not sure of the dates on it. Where do I do to find the latest DCE file sets.

# DCE-Core B.11.11 HP DCE/9000 Core Client Software
DCE-Core.DCE-COR-64SLIB B.11.11 DCE-COR-64SLIB
DCE-Core.DCE-CORE-DTS B.11.11 DCE-CORE-DTS
DCE-Core.DCE-CORE-HELP B.11.11 DCE-CORE-HELP
DCE-Core.DCE-CORE-NOTES B.11.11 DCE-CORE-NOTES
DCE-Core.DCE-CORE-RUN B.11.11 DCE-CORE-RUN
DCE-Core.DCE-CORE-SHLIB B.11.11 DCE-CORE-SHLIB
DCE-Core.DCE-JPN-E-MSG B.11.11 DCE-JPN-E-MSG
DCE-Core.DCE-JPN-S-MSG B.11.11 DCE-JPN-S-MSG
DCE-Core.DCEC-ENG-A-MAN B.11.11 DCEC-ENG-A-MAN
DCE-Core.MACR-ENG-A-MAN B.11.11 MACR-ENG-A-MAN
# DCE-CoreTools B.11.00.10 DCE-CoreTools
DCE-CoreTools.DCE-BPRG B.11.00.10 DCE-BPRG
# DCE-KT-Tools B.11.00.10 DCE-KT-Tools
DCE-KT-Tools.DCE-KT-BPRG B.11.00.10 DCE-KT-BPRG
DCE-KT-Tools.DCE-KT-NOTES B.11.00.10 DCE-KT-NOTES
DCE-KT-Tools.DCE-KT-SHLIB B.11.00.10 DCE-KT-SHLIB

[Robert-Jan Goossens]

Hi,

here are the latest patches for DCE,

http://www4.itrc.hp.com/service/patch/patchDetail.do?patchid=PHSS_27258&context=hpux:800:11:11

http://www4.itrc.hp.com/service/patch/patchDetail.do?patchid=PHSS_27259&context=hpux:800:11:11

hope it helps,

Robert-Jan.

[Daniel Cardani]

Here are the results of the server evaluation


# sar -u 5 5

HP-UX lilian B.11.11 U 9000/802 03/10/03

11:50:38 %usr %sys %wio %idle
11:50:43 81 19 0 0
11:50:48 82 18 0 0
11:50:53 84 16 0 0
11:50:58 82 18 0 0
11:51:03 80 20 0 0

Average 82 18 0 0
#

# sar -d 5 5

HP-UX lilian B.11.11 U 9000/802 03/10/03

11:53:56 device %busy avque r+w/s blks/s avwait avserv
11:54:01 c0t5d0 1.80 0.50 1 20 3.73 12.06
c0t6d0 1.40 0.50 2 6 0.85 15.79
11:54:06 c0t5d0 4.80 0.50 5 67 3.18 12.49
c0t6d0 2.60 0.50 3 14 2.04 13.41
11:54:11 c0t5d0 4.20 0.50 5 70 3.17 15.40
c0t6d0 4.60 0.50 6 51 1.40 27.69
11:54:16 c0t5d0 2.00 0.50 2 22 2.09 13.44
c0t6d0 1.00 0.50 1 4 1.25 13.39
11:54:21 c0t5d0 8.80 0.50 11 174 4.64 8.41
c0t6d0 3.00 0.50 3 22 3.70 11.86

Average c0t5d0 4.32 0.50 5 70 3.78 11.28
Average c0t6d0 2.52 0.50 3 19 1.87 19.55
#

# vmstat 5 5
procs memory page
faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
4 0 0 92593 5586 28 18 1 0 0 0 0 15 32668 173 81 19 0
4 0 0 92593 5523 7 12 1 0 0 0 0 35 34634 124 82 18 0
3 0 0 84242 5523 2 3 0 0 0 0 0 19 35144 116 85 15 0
3 0 0 84242 5523 6 9 0 0 0 0 0 59 34569 129 82 18 0
3 0 0 85054 5523 2 2 0 0 0 0 0 29 35162 114 83 17 0
#

# sar -v 5 5

HP-UX lilian B.11.11 U 9000/802 03/10/03

11:55:46 text-sz ov proc-sz ov inod-sz ov file-sz ov
11:55:51 N/A N/A 194/820 0 867/1020 0 1339/15440 0
11:55:56 N/A N/A 199/820 0 872/1020 0 1348/15440 0
11:56:01 N/A N/A 199/820 0 872/1020 0 1348/15440 0
11:56:06 N/A N/A 194/820 0 868/1020 0 1341/15440 0
11:56:11 N/A N/A 194/820 0 869/1020 0 1343/15440 0

#

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 182 842 18% 0 - 1 /dev/vg00/lvol2
reserve - 330 -330
memory 355 161 194 45%
total 1379 673 706 49% - 0 -
#

[Steven E. Protter]

I am attaching a handy dandy sar performance data collection script for you.

It uses sar and can be configured to collect data for any time period. It runs in background and is a modification on something HP support sent me.

As always, no warranty, but its not dangerous.

SEP

[Daniel Cardani]

Looks like the DCE patching fixed the issue. Thanks all for all your help