HP-UX very very slow (having oracle database)

[Sandeep_Chaudhary]

Hp _UX machine running dam slow.


ystem: suxde034 Fri Aug 15 12:47:08 2008
Load averages: 2.63, 2.16, 1.44
241 processes: 200 sleeping, 41 running
Cpu states: (avg)
LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
2.63 0.5% 0.0% 86.4% 13.1% 0.0% 0.0% 0.0% 0.0%

Memory: 987432K (496780K) real, 2002672K (1146596K) virtual, 28652K free Page#
1/18

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 23726 root 128 20 12720K 720K sleep 0:04 11.03 9.92 HPUXChildWr
3 ? 23132 root 152 20 33576K 10876K run 0:30 9.87 9.85 caiUxOs
0 ? 23625 oracle 128 20 135M 1892K sleep 0:09 9.68 9.66 ora_j001_D1
0 ? 23602 oracle 128 20 135M 1868K sleep 0:19 9.62 9.61 oracleD14
1 ? 2164 root 158 20 9280K 468K sleep 5448:17 8.68 8.66 kcmond
1 ? 23727 root 128 20 7072K 296K run 0:03 9.25 8.33 inetd
0 ? 23721 ita 128 22 5060K 312K sleep 0:02 8.91 8.11 ksh
3 ? 23710 ita 128 22 8740K 392K sleep 0:03 8.66 7.87 sftp
2 ? 23632 root 128 20 9912K 652K sleep 0:06 7.69 7.67 bpcd
2 ? 1521 root 128 20 5328K 192K sleep 12:56 7.11 7.10 cron
2 ? 18917 oracle 148 20 1762M 3444K run 0:36 6.87 6.86 oracleD14
3 ? 23719 ita 154 22 5060K 324K sleep 0:03 6.98 6.35 ksh
1 ? 23637 root 206 20 9912K 680K run 0:02 6.42 6.0


uxde034 # sar -d 2 5

HP-UX suxde034 B.11.23 U ia64 08/15/08

12:06:35 device %busy avque r+w/s blks/s avwait avserv
12:06:37 c2t1d0 60.58 2.27 171 4191 6.93 16.85
c3t0d0 55.84 2.15 159 3502 6.89 15.63
c12t2d4 2.92 0.85 39 477 0.11 1.35
c12t2d5 1.82 2.78 79 1769 0.37 1.36
12:06:39 c2t1d0 1.61 0.50 2 71 0.00 11.71
12:07:36 c2t1d0 1.84 0.50 3 43 0.02 8.14
c3t0d0 1.16 0.50 2 36 0.00 6.97
c12t2d4 0.30 0.50 5 58 0.00 0.59
c12t2d5 0.07 0.50 2 18 0.00 0.44
12:07:37 c2t1d0 1.02 0.50 1 2 0.01 2.58
12:07:39 c2t1d0 0.50 0.50 1 16 0.00 4.77
c3t0d0 0.50 0.50 1 16 0.00 4.67
c12t2d4 0.50 0.50 8 24 0.00 0.41
c12t2d5 0.50 0.50 4 22 0.00 1.35

Average c2t1d0 4.30 1.73 11 220 4.84 14.22
Average c3t0d0 3.44 1.78 9 182 5.31 13.64
Average c12t2d4 0.41 0.59 6 72 0.03 0.78
Average c12t2d5 0.16 2.05 5 93 0.25 1.09
root as root@suxde034 [/var/opt/resmon/log]
suxde034 # sar -q 2 5

HP-UX suxde034 B.11.23 U ia64 08/15/08

12:16:17 runq-sz %runocc swpq-sz %swpocc
12:16:19 2.6 71 0.0 0
12:16:21 7.2 104 0.0 0
12:16:23 7.6 88 0.0 0
12:16:25 5.2 100 0.0 0
12:16:27 2.2 62 0.0 0

Average 5.4 85 0.0 0
root as root@suxde034 [/var/opt/resmon/log]
suxde034 # sar -w 2 6

HP-UX suxde034 B.11.23 U ia64 08/15/08

12:21:09 swpin/s bswin/s swpot/s bswot/s pswch/s
12:21:12 0.00 0.0 2.78 0.0 564
12:21:13 0.00 0.0 0.00 0.0 298
12:21:15 5.00 0.0 0.00 0.0 197
12:21:17 0.00 0.0 0.00 0.0 160
12:21:19 0.00 0.0 0.00 0.0 164
12:21:21 0.50 0.0 0.00 0.0 167

Average 0.92 0.0 0.50 0.0 262
root as root@suxde034 [/var/opt/resmon/


xde034 # vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 1 0 370575 15366 208 79 5 1 0 0 10 3471 45203 1023 23 7 71
root as root@suxde034 [/var/opt/resmon/log]
suxde034 # swap

[Sandeep_Chaudhary]

4 B.11.23 U ia64 1258886500 unlimited-user license
root as root@

[Steven E. Protter]

Shalom,

It would appear that you have a lot of oracle related processes.

They appear to be waiting for i/o.

Maybe there is a contention issue or misconfiguration problem with storage.

There could be contention between swap paging and the Oracle I/O. Ideally this should be on different disks.

You need to either increase memory or decrease memory use to solve this problem.

SEP

[Sandeep_Chaudhary]

changing dbc_max_pct to 10 will help?

[TTr]

I don't see anything striking in the posted metrics that would indicate a problem. The I/O architecture might be helpful. Also "HP-UX very slow" is too general of a problem description. There is very little activity shown in general and more specificly based on the "sar -w" output. May be when you captured these metrics it was not a busy time period.

Glance output during the problem periods would be more helpful.

Also look inside oracle itself for performance tuning and in the oracle jobs for bad sql.

[Hein van den Heuvel]

So you grabbed some performance data. This is good. (even better to attach as a .TXT file to improve readability). Did you actually look at the data for real yourself?

Seems to me your'kcmond' has gone wild.
Restart it, or restart the whole box, to keep it simple.

Obvious problems from the TOP output:

>> System time at 86.4%
>> TIME=5448:17 for kcmond.

That thing should be barely visible.
On a test system which was runnign for 100+ days it clocked TIME=3:22.

If that (+google) does nto set you on the rigth track, then please fedine 'darn slow'.
Best we try, we are not mind readers.
Are general commands slow (ls, cat)? quantify! Is the application slow? Oracle access slows... (top-5 delay from AWR or STATSPACK?)

hth,
Hein.

[Hein van den Heuvel]

And yes, as pointed out, 2 drives are overly busy. Are those per chance associated with the swap space? It seems that free memory is low. Is there an surprise/excessive usage, or is there simply not enough physical memory on the box? Check teh historical performance logs for memory usage trends!

12:06:35device %busy avque r+w/s blks/s avwait avserv
12:06:37 c2t1d0 60.58 2.27 171 4191 6.93 16.85
c3t0d0 55.84 2.15 159 3502 6.89 15.63

hth,
Hein.

[Sandeep_Chaudhary]

the reboot has done the magic. Thanks everyon ewho has contributed.