1748232 Members
3865 Online
108759 Solutions
New Discussion юеВ

Re: Oracle11i

 
Faizer
Advisor

Oracle11i

I have a cluster using rp7410 servers on a SAN connecetd to a va7410.
I have a single node installation of Oracle11i. I am encountering the following problems.
1. During the day there are complains that user processess get timed out and are logged out.
2. The top command shows most of the processess are in sleep mode and a detail of the sleep mode shows that they are in STRMS state.Please refer the attachement.
Any update on this.
thedetails are as follows:
-----------------------------------------
Subject: FW: Errors on upgrade/performance




Most of the time I find the process in sleep mode.
Details of this is are as follows.

B3692A GlancePlus C.03.70.00 12:34:51 rprod1 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SN N | 11% 11% 11%
Disk Util F F | 7% 6% 7%
Mem Util S SU UB B | 98% 98% 98%
Swap Util U UR R | 44% 44% 44%
--------------------------------------------------------------------------------
Resources PID: 10795, f60webmx PPID: 7367 euid: 222 User: applmgr
--------------------------------------------------------------------------------
CPU Usage (util): 0.4 Log Reads : 28 Wait Reason : STRMS
User/Nice/RT CPU: 0.4 Log Writes: 0 Total RSS/VSS : 9.3mb/ 12.6mb
System CPU : 0.0 Phy Reads : 0 Traps / Vfaults: 8/ 0
Interrupt CPU : 0.0 Phy Writes: 0 Faults Mem/Disk: 0/ 0
Cont Switch CPU : 0.0 FS Reads : 0 Deactivations : 0
Scheduler : HPUX FS Writes : 0 Forks & Vforks : 0
Priority : 154 VM Reads : 0 Signals Recd : 0
Nice Value : 22 VM Writes : 0 Mesg Sent/Recd : 33/ 30
Dispatches : 38 Sys Reads : 0 Other Log Rd/Wt: 30/ 33
Forced CSwitch : 1 Sys Writes: 0 Other Phy Rd/Wt: 0/ 0
VoluntaryCSwitch: 30 Raw Reads : 0 Proc Start Time
Running CPU : 0 Raw Writes: 0 Mon Dec 1 12:34:25 2003
CPU Switches : 4 Bytes Xfer: 0kb :
C - cum/interval toggle % - pct/absolute toggle Page 1 of 1
ProcRsrc WaitStat MemRegns OpenFile NextKeys ProcSysc
===================================================================================================
The TOP provides the following:

System: rprod1 Mon Dec 1 12:36:20 2003
Load averages: 0.22, 0.31, 0.38
607 processes: 567 sleeping, 38 running, 2 zombies
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 0.27 0.0% 5.1% 3.4% 91.5% 0.0% 0.0% 0.0% 0.0%
1 0.22 0.2% 7.3% 2.8% 89.7% 0.0% 0.0% 0.0% 0.0%
2 0.14 1.2% 7.5% 2.4% 88.9% 0.0% 0.0% 0.0% 0.0%
3 0.27 1.0% 17.0% 7.1% 74.9% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.22 0.6% 9.3% 4.0% 86.2% 0.0% 0.0% 0.0% 0.0%

Memory: 4287408K (1683752K) real, 8779276K (5031376K) virtual, 193124K free Pag
e# 1/68

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
1 ? 8527 applmgr 154 22 75844K 22708K sleep 0:24 6.06 6.04 f60webmx
3 ? 11023 oracle8 154 22 812M 20624K sleep 0:00 7.38 5.65 oracleSAV11
1 ? 2160 oesdba 152 20 1767M 260M run 98:51 5.63 5.62 expres63
0 ? 10587 oracle8 154 20 812M 20912K sleep 0:46 4.31 4.30 oracleSAV11
2 ? 3382 applmgr 154 22 78596K 23580K sleep 3:28 3.29 3.28 f60webmx
1 ? 7502 oracle8 154 22 803M 10844K sleep 0:04 3.05 3.04 oracleSAV11
0 ? 7410 applmgr 154 22 70636K 16732K sleep 0:08 3.01 3.00 f60webmx
3 ? 10795 applmgr 154 22 70912K 18496K sleep 0:03 2.81 2.80 f60webmx


These are some screen shots of glance. These also could help for any performance issues.

We are facing the following scenarios
1. Some processes are automatically timed out.
2. The same report[ with all parameters the same] at different times takes different times. eg: 5 secs and 20 minutes

======================================================================================
sar -du 2 10

Average c0t6d0 22.39 20.85 61 2437 65.74 19.34
Average c3t6d0 17.09 16.08 53 2375 46.95 17.66
Average c5t0d1 82.91 0.69 606 22999 5.23 3.96
Average c5t0d2 5.60 0.55 24 639 4.55 3.49
Average 48 8 20 24

============================================================================================
CPU report
B3692A GlancePlus C.03.70.00 10:09:17 rprod1 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SN N | 14% 34% 89%
Disk Util FV V | 12% 12% 73%
Mem Util S SU UB B | 99% 99% 100%
Swap Util U UR R | 43% 42% 43%
--------------------------------------------------------------------------------
CPU REPORT Users= 6
State Current Average High Time Cum Time
--------------------------------------------------------------------------------
User 0.4 1.8 12.2 0.02 3.29
Nice 10.8 27.9 81.3 0.57 50.46
Negative Nice 0.0 0.1 1.3 0.00 0.18
RealTime 0.0 0.1 0.4 0.00 0.26
System 1.5 2.6 7.8 0.08 4.74
Interrupt 0.2 0.3 0.8 0.01 0.62
ContextSwitch 0.0 0.2 0.2 0.00 0.28
Traps 0.0 0.0 0.0 0.00 0.02
Vfaults 0.2 0.4 1.7 0.01 0.80
Idle 86.3 66.5 93.0 4.54 120.37

Top CPU user: PID 28005, f60webmx 9.9% cpu util
Active CPUs: 4 Page 1 of 2
ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit

==============================================================================================

memory report
--------------------------------------------------------------------------------------------
B3692A GlancePlus C.03.70.00 10:07:35 rprod1 9000/800 Current Avg Hig
-------------------------------------------------------------------------------
CPU Util S SN N | 27% 51% 89
Disk Util FFV V | 12% 12% 48
Mem Util S SU UB B | 99% 99% 100
Swap Util U UR R | 42% 42% 43
-------------------------------------------------------------------------------
MEMORY REPORT Users= 6
Event Current Cumulative Current Rate Cum Rate High Rate
-------------------------------------------------------------------------------
Page Faults 2965 34200 390.1 432.3 2336.5
Page In 1451 9860 190.9 124.6 588.4
Page Out 0 1478 0.0 18.6 129.6
KB Paged In 6.3mb 10.2mb 848.4 131.9 848.4
KB Paged Out 0kb 18.2mb 0.0 235.1 2590.7
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 289 711 38.0 8.9 38.0
VM Writes 8 3112 38.0 39.3 186.0

Total VM : 7.46gb Sys Mem : 2.70gb User Mem: 3.65gb Phys Mem: 9.97gb
Active VM: 4.61gb Buf Cache: 3.49gb Free Mem: 132.0mb Page 1 of 1


=========================================================================================================
B3692A GlancePlus C.03.70.00 10:06:27 rprod1 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util S SN N | 76% 75% 89%
Disk Util FFV V | 48% 18% 48%
Mem Util S SU UB B |100% 100% 100%
Swap Util U UR R | 43% 43% 43%
--------------------------------------------------------------------------------
PROCESS LIST Users= 6
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 400% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
oracleSAV11 7323 1 236 oracle8 96.6/91.8 1290.5 0.0/ 0.0 23.7mb 1
oracleSAV11 10087 1 237 oracle8 96.2/92.8 16.2 0.0/ 2.5 7.0mb 1
f60webmx 17924 7367 154 applmgr 33.3/ 0.6 30.4 0.0/ 0.0 15.8mb 1
f60webmx 3382 7367 154 applmgr 25.9/ 3.0 79.6 0.0/ 0.1 23.1mb 1
oracleSAV11 3758 1 154 oracle8 9.3/ 1.3 34.4 0.0/ 0.3 11.3mb 1
f60webmx 641 7367 154 applmgr 8.4/ 0.1 2.0 7.6/ 0.0 11.8mb 1
f60webmx 9846 7367 154 applmgr 4.6/ 0.4 0.6 0.0/ 0.0 6.4mb 1
oracleSAV11 18116 1 154 oracle8 3.4/ 0.4 22.3 0.0/ 0.2 33.9mb 1
oracleSAV11 1262 1 154 oracle8 3.0/ 0.0 0.7 0.0/ 0.0 4.9mb 1
oracleSAV11 10134 10133 154 oracle8 2.9/ 2.9 0.2 0.7/ 0.7 5.1mb 1
vhand 2 0 128 root 2.9/ 0.0 1309.2 270/ 0.1 32kb 1
Page 1 of 6
2 REPLIES 2
James A. Donovan
Honored Contributor

Re: Oracle11i

George Petrides_1
Honored Contributor

Re: Oracle11i

It's obvious from the metrics above that you are running out of memory. Glance shows 99-100% memory utilization and the last list of processes by glance shows 'vhand' process taking up CPU time. When you are running out of memory and start using swap as memory, then your memory access goes from nanoseconds to miliseconds and timeouts can very well be the result of this slowdown. You need to either free up a bit of memory or the best solution would be to add more memory. I wouldn't run at 99% memory utilization, just give your system some room to breath. I've seen a customer that was running 97-99% memory and was OK until one day someone ran a program that used the free remaining memory and the machine became extremely slow.
George