sema/s is over 2000 and pefromance issue with application

[skt_skt]

Total VM : 973.4mb Sys Mem : 885.9mb User Mem: 5.75gb Phys Mem: 20.0gb
Active VM: 455.0mb Buf Cache: 2.00gb Free Mem: 11.4gb

dbc_max_pct 10
dbc_min_pct 5
ninode ((NPROC+16+24*MAXUSERS)+32+(2*NPTY))
vx_ninode 262144

08:11:35 text-sz ov proc-sz ov inod-sz ov file-sz ov
08:11:38 N/A N/A 409/8212 0 0/33860 0 4059/32778 0
08:11:41 N/A N/A 409/8212 0 0/33860 0 4059/32778 0
08:11:44 N/A N/A 408/8212 0 0/33860 0 4043/32778 0

[Michael Steele_2]

Hi

Well you only have one GB of memory. And more than likely its 100% utilized. Let's see the following

ipcs -ma
vmstat 5 5
sar -m 5 5
sar -u 5 5
sar -v 5 5
swapinfo -tam
grep Phy /var/log/syslog/syslog.log

[skt_skt]

My earlier post cleary says 20Gb total and 11GB free. Are u mentioning the system memory usage??

Mar 10 07:04:01 hostname vmunix: Physical: 20971520 Kbytes, lockable: 18469592 Kbytes, available: 18800648 Kbytes

# ipcs -ma
IPC status from /dev/kmem as of Sat Nov 14 10:07:15 2009
T ID KEY MODE OWNER GROUP CREATOR CGROUP NATTCH SEGSZ CPID LPID ATIME DTIME CTIME
Shared Memory:
m 0 0x411c0e87 --rw-rw-rw- root root root root 0 348 2113 9278 3:07:09 3:07:09 7:05:06
m 1 0x4e0c0002 --rw-rw-rw- root root root root 1 31040 2113 9278 3:07:09 3:07:09 7:05:06
m 2 0x412000a3 --rw-rw-rw- root root root root 1 8192 2113 9278 3:07:09 3:07:09 7:05:06
m 14339 0x4d4e5251 --rw-r--r-- root sys root sys 2 330752 27095 27023 8:05:00 no-entry 16:09:33
m 11268 0x55315352 --rw-rw-rw- root sys root sys 1 4096 27095 1405 6:39:22 6:40:33 16:09:33
m 16389 0x44525354 --rw-r--r-- root sys root sys 3 632832 27095 22134 6:02:10 6:02:10 16:09:33
m 11270 0x53494152 --rw-r--r-- root sys root sys 1 1024 27095 27095 16:09:33 no-entry 16:09:33
m 11271 0x00005643 --rw-rw-rw- root sys root sys 1 1024 27111 27111 16:09:35 no-entry 16:09:35
m 11272 0x00005654 --rw-rw-rw- root sys root sys 1 1024 27111 27111 16:09:35 no-entry 16:09:35
m 27657 0x0c6629c9 --rw-r----- root sys root sys 2 23863660 26285 22532 8:08:25 8:08:47 12:14:56
m 1034 0x06347849 --rw-rw-rw- root sys root sys 1 77384 26285 26355 12:14:59 12:14:56 12:14:56
m 2059 0xffffffff --rw-r--rw- root root root root 0 22908 3682 3682 7:06:25 7:06:25 7:06:25
m 56332 0xcd8eaeac --rw-r----- oracle dba oracle dba 14 115318784 13952 19853 10:06:17 10:06:17 9:44:13
m 30733 0xb13ec75c --rw-r----- oracle dba oracle dba 13 144900096 2670 19905 10:06:24 10:06:24 5:15:42
m 27662 0x6cf89068 --rw-r----- oracle dba oracle dba 15 271319040 6028 19557 10:05:01 10:05:01 5:51:03
m 13327 0x59b075a8 --rw-r----- oracle dba oracle dba 13 120004608 5068 19961 10:06:31 10:06:32 7:07:47
m 5137 0x861326dc --rw-r----- oracle dba oracle dba 13 120004608 5812 20011 10:06:39 10:06:40 7:08:18
m 69650 0x00000000 --rw-r--r-- camprd camprd camprd camprd 1 2469888 14305 18674 10:01:00 10:01:18 1:37:38
m 1044 0x8be49cd8 --rw-r----- oracle dba oracle dba 12 128196608 6448 20059 10:06:46 10:06:46 7:08:55
m 9237 0x9fd1dc90 --rw-r----- oracle dba oracle dba 14 304054272 8876 20269 10:07:13 10:07:13 9:20:26
m 2070 0x8b8c4c00 --rw-r----- oracle dba oracle dba 15 123150336 6975 20111 10:06:53 10:06:53 7:09:23
m 1394711 0x412003ce --rw-rw-rw- root sys root sys 1 8785 26681 3847 19:57:48 no-entry 12:15:05
m 3096 0x08392408 --rw-r----- oracle dba oracle dba 12 824795136 4955 20162 10:06:59 10:06:59 8:22:31
m 6764569 0x4120044b --rw-rw-rw- root sys root sys 1 8785 26712 18883 10:03:14 no-entry 12:15:05
m 2074 0x732c04f0 --rw-r----- oracle dba oracle dba 18 439066624 14571 20217 10:07:07 10:07:07 8:05:36
m 4123 0x4120824c --rw-rw-rw- root sys root sys 1 8785 26725 18901 10:03:15 no-entry 12:15:06
m 4124 0x41200392 --rw-rw-rw- root sys root sys 1 8785 26737 18925 10:03:16 no-entry 12:15:06
m 4125 0x41207a6b --rw-rw-rw- root sys root sys 1 8785 26757 5581 20:02:50 no-entry 12:15:06
m 4126 0x4120045a --rw-rw-rw- root sys root sys 1 8785 26763 18942 10:03:16 no-entry 12:15:07
m 4127 0x4120807e --rw-rw-rw- root sys root sys 1 8785 26778 18959 10:03:16 no-entry 12:15:07
m 4128 0x41200396 --rw-rw-rw- root sys root sys 1 8785 26780 18975 10:03:17 no-entry 12:15:07
m 4129 0x41200400 --rw-rw-rw- root sys root sys 1 8785 26781 18990 10:03:17 no-entry 12:15:08
m 4130 0x41200356 --rw-rw-rw- root sys root sys 1 8785 26782 5658 20:02:51 no-entry 12:15:08
m 1059 0x4120768f --rw-rw-rw- root root root root 0 8785 7819 14371 19:43:09 no-entry 7:10:11
m 4132 0x41207300 --rw-rw-rw- root sys root sys 1 8785 26784 26784 12:15:08 no-entry 12:15:08
m 4133 0x41207371 --rw-rw-rw- root sys root sys 1 8785 26785 19009 10:03:17 no-entry 12:15:08
m 4134 0x412086d5 --rw-rw-rw- root sys root sys 1 8785 26788 19024 10:03:17 no-entry 12:15:09
m 4135 0x4120061a --rw-rw-rw- root sys root sys 1 8785 26790 26790 12:15:09 no-entry 12:15:09
m 4136 0x41200650 --rw-rw-rw- root sys root sys 1 8785 26791 26791 12:15:09 no-entry 12:15:09
m 4137 0x41205970 --rw-rw-rw- root sys root sys 1 8785 26792 26792 12:15:10 no-entry 12:15:10
m 4138 0x41200532 --rw-rw-rw- root sys root sys 1 8785 26797 26797 12:15:10 no-entry 12:15:10
m 4139 0x41207b29 --rw-rw-rw- root sys root sys 1 8785 26800 26800 12:15:11 no-entry 12:15:11
m 4140 0x412006c8 --rw-rw-rw- root sys root sys 1 8785 26801 26801 12:15:11 no-entry 12:15:11
m 4141 0x41200480 --rw-rw-rw- root sys root sys 1 8785 26802 26802 12:15:11 no-entry 12:15:11
m 4142 0x412004d4 --rw-rw-rw- root sys root sys 1 8785 26807 26807 12:15:12 no-entry 12:15:12
m 4143 0x41200693 --rw-rw-rw- root sys root sys 1 8785 26808 26808 12:15:12 no-entry 12:15:12
m 4144 0x41203245 --rw-rw-rw- root sys root sys 1 8785 26811 26811 12:15:12 no-entry 12:15:12
m 1073 0x41207875 --rw-rw-rw- root root root root 0 8785 7846 7846 7:10:15 no-entry 7:10:15
m 4146 0x41207512 --rw-rw-rw- root sys root sys 1 8785 26814 26814 12:15:13 no-entry 12:15:13
m 4147 0x4120793f --rw-rw-rw- root sys root sys 1 8785 26820 26820 12:15:13 no-entry 12:15:13
m 4148 0x412087a0 --rw-rw-rw- root sys root sys 1 8785 26821 26821 12:15:13 no-entry 12:15:13
m 4149 0x412059a5 --rw-rw-rw- root sys root sys 1 8785 26822 4041 19:57:53 no-entry 12:15:14
m 4150 0x412059d9 --rw-rw-rw- root sys root sys 1 8785 26829 19042 10:03:18 no-entry 12:15:14
m 4151 0x41208885 --rw-rw-rw- root sys root sys 1 8785 26830 19057 10:03:18 no-entry 12:15:14
m 4152 0x412047bd --rw-rw-rw- root sys root sys 1 8785 26834 19071 10:03:18 no-entry 12:15:14
m 849977 0x41208135 --rw-rw-rw- root sys root sys 1 8785 26845 5754 20:02:53 no-entry 12:15:14
m 4154 0x412059f0 --rw-rw-rw- root sys root sys 1 8785 26850 19088 10:03:19 no-entry 12:15:15
m 4155 0x412080ce --rw-rw-rw- root sys root sys 1 8785 26873 19103 10:03:19 no-entry 12:15:15
m 4156 0x412047be --rw-rw-rw- root sys root sys 1 8785 26885 19121 10:03:19 no-entry 12:15:15
m 4157 0x412059df --rw-rw-rw- root sys root sys 1 8785 26891 19135 10:03:20 no-entry 12:15:15
m 4158 0x412003b5 --rw-rw-rw- root sys root sys 1 8785 26893 5834 20:02:55 no-entry 12:15:15
m 4159 0x41207229 --rw-rw-rw- root sys root sys 1 8785 26905 26905 12:15:16 no-entry 12:15:16
m 1088 0x412080dc --rw-rw-rw- root root root root 0 8785 8150 14753 19:43:13 no-entry 7:10:22
m 4161 0x41207991 --rw-rw-rw- root sys root sys 1 8785 26908 19155 10:03:20 no-entry 12:15:16
m 4162 0x41208058 --rw-rw-rw- root sys root sys 1 8785 26913 19169 10:03:20 no-entry 12:15:16
m 4163 0x4120823b --rw-rw-rw- root sys root sys 1 8785 26966 19029 10:03:18 no-entry 12:15:30
m 4164 0x41208574 --rw-rw-rw- root sys root sys 1 8785 26967 26967 12:15:30 no-entry 12:15:30
m 4165 0x41208114 --rw-rw-rw- root sys root sys 1 8785 26971 19175 10:03:20 no-entry 12:15:31
m 4166 0x412078a2 --rw-rw-rw- root sys root sys 1 8785 26973 18913 10:03:15 no-entry 12:15:31
m 4167 0x41207f2b --rw-rw-rw- root sys root sys 1 8785 26981 26981 12:15:31 no-entry 12:15:31
m 73 0x00000000 D-rw------- appmgrmb dba appmgrmb dba 11 2658308 8465 8465 7:10:37 no-entry 7:10:37
m 369 0x00000000 --rw-r--r-- proteus proteus root proteus 2 4571136 18408 20511 17:42:01 17:45:31 7:23:45


:/home/kumarts# vmstat 2 2
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
0 0 0 107773 2978251 176 98 1 0 251 0 0 3880 12980 1414 8 3 89
0 0 0 107773 2977624 46 4 0 0 5 0 0 2525 4024 389 6 1 93
:/home/kumarts# sar -m 2 2

HP-UX B.11.00 A 9000/800 11/14/09

10:09:12 msg/s sema/s
10:09:14 1.00 32.34
10:09:16 1.00 40.00

Average 1.00 36.16
:/home/kumarts# sar -u 2 2

HP-UX hostname B.11.00 A 9000/800 11/14/09

10:09:25 %usr %sys %wio %idle
10:09:27 25 7 11 57
10:09:29 15 4 12 70

Average 20 5 11 63
:/home/kumarts# sar -v 2 2

HP-UX hostname B.11.00 A 9000/800 11/14/09

10:09:33 text-sz ov proc-sz ov inod-sz ov file-sz ov
10:09:35 N/A N/A 422/8212 0 0/33860 0 4154/32778 0
10:09:37 N/A N/A 416/8212 0 0/33860 0 4119/32778 0

:/home/kumarts# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 64 0 64 0% 0 - 1 /dev/vg00/lvol2
dev 9024 0 9024 0% 0 - 1 /dev/vg00/lvol9
dev 9024 0 9024 0% 0 - 1 /dev/vg01/lvol1
reserve - 4121 -4121
memory 16038 2523 13515 16%
total 34150 6644 27506 19% - 0 -

[Michael Steele_2]

Hi

a) 20 GB memory
b) You are using 2.4 GB of shared memory (Sum SEGSZ)
c) You have 11.3 GB of free memory (2978251 * 4096 * 1024**3 )
d) What is the oracle SGA size

su - orabwx
sqlplus /nolog
connect / as sysdba
show parameter sga / show sga

I'm having a hard time finding a problem.

Qustion: What is sar -d 5 5 (* disk bottleneck?)

[Michael Steele_2]

c) You have 11.3 GB of free memory (2978251 * 4096 / 1024**3 )

[Michael Steele_2]

e) swap is only 19%
f) no page outs

[skt_skt]

This application uses the database on the same system. But its a progressive database not oracle one.

Also the disk service time are below 10(sar -d) most of the time. I have the performnace data colection set for 15 minute interval

[Michael Steele_2]

Hi

Can you attached the sar -d output

[skt_skt]

Here is the sar o/p from the disks involved in those application file systems from the last problem window. Also i take the sar o/p only to get a trend not exact the service time becasue the o/p may be blended across metas.( I have a EMC Clariion)

-----------------------------
Wed Nov 11 08:20:47 MET 2009
Average c14t0d7 2.00 6.44 12 180 12.46 7.05
-----------------------------
Wed Nov 11 08:32:13 MET 2009
Average c14t0d7 24.06 3.52 283 13184 6.20 3.09
Average c12t2d6 2.90 0.50 8 2096 6.27 7.82
-----------------------------
Wed Nov 11 08:43:34 MET 2009
Average c14t0d7 1.95 30.51 31 487 19.92 3.62
Average c12t2d6 1.35 0.50 7 1791 5.47 4.16
Average c14t2d1 0.35 0.50 1 358 7.44 4.31
-----------------------------
Wed Nov 11 08:54:53 MET 2009
Average c14t0d7 7.19 2.11 44 685 6.92 4.75
Average c12t2d6 1.40 0.50 6 1893 4.90 4.54
Average c14t2d1 0.75 0.50 3 479 6.67 3.86
-----------------------------
Wed Nov 11 09:06:13 MET 2009
Average c14t0d7 11.99 5.65 35 537 10.27 7.92
Average c12t2d6 1.55 0.50 6 1740 6.05 4.24
Average c14t2d1 0.30 0.50 1 109 6.26 7.02
-----------------------------
Wed Nov 11 09:17:32 MET 2009
Average c14t0d7 17.35 8.58 97 1541 12.63 7.93
Average c14t2d1 0.05 0.50 0 0 1.85 13.54
-----------------------------
Wed Nov 11 09:28:54 MET 2009
Average c14t0d7 6.19 4.12 39 617 12.04 8.69
Average c14t2d1 1.55 0.50 4 320 4.59 5.21
-----------------------------
Wed Nov 11 09:40:13 MET 2009
Average c14t0d7 3.05 6.88 15 225 11.56 7.67
-----------------------------
Wed Nov 11 09:51:33 MET 2009
Average c14t0d7 2.70 6.93 19 298 7.66 5.84
-----------------------------
Wed Nov 11 10:02:52 MET 2009
Average c14t0d7 2.65 3.03 23 322 3.70 4.51
Average c14t2d1 0.10 0.50 0 1 3.77 7.94
-----------------------------
Wed Nov 11 10:14:12 MET 2009
Average c14t0d7 3.30 4.52 23 357 7.23 5.83
Average c14t2d1 0.85 0.50 4 316 4.75 2.89
-----------------------------
Wed Nov 11 10:25:31 MET 2009
Average c14t0d7 3.50 4.49 18 288 7.49 5.99
-----------------------------

[Michael Steele_2 · Accepted solution]

Hi

Please verify the last two columns are avwait and avserv. If so then you have a disk bottleneck on almost every disk and I've never seen that before.

Note: A disk bottleneck exists when avwait is greater than avserv