%wio 95-100 on Oracle Standby Server

[Brian Bientz]

For months our Oracle Standby Server (physical) was humming along applying logs and keeping current.

Then, on Mar 27 it started to have trouble keeping up. Since then it continues to have trouble. It got so bad we had to recreate the standby database when it got nearly 2 days behind after a big conversion of data.

The IT guys have upgraded the fiber to the SAN from 1GB to 2GB. They insist that the EVA is nearly idle and the traffic on the switch is nearly non-existant.

However, SAR reports 95-100% wio. Here's a sample of the SAR output:

11:30:27 %usr %sys %wio %idle
11:30:30 4 5 91 0
11:30:33 6 6 88 0
11:30:36 0 1 99 0
11:30:39 1 0 98 0
11:30:42 0 1 99 0
11:30:45 0 1 99 0
11:30:48 0 2 98 0

Can anyone help me understand this behavior?

[Steven E. Protter]

Shalom,

EVA guys make a change and your system suddenly messes up.

Sounds to me like a causal sitation.

You are using Oracle dataguard to get the logs?

Look for clues in the logs of whatever replication tool you are using.

Run some other reports to see if a network bottleneck was introduced.

http://www.hpux.ws/system.perf.sh

There could be SAN collision or problems with your driver needing an upgrade.

The latest bi-annual HP-UX patch set is always a good idea, as is looking for new drivers for your fiber card.

SEP

[Steve Lewis]

Hi Brian,
you didn't indicate which disks were waiting for io.
Try sar -d 5 12 and posting the results.
If the SAN is as idle as they say, then it could be a dodgy root disk or mirror that is causing these stats and not the database. Check vgdisplay -v vg00 and see if vgsync or lvsync is running in the case of LVM.
Then check the Oracle wait stats iternally.
Then use fcmsutil /dev/td0 (or whatever your FC device names are) with the devstat argument to check for fibre errors.
Then check for other things, such as recent changes/patches to the o/s and to Oracle itself.

[Steve Lewis]

Another idea: your standby server may be running low on memory and paging, which would also cause these symptoms.
Run vmstat -S 5 12 and check the 'po' column. If the number is > 5 then also check ipcs -ma to see how much memory has been taken by Oracle.

[Brian Bientz]

Here's the output from sar -d 5 12 ...

bbientz@backora2[stbypr01] /home/bbientz $ sar -d 5 12

HP-UX backora2 B.11.23 U ia64 04/17/07

14:37:08 device %busy avque r+w/s blks/s avwait avserv
14:37:13 c2t0d0 1.40 0.50 2 27 0.00 8.58
c2t1d0 1.00 0.50 2 22 0.00 7.13
c14t0d0 99.40 0.50 265 5349 0.00 5.83
c14t0d1 15.37 0.50 42 872 0.00 3.72
14:37:18 c2t0d0 2.00 0.50 4 62 0.00 8.17
c2t1d0 1.00 0.50 3 45 0.00 5.16
c14t0d0 99.60 0.50 257 4858 0.00 5.93
c14t0d1 7.00 0.50 22 499 0.00 3.20
14:37:23 c2t0d0 2.00 0.50 4 50 0.00 9.36
c2t1d0 1.60 0.50 3 46 0.00 9.86
c14t0d0 99.60 0.50 300 5638 0.00 5.74
c14t0d1 5.60 0.50 15 330 0.00 3.90
14:37:28 c2t0d0 2.20 0.50 4 50 0.00 8.97
c2t1d0 1.40 0.50 3 45 0.00 7.07
c14t0d0 99.00 0.50 321 7756 0.01 5.20
c14t0d1 17.20 0.50 37 774 0.00 4.92
14:37:33 c2t0d0 1.40 0.50 3 25 0.00 7.00
c2t1d0 0.80 0.50 2 20 0.00 5.61
c14t0d0 98.80 0.50 225 3916 0.00 5.74
c14t0d1 26.20 0.50 67 1379 0.00 3.93
14:37:38 c2t0d0 1.80 0.50 3 34 0.00 6.95
c2t1d0 1.00 0.50 2 29 0.00 5.92
c14t0d0 99.80 0.50 305 6067 0.00 5.70
c14t0d1 16.00 0.50 39 762 0.00 4.16
14:37:43 c2t0d0 0.60 0.50 1 14 0.00 6.54
c2t1d0 0.40 0.50 1 13 0.00 4.45
c14t0d0 96.40 0.50 287 5778 0.00 5.35
c14t0d1 41.80 0.50 87 1792 0.00 5.24
14:37:48 c2t0d0 1.80 0.50 4 121 0.00 6.19
c2t1d0 1.00 0.50 2 64 0.00 7.13
c14t0d0 99.40 0.50 351 7559 0.03 5.42
c14t0d1 14.40 0.50 36 813 0.00 4.17
14:37:53 c2t0d0 2.00 0.50 4 54 0.00 9.76
c2t1d0 1.60 0.50 3 51 0.00 8.50
c14t0d0 99.60 0.50 331 6339 0.00 5.51
c14t0d1 12.80 0.50 26 566 0.00 4.88
14:37:58 c2t0d0 2.00 0.50 3 36 0.00 9.12
c2t1d0 0.80 0.50 2 29 0.00 6.02
c14t0d0 93.20 0.50 271 5437 0.00 4.93
c14t0d1 12.80 0.50 31 637 0.00 4.40
14:38:03 c2t0d0 0.80 0.50 2 14 0.00 6.94
c2t1d0 0.80 0.50 1 10 0.00 6.70
c14t0d0 86.40 0.50 206 4148 0.00 4.20
c14t0d1 12.40 0.50 55 1168 0.00 2.28
14:38:08 c2t0d0 0.80 0.50 1 18 0.00 6.06
c2t1d0 0.40 0.50 1 16 0.00 5.61
c14t0d0 86.60 0.50 210 4240 0.00 4.13
c14t0d1 13.00 0.50 49 1050 0.00 2.67

Average c2t0d0 1.57 0.50 3 42 0.00 8.05
Average c2t1d0 0.98 0.50 2 32 0.00 6.97
Average c14t0d0 96.48 0.50 277 5590 0.00 5.35
Average c14t0d1 16.21 0.50 42 887 0.00 3.99

[Brian Bientz]

Sorry for the formatting (or lack of) on the previous post. I ran vmstat -S 5 12 and value for po was 0.