reading from a mirrored Disk

[Christian Schulze]

I have a strange phenomen, that happens on one of my clusters.

I have three clusters, each running oracle from mirrored disks (HPUX 11.00).

When I do something diskintensive in the DB (DB has to read from Disk, not from Buffer-Cache)
2 Systems are reading from both disks (primary and mirror), one system reads only from one disk, while the mirror is mostly idle.
It only happens, when oracle is doing some IO; when I do a cat oracledatafile > newfile then both disks are busy.

I can watch this very easily in Glance.

I think that I have set up all mirrors in the same way.

It is not really a problem, but I am wondering about this.

[Mark Greene_1]

Is your Oracle db setup in file systems (a.k.a., "cooked") or just LVMs ("raw")?

mark

[Stefan Farrelly]

Very interesting. Are you volume groups setup the same way for your oracle db on each cluster ? ie. if you do a vgdisplay -v on each are the disks in order - primary/mirror, or on the disks which read both primary mirror perhaps the disks are in a round-robin fashion, alternated between I/O channels (ie. data is on both channels, mirrored to the other channe).

[John Poff]

Hi,

The behavior you are describing sounds correct.

When your application writes to the disk, both the primary disk and the mirror disk need to be updated so they will both be active. When your application reads from disk, it will just use the primary disk. The mirror disk wouldn't get used on a read unless the primary disk failed.

If you are thinking about load balancing, you'll need to consider having multiple paths to the disks and then using some software that will help the OS take advantage of that [Autopath from HP, Powerpath from EMC, etc.].

JP

[James Murtagh]

Hi Christian,

I'm not sure its that strange to be honest. When reading data from mirrored logical volumes, assuming a parallel scheduling policy (discussed below) the data will be read from the physical device with the smallest IO queue. Hence it should always "pick" a device to read from. On writes it will need to maintain the mirror consistency hence will write to both, simultaneously if parallel scheduling is used.

Use lvdisplay to view the mirror consistency and scheduling settings - a parallel scheduling policy will, as mentioned, read from the device with the smallest IO queue. On the other hand, a sequentiual policy will always read from the primary copy (PV 0) and only if that is unsuccessful will it attempt to read from the mirror. On writes it will write to the primary first, then update the mirror after that. Hence it is almost always better to have a parallel scheduling policy.

In terms of what you are seeing on the other cluster (assuming you are indeed using parallel scheduling on both) is glance showing you that the IO are reads? If so are the read requests multiple files on one system but large files on the other?

Cheers,

James.

[Christian Schulze]

Thanks to all

well, the clusters are not exactly the same due to their age ...

I created all of the mentioned volumegroups in the same way (none of them are striped)

oracle uses a filesystem (vxfs)

disks were created this way:
vgcreate vg23 /dev/dsk/c4t3d0
vgextend vg23 /dev/dsk/c5t3d0
lvcreate -n lvol1 /dev/vg23
lvextend -L 200 /dev/vg23/lvol1 /dev/dsk/c4t3d0
lvextend -m 1 /dev/vg23/lvol1 /dev/dsk/c5t3d0
newfs -F vxfs /dev/vg23/rlvol1

so the mirror is on another Path (c4..) then the base (c1..)

I did some more testing
sure, my cat oracledatafile > testfile is silly, because it writes to the same disk

now i tried:
cat oracledatafile > /other_disk/file
and it reads from both disks (base and mirror)
the same if I write to /dev/null

could it be that it has something to do with block sizes of the filesystem and oracle ?
I did not specify on, always stuck with the default


It happens only when I analyze a Tablespace (containing an Index) with Oracle Enterprise manager


here is an output of sar -d 1 100
Average c4t2d0 1.40 0.50 1 24 4.65 10.01
Average c4t3d0 88.11 0.50 165 2775 4.88 5.60
Average c5t2d0 1.20 0.50 1 21 4.54 10.86
Average c5t3d0 10.19 0.50 13 206 5.31 7.95
Average c1t2d0 3.20 0.50 6 73 4.12 7.07
Average c2t2d0 1.00 0.50 1 21 2.67 9.55
Average c4t5d0 3.50 0.52 5 67 5.16 14.52
Average c5t5d0 3.60 0.52 5 64 5.24 15.60


Christian

[Christian Schulze]

Ah, I forgot the output of lvdisplay -v

--- Logical volumes ---
LV Name /dev/vgora/oradata1
VG Name /dev/vgora
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 6000
Current LE 1500
Allocated PE 3000
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation strict
IO Timeout (Seconds) default

--- Distribution of logical volume ---
PV Name LE on PV PE on PV
/dev/dsk/c4t3d0 1500 1500
/dev/dsk/c5t3d0 1500 1500

--- Logical extents ---
LE PV1 PE1 Status 1 PV2 PE2 Status 2
00000 /dev/dsk/c4t3d0 00000 current /dev/dsk/c5t3d0 00000 current
00001 /dev/dsk/c4t3d0 00001 current /dev/dsk/c5t3d0 00001 current
00002 /dev/dsk/c4t3d0 00002 current /dev/dsk/c5t3d0 00002 current
00003 /dev/dsk/c4t3d0 00003 current /dev/dsk/c5t3d0 00003 current
00004 /dev/dsk/c4t3d0 00004 current /dev/dsk/c5t3d0 00004 current
00005 /dev/dsk/c4t3d0 00005 current /dev/dsk/c5t3d0 00005 current
00006 /dev/dsk/c4t3d0 00006 current /dev/dsk/c5t3d0 00006 current
00007 /dev/dsk/c4t3d0 00007 current /dev/dsk/c5t3d0 00007 current
00008 /dev/dsk/c4t3d0 00008 current /dev/dsk/c5t3d0 00008 current
00009 /dev/dsk/c4t3d0 00009 current /dev/dsk/c5t3d0 00009 current
...

[Mark Greene_1]

>>It happens only when I analyze a Tablespace (containing an Index) with Oracle Enterprise manager


Because Oracle "knows" which disks the data sits on, could it be doing direct device addressing when going after an index the way it would if the db was setup on raw disk?


mark

[Jeff Schussele · Accepted solution]

Hi Christian,

James is spot-on correct. In a parallel scheuling policy (default for mirrored LVs), the system will choose the smallest queue to read from. This is one of the benefits of mirroring.

Rgds,
Jeff

[Christian Schulze]

Hi Mark,

I dont think Oracle knows which disks its data are on. Oracle only knows the Logical Volume. where the data is. (I dont use raw volumes)

Hi Jeff,
You and James are right, the System should always choose the smallest Queue to read from.
But I thought it would load the Queue in a sort of round robin order. It seems to do this, when I do the cat oraclefile > /dev/null; but not when oracle is doing some reading.



In my discussion with other colleagues another Question came up:
Is HPUX LVM doing a kind of write-caching for Oracle-files ?

I mean if i write a file with an application (for example vi), the application receives a complete file write, even if it is still in the buffer cache.
Is this also true for oracle ? esp. redologs ?
What would happen if the System crashes ?
Or does my journaling file-system this job ?

Christian