(stripe) mirror vs oracle

[A. Ardonne]

Hi,

I'm about to expand en re-arrange my disks and volumes on my hp-ux 11.0 machine. Right now I have my oracle 8i database spread over 4 disks containing 16 lvol's. Those disks are mirrored 1 on 1 in 2 fc10 diskcabinets.

Because there are several tables within my database there are more used then others I experience an unequally spreaded load on my disks.
I want to rearrange and expand those disks/volumes.

My oracle/application provides advises me to create 1 large lvol so we won't have the problem of full lvol's anymore and the also advise me to stripe the volumes so the load will be balanced.

The questions which comes to my mind are:

Will I get significant improved performance when I decide to make 1 large lvol which is striped across the disks?

Does anyone have experience with oracle 8.1 and hpux 11.0 and large volumes and offcourse large files(-parameter)?

Which conciderations must I keep in mind when making the descission the create a striped mirror? ie: it isnot as easily to expand a striped mirror with another disks then a "normal" mirror.

I know the info I supply isn't very detailed but I have to start somewhere.

Thanks in advance

Arjan

[Mark Grant]

One thing I personally would advise is to use physical volume groups and a PVG-strict allocation policy. Have your "primary" disks in one physical volume group and the mirrors in another physical volume group. This makes it much easier to expand and reduce things and also makes the mirroring itself easier.

The above scenario makes sure that mirrors and primaries remain seperated without having to add loads of physical volume paths to the LVM commands.

[Jean-Louis Phelix]

Hi,

>Will I get significant improved performance >when I decide to make 1 large lvol which is >striped across the disks?

You can't mirror a stripped volume. It works if you use PE distribution only. This could really improve performance.

I don't like having only one large lvol. It's not so easy to manage and I can't see any performance gain.

Regards.

[James Specht]

Here is what we did on flat disk. We had ten disks on two channels. I create one volume group containing 5 of the disks from one channel. I add one striped logical volume to encompass the entire 5 disks. Then I added the other 5 disks into the volume group and mirrored disk for disk on the logical volume. This gave us the best performance on flat disks. We did turn on large files for the logical volume. We don't have a overly large database, only 50 gigs, but we have only 16 dbfs and some are close to 5 gigs now. We have not seen a performance problem because of size.

--Jim

[Geoff Wild]

I just did this for our OVO server:



First, setup PV Groups for volume group (vg01)

You can't just mirror striped lv's, you have to use EXTENT- BASED MIRRORED STRIPES:


Distributed Allocation Policy
This example shows how the -D y option can be used to create EXTENT-BASED MIRRORED STRIPES.

Assume that volume group /dev/vg01 has two physical volume groups:
PVG0, PVG1.

Assume that each physical volume group has 4 physical volumes.

The distributed allocation proceeds as follows:
+ A free extent is allocated from the 1st pvol in PVG0.
+ A free extent is allocated from the 2nd pvol in PVG0.
+ A free extent is allocated from the 3rd pvol in PVG0.
+ A free extent is allocated from the 4th pvol in PVG0.
+ A free extent is allocated from the 1st pvol in PVG0.
+ A free extent is allocated from the 2nd pvol in PVG0.
+ A free extent is allocated from the 3rd pvol in PVG0.
+ A free extent is allocated from the 4th pvol in PVG0.

The following creates a logical volume in vg01 with EXTENT-BASED MIRRORED STRIPES:

vgextend /dev/vg01 /dev/dsk/c11t2d0 /dev/dsk/c11t3d0 /dev/dsk/c11t4d0

vi /etc/lvmpvg

VG /dev/vg01
PVG PVG0
/dev/dsk/c7t2d0
/dev/dsk/c7t3d0
/dev/dsk/c7t4d0
/dev/dsk/c7t5d0
PVG PVG1
/dev/dsk/c11t2d0
/dev/dsk/c11t3d0
/dev/dsk/c11t4d0
/dev/dsk/c11t5d0

lvcreate -D y -s g -m 1 -L 512 /dev/vg01
Logical volume "/dev/vg01/lvol24" has been successfully created with
character device "/dev/vg01/rlvol24".
Logical volume "/dev/vg01/lvol24" has been successfully extended.
Volume Group configuration for /dev/vg01 has been saved in /etc/lvmconf/vg01.conf

lvdisplay /dev/vg01/lvol24
--- Logical volumes ---
LV Name /dev/vg01/lvol24
VG Name /dev/vg01
LV Permission read/write
LV Status available/syncd
Mirror copies 1
Consistency Recovery MWC
Schedule parallel
LV Size (Mbytes) 512
Current LE 64
Allocated PE 128
Stripes 0
Stripe Size (Kbytes) 0
Bad block on
Allocation PVG-strict/distributed
IO Timeout (Seconds) default

[Steven E. Protter]

I have the experience you are asking for and endorse the first response to your thread.

If you want significant performance improvement then follow Oracle's guidelines for OLTP Online Transaction Processing systems.

That would be not to stripe but to lay the data down on Raid 1 or RAid 10 disk. This will give you a significant bounce in performance. Also, but the redo logs on raid 1 or raid 10. That will also help performance.

I do not believe you will be satified with performance in a striped environment.

With HP-UX lvm you cant stripe then mirror, creating raid 10. You can do this on an external disk array, letting the hardware handle it.

SEP

[A. Clay Stephenson]

If this were a smart array rather than simply a bunch of disks I would answer differently. I still wouldn't go with 1 lvol but 2 would be fine in that case. Large filesystems are not a performance problem.

In your case, you really have two choices: 1) stripe -- you will see an improvement 2) mirror -- much safer. Given those two choices I would opt for safety.

You do have one other option that will APPEAR to be the best of both worlds -- extent-based striping which allows you to stripe and mirror. You can certainly do that but the performance gains will be minimal. The reason for that is that the stripe chunks (1MB is the smallest possible) are much too large to spread the load. In the more traditional striping scheme the chunks are typically 64KB or so and the I/O gets spread very well and equally important in a timely manner.

In your case, I would continue along the path you first suggested and try to manually spread the I/O unless you have the option of getting a smarter array.

[A. Ardonne]

Hi,

Thanks for all of youre reactions. If I understand this well I almost have to conclude that it is no garantee that slicing will increase my performance but it is sure that my system maintainance will be slightly more difficult en less flexible compared with "normal" mirrored lvol's?

I wrote that I have an unequally spreaded load as a result of some large tables which are heavely used. On an oracle-site a found some information which tells me that slicing will only improve my performance if I've reached the limits of my disk-capacity (throughput) in my current situation. How can I determine if this is so?

[Bernhard Mueller]

Arjan,

regarding your last question, you would want to use sar -d 1 5 (to get the idea)
sar -d 1 3600 (to monitor during a busy hour)

You may then consider which disk (pair) is most busy, what's on it and if you can put something on a different disk (pair) which is not busy.

You can also use iostat to see the bps throughput. e.g.
iostat 1 10.

Regards,
Bernhard

[Alex Ostapenko]

Dear Arjan --

A storage performance guru once told me "The best I/O is no I/O." If you have a bad SQL, fix it. If you're doing a full tablespace scan, add an index. If you're reading from disk, try to read from memory (buffers or cache). If you're reading/writing everything from one disk, try to spread it out to many. And so on. My comments to your dilemma:

1. Check to see if you're doing full tablespace scans. If so, see if you can add an index so that you're not doing these scans.

2. Make sure that you have followed Oracle guidelines in separating out indexes from their tablespaces, redo logs from data objects, etc.

3. Check to see what the I/O's are to specific objects. Sometimes, you may have two heavily accessed objects on the same device, and if you put them on separate devices, it evens the load out. However, one object may be swamping a device all by itself.

4. See that you have adequate Oracle buffers and HP-UX dynamic buffer cache. Use "sar -b" to see that. This assures that you're getting as much from memory as possible as opposed to going to disk.

5. Ascertain the characteristics of your I/O. Maybe you don't have a bottleneck. Using "sar", you can see "%busy" (disk utilization) and "r+w/s" (I/O's per second). Service time = 10*(%busy/(r+w/s)) -- I don't trust sar's avwait and avserv. If %busy is consistently greater than 40%, you are starting to exercise the disk. If consistently over 60%, then you have potential bottleneck. If r+w/s also know as IOPS (I/O's per second) is reaching disk capability, then you have potential bottleneck. For example, disks with 8msec access time are capable of 125 IOPS, 6msec access are capable of 167 IOPS. If service time is rising above the stated disk time, then you might be having a problem. blks/s divided by r+w/s gives you the blks-per-IO. If this is large, then you're probably doing sequential reads; if it's small then it's probably random reads or writes. Remember, default JFS filesystem blocksize is 1K, although Oracle recommends 8K.

6. There are two types of striping you can implement with LVM:
(a) "lvcreate -i -I" striping -- this implements the standard conventional striping that we're all familiar with allowing stripe sizes of 4k to 32M; this method allows smaller stripes; it is not expandable, you must rebuild the whole stripe set if you add disks; it introduces some LVM overhead in performance; you need only 1 command to implement it; I don't know if you can mirror this or not
(b) extent-based striping as described by one of the responses above -- to implement, you must manually (or by script) allocate each extent across the available devices; stripe size is limited to extent size (normally 4MB); if you add disks in the future, it's easy to extend the lvol's without rebuilding the whole thing, it won't be ideal but it will work; you must document how you allocated it so you know how to rebuild it when you have problems; you can mirror with this method; mirroring has the added bonus that LVM will try to load-balance the reads; also, although the stripe size appears large (4MB), remember that LVM and JFS attempt to coalesce I/O's so although the filesystem blocksize may be only 1K or 8K, the actual number of blocks read or written in one I/O may be much larger (see item #5 above on how to compute this)

7. There are a number of mount options available in JFS, and some additional ones in Online JFS that can help improve performance from the HP-UX side. They are too numerous to describe here, but see the "man mount_vxfs" for more info.

8. If disk is a bottleneck, levelizing the load by any of the methods described above (starting with #1) will undoubtedly improve performance. If you choose the methods in 6a or 6b, we've experienced considerable improvement with both methods. When our disks had bottlenecks, these methods spread the load out over all the disks, and as load increased, it increased evenly across all the disks. The improvement is illustrated dramatically if you have Measureware (from GlancePlusPak) and can plot "before" and "after" data using Perfview or MS-Excel. But it added system administration book-keeping to our tasks.

Good luck and happy load balancing!

=:-) Alex