Multiple VS. Single RAID volume with Oracle?

[Greg OBarr]

I'm getting some new hardware in and we will be moving from an Ultra/SCSI attached RAID to a FC RAID. There are two rp5450 HP-UX 11.0 systems in a cluster, running Oracle 8i. The current setup was configured before I took this job, and is as follows:

2 disk arrays, configured RAID-5. Each is setup as one larage volume that appears to the system as a 175Gb disk. LVM is used so that each of the 2 large RAID-5 arrays is a PV. There is one LVM group with two physical volumes. Logical volumes are created on one PV and mirrored to the other PV using Mirror/UX. All Oracle data files, archive, redo, etc is in the same volume group and ultimately the same RAID volume. This has proven to be a reliable setup because more than once, one of the arrays has died and the system kept running on the mirror.

The new hardware will consist of 2 FC attached arrays. Both servers in the cluster will have 2 FC adapters. What I'm wondering is, what is the best way to set this up for maximum I/O while keeping the same or better reliability?

I'm considering:

A) Each array configured RAID 1+0 in one large volume. Mirror the PVs with LVM as with the current setup, but lose the overhead of RAID-5 on the controllers.

B) Each array configured with multiple (4-8)RAID 1+0 volumes consisting of 2 or 4 36Gb disks each. Then create multiple VGs with 2 PVs each, one on each array, using Mirror/UX. Then I can separate the Oracle data files and redo/archive logs, backup I/O (using BMC Backtrack), etc into a different VG and ultimately different disks.

Would there be significantly more overhead using Mirror/UX on multiple VGs as opposed to one? The array controllers will have 1Gb of cache (Current arrays have 128Mb each), so do I really need to do anything different?

Is there a better way to set it up that I haven't considered?

[Karthik S S]

What is the model of FC array that you will be using?. Instead of configuring AutoRAID/RAID5 on the FC Array why not directly mirror the disks?

-Karthik S S

[Jeff Schussele]

Hi Greg,

I'd definitely go with option B. You can then use different mount options (mincache=direct,convosync=direct) that will truly speed up data & index times if they're being buffered in the SGA (No need to buffer twice I say), but then still use LVM buffering for redo & logs.
And mirroring is at the LV level, not the VG level & no, there's no extra overhead because of more VGs or LVs. The overhead comes into play on the total amount of data being mirrored - not how it's mirrored.

And it's always a balancing act for cost vs. speed for the diff ways to protect data. RAID-5 is cheaper, but slower because of all the CRC calculations. RAID 1+0 is faster for reads & a little slower for writes than reads, but is more expensive.

And the cache is really going to help performance because as soon as it hits the array cache, the system will see the write as complete & signal the SW much faster than if the physical write would.

But split those LVs up. We have seperate LVs for all 4 - data, indices, logs & archives & we balance the I/O across multiple HBAs for all of those as well. Takes more time to plan & lay it all out, but it really pays off in perf & data integrity.

Rgds,
Jeff

[Geoff Wild]

If possible, on the Disk Array, I would stripe the disks as wide as possible and then mirror across controllers instead of LVM mirroring.

Yes, for simplicity, I would create separate vg's for data and redo logs.

Rgds...Geoff

[Greg OBarr]

There are already multiple LVs, but only one VG. You're right, each of the LVs is mirrored, not the PVs in the VG. See the output below from "bdf|grep vg4000". vg4000 is the single VG with all LVs on it (see vgdisplay output).

The arrays are not HP, they're Winchester. They each have 4 internal Ultra/SCSI controllers, 12 drives total, and are configured as one large RAID-5 volume with one hot spare. So, the disks are striped in hardware RAID and the load is spread out across all the disks. But the redo, archive, backup, and datafiles are all on the same volume.

The VG vg4000:
# vgdisplay vg4000
--- Volume groups ---
VG Name /dev/vg4000
VG Write Access read/write
VG Status available, exclusive
Max LV 255
Cur LV 22
Open LV 22
Max PV 16
Cur PV 2
Act PV 2
Max PE per PV 65535
VGDA 4
PE Size (Mbytes) 32
Total PE 10936
Alloc PE 9179
Free PE 1757
Total PVG 0
Total Spare PVs 0
Total Spare PVs in use 0

#bdf|grep vg4000
/dev/vg4000/lvol20 5242880 985912 4190536 19% /apps
/dev/vg4000/lvol19 5242880 3586532 1630476 69% /m21
/dev/vg4000/lvol18 7340032 6191724 1130372 85% /m20
/dev/vg4000/lvol17 5242880 4174316 1051876 80% /m19
/dev/vg4000/lvol16 5242880 3840492 1380484 74% /m18
/dev/vg4000/lvol15 5242880 4172284 1053876 80% /m17
/dev/vg4000/lvol14 5242880 3234676 1976888 62% /m15
/dev/vg4000/lvol13 5242880 3713524 1505468 71% /m14
/dev/vg4000/lvol12 5242880 3226100 1985276 62% /m13
/dev/vg4000/lvol11 5242880 4045300 1178876 77% /m11
/dev/vg4000/lvol10 7340032 5586564 1726076 76% /m10
/dev/vg4000/lvol9 5242880 3489272 1726216 67% /m09
/dev/vg4000/lvol8 7340032 6522448 804816 89% /m08
/dev/vg4000/lvol7 7340032 6737548 593076 92% /m07
/dev/vg4000/lvol6 7340032 5401216 1908528 74% /m06
/dev/vg4000/lvol5 7340032 5786244 1529516 79% /m05
/dev/vg4000/lvol4 7340032 6091332 1229196 83% /m04
/dev/vg4000/lvol3 5242880 643608 4527544 12% /m03
/dev/vg4000/lvol2 5242880 486980 4681832 9% /m02
/dev/vg4000/lvol1 5242880 2594548 2606960 50% /m01
/dev/vg4000/lvol22 5242880 2506632 2693504 48% /m22

[Jeff Schussele]

But with ONLY one VG, you could only apply ONE set of mount options. This is why you want diff VGs/LVs for the diff types of data to be stored. You want to bypass LVM buffering for data/indices IF Oracle will be doing it's own buffering via the SGA. But you DO want LVM buffering for the logs/archives. Plus you can further spread the load across HBAs by having diff VGs for data & indices, etc, etc, etc.
The point is the more you can spread & differentiate, the more you'll be able to cool the "hot-spots" that occur from how the tables & indices are layed out & accessed.

Rgds,
Jeff

[Steven E. Protter]

Mirror/UX is a very efficient, well tested piece of software.

As far as whether its better to use Mirror/UX for raid 1(it really doesn't due raid 0+1 by itself) versus doing it on the hardware, that would requier a test.

Oracle recommends Raid 1 or Raid 0+1 for OLTP databases. This also applies to Redo logs. Raid 5 hardware or software is not recommended for data or redo logs on oltp.

It would seem either your scenarios meet that critera and a performance test might be in order.

SEP

[Geoff Wild]

If that's the case, then I would reconfigure the RAID to have at least 2 LUNS - one small one for redo and the second for data...

Rgds...Geoff

[Greg OBarr]

With the new hardware, there will be one tray of sixteen 15K 36Gb disks per FC RAID.
Would I benefit from splitting each array up into, say, four volumes.

PV1a = RAID Vol1 = 6 36Gb Disks RAID 1+0 = 108Gb (for Oracle Data files)
PV2a = RAID Vol2 = 4 36Gb Disks RAID 1+0 = 72Gb (for Oracle Data files)
PV3a = RAID Vol3 = 4 36Gb Disks RAID 1+0 = 72Gb (for Archive/Redo/Backups)
And Two Global Hot Spares in the array

Set the other array up identically, labeling PV1b, PV2b, and PV3b.
Then, create 3 VGs with LVM.

vg1000, consists of PV1a, PV1b
vg2000, consists of PV2a, PV2b
vg3000, consists of PV3a, PV3b

Then create multiple mirrored LVs, with data files residing in the LVs in vg1000 and vg2000:

/dev/vg1000/lvol20 5242880 985912 4190536 19% /apps
/dev/vg1000/lvol19 5242880 3586532 1630476 69% /m21
/dev/vg1000/lvol18 7340032 6191724 1130372 85% /m20
/dev/vg1000/lvol17 5242880 4174316 1051876 80% /m19
/dev/vg1000/lvol16 5242880 3840492 1380484 74% /m18
/dev/vg1000/lvol15 5242880 4172284 1053876 80% /m17
/dev/vg1000/lvol14 5242880 3234676 1976888 62% /m15
/dev/vg1000/lvol13 5242880 3713524 1505468 71% /m14
/dev/vg1000/lvol12 5242880 3226100 1985276 62% /m13
/dev/vg1000/lvol11 5242880 4045300 1178876 77% /m11

/dev/vg2000/lvol10 7340032 5586564 1726076 76% /m10
/dev/vg2000/lvol9 5242880 3489272 1726216 67% /m09
/dev/vg2000/lvol8 7340032 6522448 804816 89% /m08
/dev/vg2000/lvol7 7340032 6737548 593076 92% /m07
/dev/vg2000/lvol6 7340032 5401216 1908528 74% /m06
/dev/vg2000/lvol5 7340032 5786244 1529516 79% /m05
/dev/vg2000/lvol4 7340032 6091332 1229196 83% /m04
/dev/vg2000/lvol3 5242880 643608 4527544 12% /m03
/dev/vg2000/lvol2 5242880 486980 4681832 9% /m02

And Archive/Redo/Backup data residing in the LVs in vg3000

/dev/vg3000/lvol1 5242880 2594548 2606960 50% /m01
/dev/vg3000/lvol22 5242880 2506632 2693504 48% /m22

[Geoff Wild]

That works too - but I would put all the Oracle data in 1 LUN - of 10 disks - that way you are striped across all 10 - as opposed to 6 and 4....

[Greg OBarr]

Cool, so just separate LUNS for Oracle datafiles and archive/redo/backup. That was the minimum I wanted to do. But apparently, it's better to stripe all the Oracle data across 1 LUN rather than 2 separate ones. At a minimum, I didn't want the disks competing for Oracle I/O and writing backups, redo, and archive logs out to the same disks. I should see a good performance increase by

A) going from 10K disks to 15K disks
B) going from 128Mb array controller cache
to 1Gb cache
C) going from RAID 5 to RAID 1+0

Considering that I have A and B no matter what, just by upgrading the hardware, I'm not sure the payoff would be worth the wasted disk space of going from RAID 5 to 1+0.

Anybody done any comparisons on the effect of RAID 5 vx. Raid 1+0 on Oracle OLTP using a disk array with very good I/O performance? (the vendor claims there is 10% or less performance impact using RAID 5 on their arrays with Oracle)

[Greg OBarr]

Geoff, others, please elaborate on convosync, etc in respect to LV/VG configuration.

We are mounting the filesystems now with only "-o rw, suid" options. I recall hearing that some benefit could be gained by using convosync in the mount options, but I wasn't aware that this had anything to do with the LV/VG configuration. I just thought you added "convosync" to the mount options. Obviously, I haven't tried this yet, but what information am I missing here?

[Todd McDaniel_1]

Greg,

This page will give you a good idea of mincache and convosync both of which are very good for attached Oracle filesystems.

http://uw713doc.sco.com/en/ODM_FSadmin/fssag-8.html


NOT using the unix system cache is advantageous b/c you dont tie up system resources handling I/O and logging etc...I use the following on my Oracle EMC mounted filesystems.


delaylog,largefiles,nodatainlog,mincache=direct,convosync=direct

REdirecting the convosync and mincache is an speed enhancing procdure.

The are advanced features so be sure you have enabled vxenablef -a to ensure advanced features of vxfs are enabled.

these can be applied to a filesystem by using this command.

mount -o remount,delaylog,largefiles,nodatainlog,mincache=direct,convosync=direct


This provides maximum speed... for your DB... I also use largefiles b/c my DBAs require it... not necessary for you though

[Leif Halvarsson_2 · Accepted solution]

Hi,

I will try to give some ideas as I have the same problems as you. We are going to move a number of databases to a EVA array.

"But apparently, it's better to stripe all the Oracle data across 1 LUN rather than 2 separate ones."

I suppose you mean one or two Disk groups or RAID sets, 1 or 2 LUNs makes no difference if the LUNs is in the same RAID set. The general recommendation (for performance)is to use as many disks as possible in each RAID set (or disk group). For an EVA array, just one disk group.


"Anybody done any comparisons on the effect of RAID 5 vx."

It may be differences between RAID controllers but, as I understand, very important is the database read/write ratio. If this ratio is high (for example 80/20) you will get very little performance improvement with RAID 1+0.