Oracle on SAN disk array.

[Leif Halvarsson_2]

Hi,
We are going to migrate a number of Oracle databases to a SAN disk array. The current configuration is traditional JBOD where the database is spread across a number of physical disks.

Any good ideas about LUNs and filesystem layout ?

My own assumptions:
- One LUN for OS.
- One LUN for Oracle binarys.
- One LUN for database tables.
- One LUN for transaction logs.

Where to locate rollback segments, control files etc. .

The databases is not very write intensive so I plan to use Vraid5.

Servers is rp2470 OS HP-UX 11i, disk array EVA 3000, Oracle versions 8.1.17 and 9iAS.

[Jean-Luc Oudart · Accepted solution]

Hi Leif,

In Our environment, the OS sits on the internal disks. teh SAN is XP128 and we use RAID1 and lvm stripping on top of it.
I would recommend you take a baseline of your current performance with the usual suspects (Glance, sar, ...) and statspack for Oracle. Therefore you will be in a position to compare when we you live with your new settings.
If you can benchmark beforehand this is even better.

Rgds,
Jean-Luc

[Duncan Edmonstone]

Lief,

The doc Jean Luc has pointed you at is a good one - SAME (Stripe And Mirror Everything) is Oracles reccomendation for all databases now - a more generic document which doesn't just focus on the XP disk array can be found here:

http://otn.oracle.com/deploy/performance/pdf/opt_storage_conf.pdf

You may need an account with Oracle Technet to read this, but thats not hard to get...

HTH

Duncan

[Mark Greene_1]

We have three systems each with an Oracle database, and all attached to an EMC CX600. We setup on LUN for /oracle where the oracle software is put, including the control and config files; one LUN for /oraclelogs where the redologs and archives are stored; and then we setup LUNS for /oradata1 through as many as /oradata6 for the databases to be stored. All the LUNS are RAID5 except the /oraclelogs which is RAID 1 for performance reasons. The RAID 1 LUN is on a a seperate mirrored pair of drives from the other LUNS. The busiest databases are stored on different drives from where /oracle is stored to avoid potential IO bottlenecks.

We have dual fibre connections from each system to the SAN, and each system has multiple redundant paths to each LUN for maximum throughput and redundancy.

HTH
mark

[Alzhy]

Oracle Storage on SAN Arrays is dependent on what kind of array one has. Most array manufacturers would have a "best practices" document that would suggest whether to do "striping" of the LUN's that are already RAIDed or not. Cache-Centric Arrays -- ie. large HDS (XP) and EMC tend to suggest Striping the already protected LUN's - preferably accross FC PATHS or HBA's. Controller Centric Arrays - like the EVA, the suggestion is to leave the LUN's as is..

For the EVA Array - HPUX Combo and normal/DSS/light-OLTP Oracle Storage, have your LUN's from the EVA as VRAID5's (RAID5) -- and optionally allocate VRAID1's (mirrors) for your REDO LOGS. Of course, if your I/O bandwidth need could not be satisfied by VRAID5's -- then have them presented as VRAID1's (stripe-mirrors). In our experience though - for most Oracle applications -- VRAID5's would suffice and the economics of gaining more capacity mostly defeats the cost and the slight performance of VRAID1's.

If you can, having more FC-HBA's to your EVA SAN is the most effective way in increasing I/O throughput... If you can dedicate the LUN's for Oracle REDO's on it's own HBA/FC-Channel the better. I've a server that's got 8 FC-HBA's on 1 system, LUN's are grouped and presented according to the tablespaces they will hold and on separate FC Channels... Ensure your hot tablespaces are contained on it's own LUN or LUN's.

[Steven E. Protter]

Your layout is pretty nice.

We use it for our low intensity oracle apps which are on the same version as yours at this time.

I've wanted to experient with keeping the Index and Data files on different luns to prevent contention. We run our oracle data/index run pretty hard on our oltp raid 10 LUN.

Just make sure, in case you do have performance issues you can go raid 10 with data/index and redo logs, thats what oracle recomeneds. I skipped that on redo logs once and it made a big difference.

SEP

[Stefan Farrelly]

We run all our ORacle databases on SAN LUNs.

We dont allows the OS to be on a LUN for reliabilty and performance reasons (too many things that can take down a SAN and we dont want our OS to go down) - we keep the OS on local disks.

We now use micro SAN's - on your disk subsystem (wether it be XP/EMC/EVA) keep some disks aside so they are used exclusively for a LUN which you want for high performance - this is so that no other luns get to use those spindles. Probably doesnt matter so much if you dont have an intensive database.
As most LUNs today use all available spindles it doesnt matter to have different luns for oracle binaries/db/trans logs - just lump them all together unless you use a high performance micro san for some of them.

[Alzhy]

I have been contending with DBA's that seem have gotten stuck in the 20th Century -- unaware of developments in storage and servers that the mere mention of RAID5 sends chills up their spines. My standard decorum these days is to cut an SLA with our DBA's and tell them not to mind what disks are we preenting to them -- and only come to us if they have proof that the storage landscape we have provided them -- is not meeting SLA's.. This works out very well and have resulted in very good relationships -- they now do not care whether these LUN's are RAID5,10 or whatever... as long as they meet performance exxpecetations..

What I seem to be wondering in all the "recipes" - SAME, Quest, Metalink, etc.. is the often mentioned recipe -- have your REDO logs on dedicated LUNs -- to me, this is rather inpractical specially for Cache-centric arrays... What I found rather really performance boosting is to have these HOT luns be on their own FC-Channels.. What good is a superfast LUN is it is sharing FC bandwidth with the rest of the normal LUNs?

[Ralph Haefner]

The last project at my last job (they laid me off an hour after I turned the final server over to the DBA to move his database, nice) was to move about 10 database servers from a 3 year old EMC symmetrix onto a new StorageTek d280.

We found something very interesting. We set up Raid 10 luns on the array hardware, which performed pretty well, much better than the old storage. But when we took 2 of those and striped across both luns using Veritas Volume Manager, we got a HUGE performance boost, which we didn't expect. The gain was in the ballpark of 80-90%. We thought the hardware raid alone gave you almost all the performance you could get, but adding Veritas was a big help. If you can, test something like that before deciding on your final configuration.

[Alzhy]

That is right Ralph.. StorageTek arrays fall into those arrays whose LUN's could be striped...

But for very large configurations - STRIPING LUNS would always yield the best performance regardless of what Array are you using.. This is true in the following scenario: You have an Array to each HBA pair and you have several of those. I've benchmarked a huge server before that's got 8 HDS Arrays (XP512's) -- each to a pair of 2GBPS HBA links (16 All). Using VxVM DMP, my paths are protected and I have Active/Active balancing.. My FS benhmarks easily reach the maximum theorethical bandwidth of the FC's on volumes that are striped accross the 8 HDS arrays and 16 FC HBA's... It was an environment for a very large DB with large record sizes..