LVM Layout for Oracle under SAN EVA 5000

[Simon Stanford]

Hi,

Has anyone got any experience of using Oracle 9i and 10g when the databases will be physically stored on an EVA 5000 SAN?

We are used to having FC10 fibre arrays where we have separate oracle file systems based around disk spindles etc.

On the SAN we will have a single Oracle LUN presented to us. Should we slice this up using LVM to separate filesystems or have it as a single filesystem and just use directories to give it structure.

Does anyone have any recommendations?

Thanks.

[James R. Ferguson]

Hi Simon:

My comments are general ones. I have no experience with Oracle 10i.

I would opt for presenting multiple logical volumes (with filesystems therein) as opposed to a single filesystem. This gives you the ability to utilize different VxFS (JFS) mount options for performance should you choose.

Regards!

...JRF....

[Piergiacomo Perini]

Hi Simon,
in my opinion using LVM to separate filesystems
is better.

regards

[Luk Vandenbussche]

Hi Simon

If possible ask for multiple LUNs

It is better to spread the data over multiples LUNS.

If possible ask also for VRAID-1 LUNS

[A. Clay Stephenson]

Ideally, give the SAN back its single LUN and ask them to carve this single LUN into as many LUN's as your have SCSI path's to the SAN. For example, if your single LUN is 1000MB and you have 2 SCSI paths to the SAN then ask for 2 500MB LUN's. You then create your volume group isung these 2 LUN's. The Primary path to LUN0 should be through SCSI Path A (Alternate B) and the Primary Path to LUN1 should be SCSI Path B (Alternate A).

The idea is to spread the I/O over as many SCSI paths as you have. For maximum flexibility, you might choose to create 4 LVOL's, striping each in 64K to 128K chunks. I would use one LVOL for datafiles, another for indices, another for redologs, and another for archives. You can choose the OnlineJFS mount options that suit each best although I have found that fully-cooked i/o in Oracle now yields better performance on my data.

A simpler configuration would divide into 2 LUN's; datafiles-indices; redo-archives.

You should also request that the RAID level be 1/0 as opposed to 5 for better performance at the cost of storage efficiency.

[S.Arunkumar]

Hi Simon,
For the Better I/O utilization ask the SAN Administrator to create Multiple vdisks for your Oracle FS.

Since in EVA there is option of creating a single LUN to one host.Mostly Storage admins prefer this for ease of management.

Once you get multiple vdisks created as per your requirement use wither LVM or VxVM to create a VG and cretae the FS as you required.

S.Arunkumar

[Sameer_Nirmal]

Hi,

Yes, we are using Oracle 9i with EVA 5000 in SAN environment.

We have configured 2 DGs in EVA with one LUN each. All Vdisks use VRAID1 . In DG1 with one LUN we have separate file system for Data Files , Indexes , Redo logs(1). In DG2 we have comparativly small LUN having LVOLs for archive and (redo log(2) + control file. The ensures the redundancy. If you have EVA with sufficient disks , you can have such layout. Consider future storage requirements as well for sizing of DGs.

One LUN is specifically active on one HSV controller and another on other . This way you are utilizing the both the controller's power and better i/o performance. In future requirement as well , you should try to equally distribute LUNs across the two HSV controllers of EVA.
Be-aware EVA is quite sensitive to I/O patterns put into it. In order to have optimized layout of DB on EVA , Oracle I/O and application I/O ( if any ) patterns should be considered.

Secondaly the HP-UX default scsi_queue_depth 8 is quite low for EVA. You need to increase it to 16 or 32 to startwith.

[TwoProc · Accepted solution]

I'm a fan of the "lotsa-lun" theory rather than the "big-lun" theory.

I also like to make sure that I don't have disks holding data or indexes that also hold rollback, temp, redo, or archive logs. I like index areas separate than data areas. I like the system and user tablespaces separated too. I also like printout areas, and trace areas separate from the data, index, system, rollback (undo), redo, and temp areas. You'll get arguments on many or all of these topics. Ignore them. I'm right on this. :-) (just kidding, they all bring years of experience and their own very valid methodologies, this one is just mine - which many of them won't agree with).

Which of these above I'm MOST anxious about to separate? Or, a simple list:
A) Rollback,
B) Temp,
C) redo (better if you can have two different disk sets to interleave),
D) archive logs(better if you can have two different disk sets to interleave),
E) data/index(better if u can separate, and I like to have quite a few different large disk sets to move things around with - pairing large demand tablespaces with small demand ones on same disk sets),
F) Oracle_Home, and application code, printout/trace/debug where u like it...

Also, I don't care to use one big LUN for each of these - again, an arguable point. I prefer lots of luns for each of the first A/B/D/E (and of course smaller sets for C, just b/c it's not that big).

Also, when I get my bag-o luns, for a disk set, I prefer to use extent based (another arguable point) stripes to get my data sprinkled over all of the disks that make up a mount point. Most others prefer to use regular striping here, you should research and consider both, and, even better, do some benchmarks after you get the storage system on both for various scenarios and see which one delivers the best I/O for you.

Also, like others, I prefer R0/1 - not R5.

[Simon Stanford]

Hi

Thanks for all your input it's good to see a number of options available.

I also found this very interesting document on the Oracle web site

http://www.oracle.com/technology/deploy/performance/pdf/EVA_ORACLE_paper.pdf

It compares the performance of a number of deployments on a very similar set up to hardware we have. It compares such things as one vs two disk groups and Raid 0+1 or Raid 5.

The conclusions are interesting. Apart from certain special circumstances there isn't a lot in it which ever way you slice and dice the EVA. The trick appears to be to ensure you use the HP Secure Path and Preferred Path software:-

"For most applications using Oracle Database 10g, the HP StorageWorks EVA configured with a single disk
group, no zoning and using the Preferred Path feature of HP StorageWorks Secure Path software is the ideal
solution to ensure maximum performance and flexibility."

[Andrei Lica_1]

My current production env is based on EVA5000 and Oracle9i.
So my advices are :
-read and apply EVA best practices document from HP ( I don't remember the link but if you need the doc I can attach it later ). Use fewer large disk groups instead of many small ones !
-use the right block size for Oracle. For EVA the optimum seems to be 64kb
-define as many LUNs as you need with the right RAID level ( RAID1 for busy oracle files )
-spread the LUN's on both controllers but remember the controllers have cache mirroring so if you load one with many writes the second one is also loaded. So LVM striping doesn't help ( my personal opinion, I've never tried ) here
The second controller is mainy used for failover not for load balancing.

[Alex Georgiev]

Simon, just a few quick things that I don't think anyone has mentioned yet:

. old strategies about striping and spreading loads over different LUNs generally do not work on the EVA because of the virtualization that it does

. when creating disk groups, keep in mind that the more disks you have in a single group, the more IO requests per second that group can service

. thus, more disk in a single group = higher performance

. for better protection they recommend that your data and your logs reside in different disk groups... if one disk group goes to hell, you might still have the other with good data on it

. try to spread your disk group(s) vertically - across all the disk shelves

. do read the best practices docs; you can find them by going to www.hp.com/go/eva5000 and clicking on the "Technical docs" link on the right side

. if you'll be doing large data loads, and feel like taking risks, you can disable the write cache mirroring feature - it will boost your IOs per second... remember to turn it back on after the data load is done

. when defining LUNs, set the "preferred path" through the "Command View" software but keep in mind what others have said about the caching and load balancing

. finally, it is VERY IMPORTANT to make sure that you have the latest EVA controller firmware (aka VCS code); you want at least 3.028

Hope that helps!