Disk - block device for Oracle RAC in RHEL5 - any tutorial?

[mar_q]

Hello. I need some advice.

Assuming  I have already partitions created on particular LUN from EMC array with usage of fdisk.

Now what I have found is that for oracle RAC 10 and higher disks as block devices are recommended rather than raw devices.

What are the next steps to prepare block devices for Oracle RAC? I've done some research many but have no clue.

Is there clear tutorial about it?

[Matti_Kurkela · Accepted solution]

There are so many possibilities, a single tutorial might be too complex if it tried to cover all the possibilities.

There are many configurations that can be made to work, but not all of them are what I would consider optimal.

So you have an EMC array. How are you connecting to it: FibreChannel, iSCSI, other?

Do you have multiple connections to the array? That would be a good idea if you want high availability/fault tolerance/ability to change SAN components while the system is running. The fact that you are implementing a RAC suggests that high availability is a requirement, so multiple connections would be expected.

If you have multiple connections, you will need a multipath solution. Which one? RHEL5 built-in dm-multipath, or commercial EMC PowerPath? Or something else?

Are you going to use Oracle ASM for disk management, or traditional raw disk devices? In a system that has dm-multipath or EMC PowerPath multipathing, ASM must be restricted to use multipathed devices only. ASM wants to see one and only one device per actual storage LUN: the Oracle administrator can control what disk devices ASM looks at by using the ASM_DISKSTRING configuration setting.

Whatever choices you make, the last step will be assigning custom permissions for the appropriate block devices, so that the ASM or the database engine can access them while not running as root.

If you use whole-disk devices, the dm-multipath subsystem of RHEL5 updated to the current patch level (= RHEL 5.8) allows the configuration of multipath device owner/group/permissions in /etc/multipath.conf.

If you use LVM or partitioning, you will have to configure an udev rule for your data LVs/partitions, because the /dev filesystem is RAM-based and will be regenerated from scratch at every boot.

Why on earth are you using fdisk-style partitions on data LUNs of a modern enterprise system???

The use of fdisk-based partitions will restrict you to a maximum LUN size of about 2 TB, as the partition table format has a hard design limit. If you have multiple partitions on a LUN, you cannot easily resize them: only the partition that is located physically last on the LUN can be resized easily. This will bite you when (note: not "if") your system outgrows your estimated capacity requirements. Remember Parkinson's law and its corollaries:

http://en.wikipedia.org/wiki/Parkinson%27s_law

Data expands to fill the space available for storage.

A RAC set-up is a significant investment for maximizing availability and minimizing downtime: if the system is intended to be used 24/7/365, getting downtime for capacity expansion and other upgrade or maintenance operations can be hard. So you should be choosing a configuration that will allow you to do as much as possible while the system is running. LVM or straight ASM on a multipathed device wins here.

A RAC system may also be long-lived: if it works "well enough", you might not get downtime for major upgrades or reorganizations any time soon. So make your configuration future-proof: eliminate arbitrary limits whenever possible, so you won't have to implement ugly workarounds for them later.

Today, it is not unusual for a new gaming desktop PC to have more than 1 TB of storage space on a single physical disk, and 3 TB disks are available off the shelf right now. Using fdisk-style partitioning with its hard 2 TB design limit on enterprise data LUNs today is just silly in my opinion.

(On a system disk, you must do whatever is required by the BIOS/firmware: this usually means fdisk partitioning for a traditional BIOS, or GPT partitioning for systems with UEFI firmware.)

The most straightforward configuration would be to give the whole-LUN multipath devices directly to ASM.

If you need to slice a single LUN to multiple devices at the OS level for some reasons, I would consider using LVM rather than partitioning.

If you really need to use partitioning instead of LVM, I would suggest using GPT partitioning: use "parted" instead of fdisk to create your partitions, and choose to create a new partition table of type GPT. The GPT partition table format does not have significant size limitations, and always includes a place-holder fdisk-style partition table, so even old Windows systems will not "helpfully" offer to auto-format a GPT-formatted disk if they see one for some reason.