Old debate: Raw Device Vs. File System

[Filipe_1]

Hi Fellow forumers,

This debate, I know, is sort of old fashioned but...

We are provisioning an Oracle OFA anvironment for one of our customers.

We work here as (I think) the vast majority Oracle environments does: using filesystems for a number of reasons.

But the client stands that the use of raw device should achieve better performance anyway.

The problem is showing an "official" doc/paper/whatever wich supports filesystems.

I Know there is some good information on this issue by Oracle itself, and we already got something, but we need further information from HP/Veritas.

Does anyone knows any document like that? Wich arguments would one use to support filesystems instead of raw devices?

Here: Oracle 9.2, HP-UX 11.i v1. Online JFS.

Thanks in advance,

Filipe.

[Mark Greene_1]

You don't need a white paper, just explain the spec's of the hardware. What type of disks, what's their transfer rate, and what's the transfer rate on the controllers? Are you going to use mirroring or RAID 5? How many CPUs and how much memory?


mark

[Geoff Wild · Accepted solution]

Yes - a lot of threads on this...

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=29674

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=21382

Good doc on disk planning:

http://docs.hp.com/cgi-bin/fsearch/framedisplay?top=/hpux/onlinedocs/B2353-90698/B2355-90698_top.html&con=/hpux/onlinedocs/B2355-90698/00/00/32-con.html&toc=/hpux/onlinedocs/B2355-90698/00/00/32-toc.html&searchterms=Device%7cVs.%7cFile%7cSystem%7cRaw&queryid=20040315-065509

In the old days (8 years ago), the following was true:

3. What are the benefits of raw devices?

There can be a performance benefit from using raw devices, since a write
to a raw device bypasses the UNIX buffer cache; the data is transferred
direct from the Oracle buffer cache to the disk. This is not guaranteed,
though; if there is no I/O bottleneck, raw devices will not help. The
performance benefit if there is a bottleneck can vary between a few
percent to something like 40%. Note that the overall amount of I/O is
not reduced; it is just done more efficiently.

Another, lesser, benefit of raw devices is that no filesystem overhead
is incurred, in terms of inode allocation and maintenance, or free block
allocation and maintenance.

But no longer:

You could also mount file systems that you don't want buffer cache by:
-o mincache=direct
That will bypass the buffer cache all together - good option for Oracle on LVM.

Using raw device partitions introduces a level of complexity in configuration planning, administration, and the movement of databases.

Rgds...Geoff

[Jean-Luc Oudart]

Filipe,

"But the client stands that the use of raw device should achieve better performance anyway. "

If you have OnlineJFS i understand you can achieve the same with the extra manageability of fs over raw device

see this thread :
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=193104

Would you have the chance to benchmark it ?
I'd like to see the results and differences (if any) ?

Then, in regards to "better performance" I think database design and application tuning may give you more than tuning the IO subsystem.
My opinion.

Regards,
Jean-Luc

[The Real MD]

I'm going to answer this with the another question i'm afraid. Does the extra 10-15% extra performance benefit of using raw devices make it worth while for the lack of device management. Raw partitions are unmanageable without third party products, so in the event you need more disk space, suddenly its not quite as simple as extending a logical volume. Its also probably worth a look on the veritas website for official documentation.

Regards

Martin.

[Alzhy]

For VERY VERY LARGE Datawarehouse or DSS systems, I will actually recommend going RAW with HP-UX11i + Oracle9/10i. The perceived diferences of 10%-15% DOES matter if you run a very large system -- how large is LARGE? -- I'd say those above 10TB and systems with no less than 64 GB of real memory.

Management of RAW device based Oracle instances need not be complicated anymore. In fact transitioning from Cooked to RAW is relatively easy ... as A. Clay points out.. the use of links instead ot specifying the actual raw LVOL makes things a whole lot easier..

[Filipe_1]

More information:

Yes, the whole environment is on EMC.

The performance gain is easily bypassed by the three clear things:

1 - Ease of admin
2 - Veritas mount parameters wich disables or mimimizes the unix buffer cache.
3 - The VXFS robustness.

Jean, I din´t any benchmark, but these guys did. Very interesting:

http://www.jlcomp.demon.co.uk/raworfs_i.html

Its VERY interesting.

Also, there is this OTN link (requires registeration)


http://download-west.oracle.com/docs/html/A97297_01/ch2_tun.htm#i21051

Regards, and still needing more information on this issue,

Filipe

[Steven E. Protter]

A. Clay Stephenson actually took the time to run performance tests on this issue.

Raw versus filesystem.

On 10.20 there was a significant improvement going raw

On 11.00 there was an improvement though I believe he said it was debatable the improvement.

On 11.11 there was a negligible improvement raw versus filesystem.

Don't beleive me, believe the words themselves:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=234347

Yeah, i kinda sorta already asked this question.

:-)

SEP

[Michael Schulte zur Sur]

Filipe,

there is another small argument for file system based oracle.
You can shutdown the instance and do a cold backup.
Try that with raw devices!

greetings,

Michael

[Todd McDaniel_1]

I have a large system... 72GB memory. 2TB of DB space on one box... and another one with 32GB memory with 4TB of DB space...

We went with mounted b/c it is MUCH easier to manage all that disk space while mounted... and with Loadbalancing tools like PowerPath along with our latest Hardware of a Superdome.

Also, we use the caching mount options along with striping for better read times.


nodatainlog,mincache=direct,convosync=direct


My argument as an SA for mounted is:

1) much easier to manage hundreds of LUNS.
2) Powerpath or similar solves many of the problems with throughput
3) did I mention easier to manage?
4) DBAs can manage their data more efficently


My DBAs were the ones who actually recommended it...


Here are a few docs online...

http://www.oracle.com/ip/std_infrastructure/blueprints/emailonhp/blueprint_email/Design/Configuration_Options/Files_Systems_versus_Raw.htm

http://www.ixora.com.au/q+a/0010/18224940.htm

[Jean-Luc Oudart]

Filipe,

the powerPoint presentation is dated Feb'98 I guess vxfs may have improved since.

If the test program is available, has anybody run this test on HPUX 11.0 11i ?
What are the results ?

In SEP's thread I was wondering if redo on raw would be a good choice (this is as I do not have OnlineJFS). I suppose the choice is much clearer for you.

Jean-Luc

[Todd McDaniel_1]

Hey,

I found some more my DBA sent me... I can give you the metalink IDs and post them here...


Doc ID: Note:37914.1 and Note:29676.1

[Wim Rombauts]

Filipe,

As you can read in the previous entries : using a filesystem for Oracle datafiles does not really bring performance down, and it gives you a lot of flexibility : autoextending datafiles, cold backups, a lot of filesystem management tools, ...

The only reason I see to ise RAW device files, is when you plan to use oracle RAC in the future, because that is only supported on raw device fies.

I am using oracle RAC and thus need raw device files, and I think raw device files are a serious disadvantage in managebility and flexibility. Sometimes I still dream about the days before RAC made everything more difficult.

[Christian Gebhardt]

hi

to Wim:
we have built an Oracle RAC on Solaris 2.9 with "Veritas DBE/AC for Oracle RAC,Version 3.5". This Package from Veritas allows you to build an Oracle RAC on Filesystems.

We tested RAC on Solaris 2.9 with Solaris Cluster 3.0 on Raw Device and there was no measurable difference.

Details:
Oracle 9.2.0.4, 4 Nodes in 2 locations (about 14 km distance), EMC L1000 on both sides, Size of the database 100GB.

Chris

[Mobeen_1]

Filipe,
You have bought to the forefront a discussion which i deem interesting.

Please review the link below for some advantages of filesystems
http://oss.sgi.com/projects/xfs/papers/xfs_white/xfs_white_paper.html

The following is what Oracle thinks :-)

Raw Devices vs. Filesystems
---------------------------
When possible, advanced file systems such as xfs, jfs, or vxfs should be preferred over ufs file systems. This is to utilize the improved journal and performance features (such as elimination of double buffering) that advanced file systems offer over ufs. It must be noted that if you configure your system using ufs file systems the necessary file system level parameters need to be configured and tuned, as the default values will not provide optimal performance.
When using Veritas vxfs file systems, Quick I/O can provide raw device comparable performance without losing the benefits of a file system. This is because the Quick I/O driver intercepts all DBWR writes (when enabled) thus bypassing the file system buffer cache. This provides raw device comparable performance, as the classic problems of double buffering and wasted CPU
cycles in managing the file system buffer cache are avoided. When using other types of advanced file systems, Direct I/O can be preferred if supported by the OS, as it too provides raw device comparable performance. The use of raw devices does not add any significant value when compared to advanced file systems with Quick/Direct I/O drivers (wherever applicable).
Raw devices do add a level of operational complexity without justifiable performance benefits, when compared to advanced file systems configured with Veritas Quick I/O or Direct I/O (supported by the OS). In these cases, the configuration and use of raw devices should be reserved for Oracle Parallel Server implementations.
Asynchronous I-O
Oracle configured with asynchronous I/O has been found to work effectively only on raw devices across most flavors of UNIX. Asynchronous I/O on regular file systems is supported on some operating systems such as Solaris. Depending on your I/O system configuration, you may observe that Direct I/O or Quick I/O can offer comparable performance when using specialized file
systems such as Veritas (vxfs). You may also have the option of configuring the relevant Oracle instance parameters for multiple database writer support. But normally you either enable asynchronous I/O or multiple database writers, not both.

regards
Mobeen

[Hein van den Heuvel]

Wim> Sometimes I still dream about the days before RAC made everything more difficult.

Ha, it makes me dream of Alpha just continueing (with A |d|i|g|i|t|a|l| logo on the box :-), with Tru64 and a native Cluster File System, Single System images, transparant cluster management, transparant disk serving, transparent mutli-path.

Soon, but not soon enough to hold you breath for it, coming to an hpux (11.31) system near you! (or 11.33, whatever). In our (Nashua, NH) labs today.

Grins,
Hein.

[Todd McDaniel_1]

NO points here...

I just rememebered a few more Pros for mounted oracle FS...

1) BCV Snapshots for Cold backups and Point-in-Time transaction reporting...
2) RMAN backups


I never liked RAW even when there was no other option, so I may be a bit jaded against raw devices for Oracle.