Should Oracle database filesystems be on actual different EMC disks?

[Stuart Abramson_2]

All:

We had some performance problems recently and discovered that one of our Oracle Databases, although carefully distributed across some 35 different HyperVolumes, was actually jammed onto only 5 EMC physical disks.

(We filled out Symmetrix frame in "chunks", 50 disks on initial delivery, 20 more, 20 more, etc..)

Now, I know that all reads and writes from cache. But is there any value in actually distributing the Oracle Database file systems across separate PHYSICAL DISKS.

Please don't tell me about Oracle SAME (Stripe and Mirror Everything). We're NOT interested in doing that.

We're also not interested in doing the entire Symm as striped meta-volumes.

Stuart

[Leif Halvarsson_2]

Hi,
I am not sure about what kind of answere you want.
Each physical disk han handle a limited number of IOPS (perhaps 150 to 200) and, of course, the more disks your database is distributed across the better performance you can get. If your database realy need that performance is another question.

[Michael Schulte zur Sur]

Hi,

off course the standard statement from Oracle is: Use as many disks as you can. Do you have any hint, that I/O is a bottleneck in your system? Avoiding I/O is still better than distributing it. ;-)

greetings,

Michael

[Stuart Abramson_2]

1. I agree that the more disks across which you are distributed, the better your performance. The question pertains to the difference between DISKS and HYPERVOLUMES. A HyperVolume looks like a "physical disk" to UNIX and to the DBA, but is really a "slice" of a physical disk. The EMC Symmetrix Disk Array does all reads/writes to cache and "backends" the data to/from cache to disk.

2. Our "sar -d" avwait and avserv went from 10 ms normal to as much as a 100 ms on a recent stress test; causing everybody to wonder about the disks.

[Ashwani Kashyap]

We also have our symms configured as combination of striped metas and hypervolumes . We also filled our frames in two or three chunks .

We try to put all dtabase files on the striped meta as far as possible and we have large amount of cache . But sometimes we are not able to put all of them on the striped meta and so we put it on the hypers but we haven't seen any performance issues since all read and write is through cache .

In your case may be the front end FA was hogged up if it was shared with other hosts possibly due to a 1Gb and 2 Gb HBA's zoned to the same FA port .

35 hypers onto 5 Physicals . Whats the size of your physicals and hypers . Usually hyper volume are assigned from adjacent physicals and there is low chances for one host to hypers from the same physical disk .

[Ashwani Kashyap]

On EMC frames results of sar can be bogus .

TO get the real picture of the I/O you will have to run I/O tests agains the frame itself , specially against the FA and the backend DA's .

YOu can gather performance data on the symm through ECC .

[Jordan Bean]

Always concider the performance of the backend. When striping, always select hypers on different spindles. The storage array is a workhorse, but do treat it with the same respect as a host server. The cache is only there to help, but it really only helps when the same small amount of data is accessed most of the time. If your database activity is all over the place, then backend performance becomes a significant issue where separate spindles are crucial.

I try to avoid using metas unless it is administratively effecient to employ them. I also [try to] limit my metas to 8 hypers (the number of backend buses in the frame), with each on a different spindle and bus. For example, I would stripe logical volumes over 20 striped metas of 8 hypers each in a single volume group rather than manage 160 individual hypers in 10 volume groups. Further, I would distribute the metas over as many spindles as possible (say 48), but if the database acitivity is low, I may only need 16 spindles.

Keep in mind that, in older microcode, a single meta could only handle the same amount of io as, I think, four hypers. So you wouldn't gain any io performance by making the metas wider. Fortunately recent microcde has improved the io performance of meta-volumes.

I'll stop here because I'm afraid I may have made the issue more confusing.

[Alexander M. Ermes]

Hi there.
Did you setup some kind of load balancing with alternate paths ?
What about using PowerPath ?
How many connections from server(s) to EMC ?
How much cache installed ?
What kind of connection interfaces do you use ?
Some more info could be useful.
Rgds
Alexander M. Ermes