To "LVM" or not to "LVM"

[Eli Codd]

I am building an Oracle DB on a new HP VA2405.
I am running hpux 11.11 and I have (2) HBA's connect to the storage. I have redundancy groups 1 & 2 (RAID 0+1). I am in a quandry as to whether to use LVM or not. If I do, I can build LV's that utilize disk across the multiple redundancy groups and I/O channels. However this add overhead to the OS for LVM routines. It certainly gives me some flexabiity going forward, but is it worth it? If I use straight mounts with JFS and balance my I/O within Oracle, I save the overhead of LVM and let Oracle & Hardware dictate the performance. Does anyone have an experienced opinion?

[A. Clay Stephenson · Accepted solution]

Don't give it a second thought; LVM is your best choice. There is a very slight overhead to using LVM but in almost everycase it's impossible to measure. All that really happens when I/O goes to a major device 64 (LVM) is that a table lookup is done to see that that I/O should really go to the scsi disk driver. It is essentially one extra lookup in the device switch table. All of this takes place in the kernel and, in general, this delay is trivial compared to the time required for the physical I/O.

The ease of increasing LVOL size and the ability to stripe across multiple paths mores than offsets the tiny additional I/O overhead.

By the way, 11.11 Oracle seems to actually perform better with 'cooked' I/O (i.e. using the buffer cache) rather than bypassing with convosync=direct,mincache=direct which usually performed better on 10.20 and 11.0 Oracle).

[Jim Turner]

Hi Eli,

VA2405? Did you mean VA74x0 with one or more DS2405 shelves added? The 74x0 is a Virtual Array, and it can have one or more 2405 disk shelves added to it. By itself, a DS2405 is simply dual-pathed FC JBOD.

Nothing I can add to what Clay said other than, "Yeah, what he said." However, I can mention a bit on segregation since you have the luxury of two redundancy groups with the VA74x0.

Examine current database usage, and analyze where the bulk of your I/O resides with respect to data, index, RBS, redo logs, archive logs, etc. You will want to split that I/O between the redundancy groups. BTW, you should probably set each redundancy group to use a unique FC path. That way, when you hammer data and index (for example), they will be firing on different sets of spindles and FC paths.

Cheers,
Jim

[Eli Codd]

Clay.. Thanks..
Jim... You are correct about the VA model...

A side question for you folks.. Being a VA newby... I am assuming when I grab XXXgb from a redundancy group it grabs an equal amount from each drive in the redundency group. Thereby spreading out the I/O as far and wide as possible. Correct??

[Jim Turner]

Correct. (Dual) Parity is striped (spiralled, actually) across all disks in the RG. One RG group uses the even-numbered drives, and the other RG uses the odd-numbered disks. So don't scratch your head when you see that pattern during operation.

How the data/parity is striped across the disks, and (in the case of the "AutoRAID" setting) what data is kept as RAID0+1 vice RAID5 is determined by the controller firmware and is out of your reach.

In case you're wondering, all frequently written or read data is kept at RAID0+1. Seldom used data is shelved to RAID5 thus maximizing the available space. On some installations, I segregate data (mountpoints) onto different RG's and FC paths as I mentioned before. In other instances, I've created one big LUN in each RG and striped the LV's across both.

And talk about bandwidth! These little hummers fly. I set up a test with dd to perform some very vile acts upon a VA w/one DS. This went beyond even some of the unspeakable things that Oracle tries to do. I was able to sustain a measured throughput of 90MB/s per FC path with no disk queue at all -- zero. Every I/O request was being satisfied in real time. I think I had about 30 - 35 read and write streams all cranking at the same time using various block sizes. I also made sure I was touching far more data than the controller cache could hold. I started to build a consistent disk queue when I cranked the number of full-blast, concurrent I/O streams up to around 50. Needless to say, I'm very pleased with the VA performance.

I've attached a technical paper that discusses the mathmatics behind RAID5 Dual-Parity in some detail.

Cheers,
Jim