Steven - I'd stay away from putting together a large Oracle systems with just two big giant luns. And, I'd not use R5.
If you're using only two luns then your scsi queue depth is probably large. Please review sar -d and look at the avg queue depth. Also,
make undo, temp, redo logs, and archive logs separate luns on R0/1 and if I were you I'd make sure that they don't share disk on the storage array. Index and Data areas can be considered for R5 if your i/o is overwhelmingly reads, but for oltp, I prefer R0/1 pretty much regardless. Make sure that data and index areas are also on separate luns, and preferably on different physical disks on the storage server.
These suggestions are not hard and fast, they are preferred - if you have to bend in some areas, try to minimize them.
Look at statspack reports, and examine carefully those top I/O consuming sql statements. Look also at how your cache hit ratio is holding up (both on data cache and code - shared_pool, large_pool and db_cache_buffers.
If your cache hit ratios are low, you'll have to start increasing the SGA footprint in those key areas.
Make sure to interleave the redo logs, interleave the archive log output areas, and interleave redo space when possible (requires multiple mount points for each if you are trying to achieve this).
I think there are three things that are going to pull you out.
1) First and best one - identification and tuning of crud code. This takes the longest but has the biggest impact if the amount of code that needs tuning is large.
2) R0/1 instead of R5. This will be your second largest impact, but is conceptually quicker to do.
3) After you've tuned your code, if you hit ratios aren't in the high, high 90% numbers *(like 98% and staying there), then you should consider bigger cache areas, possibly bigger pool sizes - statspack will guide you on this).
We are the people our parents warned us about --Jimmy Buffett
