I concur with the others that say that there isn't much you can do here from a kernel tuning standpoint. There is, however, a lot you can do from an Oracle perspective to fix this.
Usually, the application dumps all of its data into a single tablespace. This is easy and convenient for the programmers, but is tough on the machines. Without redesigning the application, about all a sysadmin can do is stripe the filesystem on which the tablespace resides across several disks. HP, EMC and others sell systems that do this very nicely. This can provide a substantial increase in disk speed--especially with sequential reads that can come from cache. Random read slowness is usually impervious to hardware fixes.
Best deal is to redesign the application(s) to use more than one tablespace. It is far, far better to have several (even dozens) of tablespaces spread across numerous filesystems than a single (or very few) tablespaces on a single (or few) filesystems. The more filesystems in use, the more the system can access multiple data streams at the same time. Ultimately, a disk drive is its own bottleneck. Spread the load as much as possible amongst as many as possible.
Lastly, check to see that the application(s) are not doing full table scans, and that they're indexed as fully as possible. All by itself, using indexes--and otherwise properly designing the Oracle database--can yield a tenfold increase in I/O speed.
Use iostat to find out which filesystems are getting hammered. Usually, you'll find that only a few are so busy. Whichever tablespace, index or other file is so busy is the first one to move to faster and better i/o systems, or be subject to a redesign.
This is not a fast/easy fix. As a rule, Oracles suggestions for kernel tuning parameters should be followed, and modified only very carefully, and only after exhausting other possibilities.
Good Luck
Chris
