Low Buffer cache Hit Ratio's & I/O bottlenecks

[Conal O'Kelly]

Hi

Reviewing the previous weeks performance stats, I have noticed that Cache Hit % is only averaging about 75%.
Today the DBAs are copying large oracle datafiles from 1 filesystem to another. Cache Hit % is now averaging about 50%.
System is connected via a SAN to an EMC Symmetrix.
System is configured with a fixed Buffer cache of 200M. (System has 6GB of phys memory) and we have no memory or CPU bottlenecks.
As a result of the all the copies the system appears to be bottlenecked on I/O (Processes blocked on resources averaging about 6-8). It takes a long time to login and even running simple 'bdf', takes about 20-30secs.
Is there likely to be any benefit in increasing the Buffer Cache or is there anything else I should be looking at??

Thanks for any help.

Cheers
Con

[Michael Tully]

Hi Con,

These two links provide some good information. I generally use the buffer cache somewhere over 300Mb, but not a fixed value.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x6752a22831ebd5118ff40090279cd0f9,00.html
and
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0xf49203bbece8d5118ff40090279cd0f9,00.html

HTH
~Michael~

[Conal O'Kelly]

Thanks Michael.
I have read alot of the discussions in the past on buffer cache sizing, however I guess what im really trying to ask is if the current cache hit %'s are a cause for concern (especially since we are using a EMC symmetrix with its own cache.). Also I presume any disk to disk copies will stil use the buffer cache.
Cheers
Con

[Stefan Farrelly]

"Your system appears to be blocked on I/O while doing large databse copies" - of course its going to be blocked on I/O !! What else would it be blocked on ? Doing large file copies is going to max out the I/O interface to your EMC for certain - thats the bottleneck. If youre running fibre then the speed of the copy will be limited by the speed of the fibre card on the HP. The EMC with tons of cache and quick disks can go a lot faster - but its designed to cope with multiple hosts all doing copies at the same time.

Your buffer cache hit ratio only being about 50% is absolutely fine, nothing to worry about - because the copy is constantly loading new data into the cache before it goes to the new disk(s) and the cache is designed to have a high hit ratio if it has the same data kept in it (ie. an application program - once loaded into cache if multiple users access it then its always in cache after the first load).

[Nick Wickens]

If your Oracle databases are set up to use "raw" access to the disks rather than via LVM (Ie you access the disks via the "rlvol" device file rather than the "lvol" ) as is the case with most Informix databases then buffer cache is largely irrelevent unless the system is used for more than just a database server because buffer cache will not be used for database access. Informix (or IBM as they like to be called now) infact do not like supporting LVM at all considering buffer cache to be a danger in the case of system failures as it can lead to data integrity problems (ie the database logging thinks that the data has been written to the disk when in fact it was still in buffer cache when the failure occured.)

[Stefan Farrelly]

Hi Nick,

interesting info. I find it wierd that Informix dont like using LVM (and buffer cache) for DB consistency reasons yet Oracle dont mind at all. Most places ive worked use Oracle and always they use LVM (not raw). So Oracle must have a safe way around a crash in case the buffer cache wasnt flushed yet - in the event of a crash it simply rolls back to the last point where writes were committed. And lets face it, Oracle is used in a lot more companies than Informix. I guess now that IBM own it theyre going to merge with DB2 or close down informix altogether ?

How is it still down at FE ?

[Trevor Dyson]

Even if buffer cache is used, the application can choose to do a synchronous write which ensures that if the app/db needs the data to really hit the disk straight away, then it will.

[Bill Hassell · Accepted solution]

A higher buffer cache size (perhaps 500 megs) will help, but the interactive delays you are seeing (ie, login) are largely due to sequential I/O such as copying files to another server. HP-UX gives preference to sequential I/O, so much so that it can severely delay random I/O such as a login request.

A new parameter has been introduced with recent patches: disksort_seconds. Note that the SAM Help on Context file and a section 5 man page are still missing but the patch documentation has the details. Here is an excerpt from the 11.00 patch:

PHKL_21768:

The system sometimes takes a very long time to respond to a disk read/write request (could be up to several hundred seconds) while it is busy processing other I/O requests on the same disk, especially when there are sequential file accesses going on.

This is a fairness problem with the disk sort algorithm. The disk sort algorithm is used to reduce the disk head retractions. With this algorithm, all I/O requests with the same priority are queued in non-descending order of disk block number before being processed if the queue is not empty. When requests come in faster than they can be processed, the queue becomes longer, the time needed to perform one scan (from smallest block number to largest block number of the disk) could be very long in the worst case scenarios.

It is unfair for the request which came in early but has been continuously pushed back to the end of the queue because it has a large block number or it just missed the current scan. These kind of unlucky requests could line up in the queue for as long as the time needed for processing a whole scan (which could take a few minutes). This situation usually happens when a process tries to access a disk while another process is performing sequential accesses to the same disk.

Resolution:

To prevent this problem from happening, we have to take the time aspect into consideration in the sorting algorithm. We add a time stamp for each request when it is enqueued, which is used as the second sorting key for the queue (1st key: process priority; 2nd key: enqueued time; 3rd key: block number). The granularity of the time stamp value is controlled by a new tunable "disksort_seconds".

If we set "disksort_seconds" to N (N>0), for all the requests with the same priority, we can guarantee that any given request will be processed earlier than those which come in N seconds later than this request. Within each N second period (requests have the same time stamp), all requests are sorted by non-descending block number order. By choosing the right "disksort_seconds" value, we can balance the maximum waiting time of requests and the efficiency of disk accesses. The tunable parameter can be set to 0, 1, 2, 4, 8, 16, 32, 64, 128 or 256 second(s). If "disksort_seconds" is 0 (default value), the time stamp is disabled, which means that time aspect is not taking effect.

[Nick Wickens]

Following on from Trevors comment and for the record I have found that there is an option to the mount command which will allow direct access to LVM volumes avioding buffer cache and the option is mincache=direct (one of the -o variables to that command). I have yet to try this though.

Hi Stefan - yes things are fine - You still hanging around with Richard S ?

[Bill Hassell]

To bypass the buffer cache completely (for vxfs filesystems only), you need 3 mount options:

mincache=direct
convosync=direct
nodatainlog

These option will not exist without the Online JFS product installed. The fstab entry would look like:

/dev/vg02/lvol3 /u03 vxfs rw,nosuid,delaylog,mincache=direct,convosync=direct,nodatainlog 0 3

This means that every write to the filesystem is unbuffered. This is often desirable for Oracle data directories (but not for rollback logs, archives, etc) when the Oracle SGA defines a large data cache.

[keith persons]

A word of caution, the mincache and convosync mount options should be used ONLY for the database mount point ONLY. Using these options on other file systems can heavily degrade throughput.

[Tim D Fulford]

On the cooked V raw I have an opionion...

I'm an Informix guy and saw the comments about Informix using raw LVM and Oracle using cooked. The reasoning is really quite simple.

1 - Efficient use of memory - Dbase buffers keep most frequently/recent used pages there - HP-UX buffer cache is used to keep most recent pages there. If you have an OLTP DBase (NOT OLAP, or DSS as these tend to flush the buffers anyway), then using cooked filesystems means that the pages are in memory twice, one in the DBase buffers & one in buffer cache, very inefficient.
2 - There is a possiblity that the DBase could do a write & crash before the buffer cache has flused to disk
Given the above you would pick raw....

On the other hand....
1 - It is easier to defragment a cooked filesystem & therefore improve performance this way.
2 - Memory is cheap & plentiful.
3 - The performance of cooked filesystems can be better than raw (so I'm told).
4 - JFS should not allow any data loss

From the above you can see there are arguments either way and it tends to be that people take up what their vendors say.

On the % cache ratio, if you copied the raw LV's to raw LV's it should not pass through the buffercache at all....

That way my 0.02??? worth

Tim