topic Re: Direct io (mincache and convosync) not working? in Operating System - HP-UX

Direct io (mincache and convosync) not working?

Danny Petterson - DK — Wed, 16 Jun 2010 16:30:33 GMT

Hi!

Have a 11.31-box, with somes Oracle Database. The DBA wants direct IO. Mounted the database-disk with mincache and convosync:

/oracle/ORADB1/databases on /dev/vg21/lvol2 ioerror=mwdisable,largefiles,mincache=direct,delaylog,nodatainlog,convosync=direct,dev=80210002

So far so good, however, this is what tusc tells me about direct IO:

[17520] open("/oracle/ORADB2/databases/XTL/oradata/test01.dbf", O_RDONLY|0x800, 01210) ................................................. = 17
[17520] open("/oracle/ORADB2/databases/XTL/oradata/test01.dbf", O_RDONLY|0x800, 07645) ................................................. = 17
[17520] open("/oracle/ORADB2/databases/XTL/oradata/test01.dbf", O_RDWR|0x800|O_DSYNC, 030) ............................................. = 17
[17520] open("/oracle/ORADB2/databases/XTL/oradata/test01.dbf", O_RDONLY|0x800, 01210) ................................................. = 17
[17520] open("/oracle/ORADB2/databases/XTL/oradata/test01.dbf", O_RDONLY|0x800, 07645) ................................................. = 17
[17520] open("/oracle/ORADB2/databases/XTL/oradata/test01.dbf", O_RDWR|0x800|O_DSYNC, 030) ............................................. = 17

...indicating that Direct IO is NOT used. And the DBA tells me that to.

How come? What to do?

Thanks in advance.

Greetings
Danny

Re: Direct io (mincache and convosync) not working?

Tim Nelson — Wed, 16 Jun 2010 17:48:02 GMT

I believe you need OnlineJFS licensed in order to enable these.. ( could be wrong, it has been a long time )..

Re: Direct io (mincache and convosync) not working?

Tim Nelson — Wed, 16 Jun 2010 18:05:07 GMT

Disregard my response.. I cannot find proof of the comment anywhere.. I must have been dreaming...

Sorry...

Re: Direct io (mincache and convosync) not working?

James R. Ferguson — Wed, 16 Jun 2010 18:19:00 GMT

Hi:

> Tim: Disregard my response.. I cannot find proof of the comment anywhere.. I must have been dreaming...

No, you're not :-) The OnlineJFS requirement for 'convosync' and 'mincache=direct' are documented in the 'mount_vxfs(1M)' manpages.

However, if I recall correctly, without OnlinejFS the mount would fail if these options were used. A 'cat /etc/mnttab' will show what mount options are in force.

Regards!

...JRF...

Re: Direct io (mincache and convosync) not working?

Danny Petterson - DK — Wed, 16 Jun 2010 21:43:11 GMT

Hi Gurus!

Thanks for your answers - and its true, these mount-options only work with OnlineJFS, which is installed on the system - the line in my initial mail, showing the mount-options for the mountpoint, are in fact from the "mount"-command - showing that the convosync and mincache are already in effect.

But still - the tusc should return direct-io for the database-files, not O_RDONLY.

So - any ideas would be appreciated.

Thanks.
Danny

Re: Direct io (mincache and convosync) not working?

Duncan Edmonstone — Thu, 17 Jun 2010 06:19:35 GMT

Danny,

ARe you sure you are interpreting what you see correctly here?

Cos what I see is the same file (/oracle/ORADB2/databases/XTL/oradata/test01.dbf) being opened read-only (maybe as part of the header checks oracle does at startup), followed by it being opened read-write with the O_DSYNC flag. If you look at the mount_vxfs man page you will see when you mount with mincache=direct,convosync=direct then any files opened with O_DSYNC will bypass the filesystem buffer cache and do direct IO.

HTH

Duncan

Re: Direct io (mincache and convosync) not working?

Danny Petterson - DK — Thu, 17 Jun 2010 08:22:31 GMT

Hi Duncan!

Thanks a bunch for your reply.

Hm - you have a valid point there, no doubt. I was trusting a MetaLink-document 555601.1, which describes how to check this. This note says, that if a datafile is opened with direct IO, the open-statements from tusc should include O_DIRECT.

To kind of verify this, I tried to:
1 - create a database-file on a filesystem, mounted without convosync and mincache=direct. It still says O_DSYNC.

2 - Alter the database to not use direct IO - still, database-files on filesystems both with or without the aforementioned mountoptions is O_DSYNC.

I might misinterpretate something but initially this kind of shows that direct IO is not working.

Im pretty sure I'm missing something crucial here, I just don't know what.

Again, thanks a lot so far.
Greetings
Danny

Re: Direct io (mincache and convosync) not working?

Duncan Edmonstone — Thu, 17 Jun 2010 09:27:20 GMT

Oracle _always_ opens files R/W with the O_DSYNC flag, regardless of what the filesystem is set up to do.

Read the mount_vxfs man page carefully, and also have a look at what the VX_DIRECT caching advisory does when direct IO is enabled (see the vxfsio(7) man page)

HTH

Duncan

Re: Direct io (mincache and convosync) not working?

Danny Petterson - DK — Thu, 17 Jun 2010 10:00:21 GMT

Hm - interesting - thanks.

IF it succeeds to get the Oracle database-file opened using Direct IO, will tusc then show the O_DIRECT-flag?

Sorry for all the stupid questions.

Greetings
Danny

Re: Direct io (mincache and convosync) not working?

Dennis Handly — Thu, 17 Jun 2010 10:00:54 GMT

>indicating that Direct IO is NOT used. And the DBA tells me that to.

To get O_DIRECT, the application must use that in its open. I suppose open(2) could modify the system call based on the filesystem mount types.

Re: Direct io (mincache and convosync) not working?

Duncan Edmonstone — Thu, 17 Jun 2010 12:48:01 GMT

OK, now I'm doubting what I told you!

Just read an Oracle metalink article (555601.1 if you or your DBA has access).

Now I don't have 100% faith in the accuracy of metalink articles, but this one does suggest you should see O_DIRECT in the open call...

So a couple of questions...

1. This is a tusc trace of a database writer process isn't it? (ora_dbw*_*)

2. What is the oracle parameter "filesystemio_options" set to??

Sorry I can't test this myself until next week...

HTH

Duncan

Re: Direct io (mincache and convosync) not working?

Danny Petterson - DK — Thu, 17 Jun 2010 12:54:23 GMT

Thanks again for your prompt replies.

1) Yes, precisely

2) filesystemio_options has been tried with "setall" and "directio" - no difference.

Greetings
Danny

Re: Direct io (mincache and convosync) not working?

Duncan Edmonstone — Thu, 17 Jun 2010 13:36:58 GMT

What does:

pfiles

say?

e.g. if your dbw PID is stil 17520

pfiles 17520

HTH

Duncan

Re: Direct io (mincache and convosync) not working?

chris huys_4 — Thu, 17 Jun 2010 20:41:40 GMT

Hi,

Im also not so sure, what tusc should show, but I do know that the tusc output isnt directly corresponding with the mount output in the "opening statement".

I.e. in the mount output, "/oracle/ORADB1/databases" output is shown and in the tusc output, "/oracle/ORADB2/databases" output is shown, what unless the 2 directories are linked somehow, looks like different mountpoints.

Also normally for oracle databases, you only want direct IO for oracle database access and buffered IO for all "other access". And for direct IO, for only oracle database access, convosync=direct is enough.

NOTE: what definitely will show if direct IO is used, is kitrace output, but for being able to interprete the kitrace output you will need a call logged with HP support.

Greetz,
Chris

Re: Direct io (mincache and convosync) not working?

Volker Borowski — Fri, 18 Jun 2010 06:44:00 GMT

Hi,

is this open-flag still relevant with async in place?
If async is used, db_wr talks to the async driver,
which should do the effective IO
depending on the mount options.

I think you might just see the expected options in open, when you disable async and configure the old fashioned io_slaves???
(guessing)

Volker

Re: Direct io (mincache and convosync) not working?

Duncan Edmonstone — Fri, 18 Jun 2010 06:55:13 GMT

Volker,

I don't think async can be in place... unless I'm mistaken async either happens on raw/ASM or when using ODM which requires VxVM.

this appears to be on a filesystem on an LVM logical volume, so I can't see that async could be used...

HTH

Duncan

Re: Direct io (mincache and convosync) not working?

Danny Petterson - DK — Fri, 18 Jun 2010 07:32:26 GMT

Hi guys!

I really appreciate all the comments.

Duncan - pfiles for the databasewrite shows this for all database-files using this mountpoint;
flags = O_RDWR|O_DSYNC|O_LARGEFILE|O_EXCL

...and yes, its a vxfs-filessytem on a lvol.

Chris - We only use convosync=direct on mountpoints with database-files

Hope we are getting there :-)

Greetings
Danny

Re: Direct io (mincache and convosync) not working?

Volker Borowski — Fri, 18 Jun 2010 07:57:46 GMT

Hi Duncan,

is that still true? I thought that restriction
is outdated meanwhile. But I did not check
recently directly at Oracle.

Volker

2004: http://www.oracle.com/technology/deploy/performance/pdf/TWP_Oracle_HP_files.pdf

See 5.1.1.

On HP-UX, asynchronous IO is only supported with a raw device (raw disk partition or raw logical volume), although this will change with HP-UX 11i v3 (internally known as 11.31).

2006: http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=992511&admit=109447626+1274960107832+28353475

Re: Direct io (mincache and convosync) not working?

Duncan Edmonstone — Fri, 18 Jun 2010 08:54:15 GMT

Volker,

Yes and No!

I think that WP is possibly old enough to have expected 11iv3 to have AdvFS with I think supported async IO - of course we never did get AdvFS on HP-UX.

We do now have concurrent IO, with VxFS5.01 on 11iv3 which delivers near raw performance. I've not had the opportunity to understand CIO properly and understand if it is just another name for AIO, but it does sound similar to my (simple) mind. There's a WP on it here:

http://www.hp.com/go/ojfsperf

Of course in this case Danny isn't using cio, as it would have shown up as one of his mount options...

HTH

Duncan

Re: Direct io (mincache and convosync) not working?

Duncan Edmonstone — Fri, 18 Jun 2010 08:55:55 GMT

Danny,

Sorry but I really can't do anything else now until I can get back to my test system in the UK - that will be next week.

Did you get a call logged with the RC?

HTH

Duncan