New 11.31 NFS patch bundle improves database performance

[Dave Olker]

The NFS lab has just released a new ONCplus update called B.11.31.04. This release is available for free download from:

http://software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=ONCplus

The B.11.31.04 version of ONCplus delivers the following new features and defect fixes:

* rpc.ypupdated, ypxfrd, and keyserv are not killed when the NIS Master Server is enabled and disabled by smh.
* Documentation issues in /etc/rc.config.d/namesvrs.
* Support for NGROUPS expansion in ONC.
* IPv6 support for Rcommands and Rdaemons Phase 1.
* Optimize direct I/O for NFSv2 and NFSv3 reads and writes.
* Remove nfs4_nrnode and nfs4_nacache private tunables from 11i v3.
* An NFSv4 mount overwrites the mount error returned by the server with ENOENT.
* NUMA support changes for CacheFS.
* Spinlock contention in checkexport/exi_rele.
* Serviceguard NFS failover fails in the presence of the "fsid=" option.
* Remove the NFSv4.1 server stubs for referrals.
* The xdr_hyper() and xdr_u_hyper() functions incorrectly process data.


One of the biggest performance improvements in this bundle is the revised Direct I/O behavior for NFS v2 and v3. This change has been shown to greatly improve database workloads, such as Oracle, when run over NFS.

Feel free to contact me if you have any questions about this new ISU bundle.

Regards,

Dave (dave.olker@hp.com)

[Steven E. Protter]

Shalom and Thanks Dave.

When is this stuff going to be back ported to 11.23 and 11.11?

SEP

[Tim Nelson]

Dave,

Thanks for the info.

Question: Are all the fixes/updates to ONCplus going to be a product release vs a typical HPUX patch ? ( maybe I know the answer to this if I think about it more, but.. )

Cheers.

[Dave Olker]

Hi SEP,

Most of the improvements we make to 11.31 will never be rolled back to 11.11 or 11.23.

There are several reason for this. The 11.31 NFS code base is anywhere from 10-15 years newer than the code base used for 11.11 and 11.23, so porting some of these things, like NFS v4, to that older code base would take a huge amount of resources and the port would never be as clean as on 11.31. Also, 11.31 has other significant subsystem changes (such as Unified File Cache) that enables us to provide some of this stuff, and since those features will not be rolled back to 11.11 and 11.23, NFS can't roll our stuff back without them.

If there is a specific feature you're curious about I'd be happy to research the feasibility of back-porting it, but for the most part we're investing the majority of our resources into making 11i v3 as feature-rich as possible.

Hope this helps,

Dave

[Dave Olker]

Hi Tim,

> Question: Are all the fixes/updates to
> ONCplus going to be a product release vs a
> typical HPUX patch ? ( maybe I know the
> answer to this if I think about it more,
> but.. )

The ONCplus bundles are effectively very similar to the patch bundles we deliver to previous HP-UX releases, but the lab decided to change to the ISU (Independent Software Unit) release model as this allows us the flexibility of providing bug fixes as well as new features.

As you're probably aware, we've never been allowed to deliver new functionality in patch bundles (we've tried working around this many times and often gotten grief for it) and the lab felt it was important to use a delivery model that gave us the ability to release new features mid-release rather than hold them for a future HP-UX release. We'd rather get the new stuff into customers hands as soon as possible, and the old patch bundle method hinders this.

So, in answer to your question: No, we have no plans to release the ONCplus fixes in a "patch". We will continue to update the ONCplus ISU bundles every 3-6 months and post them to http://software.hp.com. We also deliver these bundles to the periodic Application Releases, which means new systems will ship from the factory with some version of the ONCplus ISU bundle. It may not be the very latest version of the bundle, since there is a 6-month lag time when compiling the AR releases, but you can always grab the latest bundle from software.hp.com.

Hope this answers your question,

Dave

[Steven E. Protter]

Yes Dave,

Spinlock, database performance. We have a lot of oracle databases running on 11.11 and a performance boost would mean a lot to us.

SEP

[Tim Nelson]

Thanks Dave,

I am not sure if I ( as an admin ) like the IPR. In addition to staying current on HPUX I now need to add another product to the list.. I guess we already do it with products like ssh, glance, etc... one more won't hurt. just sometime like NFS is/was such a core functionality...

I'll get use to it.


Keep up the good work.

T.

[Dave Olker]

Hi SEP,

The database performance issue with 11.11 is difficult because we don't even have direct I/O support for 11.11, and most databases really like using direct I/O.

We do offer direct I/O support for 11.23 and 11.31 but I believe (could be wrong, this is going back a ways) there was an architectural reason why we could port the "forcedirectio" feature to 11.23 but not back to 11.11. I can try to find out if this is something important to you.

Regards,

Dave

[Steven E. Protter]

It is important.

With a 5 year life cycle before obsolescence, this feature is worth putting into 11.11 and 11.23

We do a lot of lab work with oracle on NFS and Linux has had v4 for a very long time.

SEP

[Dave Olker]

Just to be clear -

11.23 already has the forcedirectio feature.

Also, NFS v4 will not be rolled back to a release prior to 11.31. That feature required the Unified File Cache, which is only available on 11.31.

So, since direct I/O is available on 11.23 and NFS v4 will not be back-ported, are you still interested in the possibility of getting direct I/O for 11.11, or are your customers willing/able to upgrade to 11.23 to get this feature?

Dave

[rick jones]

Being pre-obsolescence neither precludes, nor prescribes new features will be added.

The "dynamic" between upgrade to later/latest OS to get new features and backporting is nothing new of course, and we can probably find examples of both.

However, in this case, it seems pretty clear that to get these features, it will be necessary to upgrade the base OS.

[Oviwan]

hey

oracle 11g includes his own direct nfs client.

are any test results available which one is faster? oracle direct nfs or the patch bundle?

here the document:
http://www.oracle.com/technology/deploy/performance/pdf/directnfsclient_11gr1_twp.pdf

Regards

[Steven E. Protter]

We have decided to bypass 11.23 altogether.

One of our major customers uses it in Itanium and we provide support because they are 8000 pound gorillas in revenue terms.

Are our other customers willing to upgrade at this time? No. The refuse. But they would benefit from a NFS v4 roll out in 11.11.

Greatly.

11.31 is where we are working on as a new HP-UX platform for those that want to stay with HP-UX. There is a lot of internal opposition to Itanium in general, but if we have to support HP-UX we want the longer shelf life.

HP's position on NFS v4 is costing it customers within the NDS customer base. They are going Linux.

SEP

[Dave Olker]

Hi SEP,

> But they would benefit from a NFS v4 roll out in 11.11.
>
> Greatly.
>
> HP's position on NFS v4 is costing it
> customers within the NDS customer base. They
> are going Linux.


This is something I'm really curious about. What specifically is there in NFS v4 that your customers feel they require, or that they cannot get outside of NFS v4?

Thanks,

Dave

[Steven E. Protter]

I may have to actually ask them Dave.

That is an act of political significance in my company.

An answer will take time.

Since there seems to be a benefit on the Oracle side, I will sit down with a DBA and ask them which of the features originally listed they think are important.

Assuming they understand the features, I can give you the internal answer in a day or so.

Talking to our customers formally versus how it has been done up to now (second and third hand) is going to take time.

SEP

[Dave Olker]

Hi Oviwan,

> oracle 11g includes his own direct nfs client.
>
> are any test results available which one is
> faster? oracle direct nfs or the patch bundle?


I apologize in advance for the length of this reply, but the answer is not really straight forward. Like most performance questions, the answer to this is "it depends". :)

The Oracle Direct NFS client can potentially outperform the built-in NFS stack in certain circumstances, but we've closed the gap significantly with the Direct I/O changes in ONCplus B.11.31.04.

The Oracle Direct NFS client is able to perform "asynchronous" direct I/O. The HP-UX 11.31 native NFS client currently doesn't have this capability, but the NFS lab is investigating the possibility of adding this feature.

The times where this asynchronous direct I/O capability really matters is when the database application attempts to do I/O's larger than the read/write size supported by the NFS client and server.

An example will probably help.

Let's say your application wants to issue a 256K write request to the NFS filesystem using direct I/O. If both the NFS client and server system support read/write sizes of 256K or larger then the HP-UX 11.31 (ONCplus B.11.31.04) client will issue the write requests as a single 256K transaction - even if you've mounted the filesystem with an rsize/wsize of less than 256K. In other words, if your NFS filesystem is mounted using an rsize/wsize value of 32K but both the NFS client and server systems *support* larger read/write sizes then the 11.31 client will figure this out and automatically send the request in a single 256K chunk.

The 11.31 NFS client supports read/write requests as large as 1MB by default, as does Solaris and newer versions of Linux. However, NetApps only supports 64K read/write requests. So if you're dealing with an HP-UX 11.31 NFS client and a NetApps filer as the NFS server then the largest direct I/O request the client can send in a single chunk is 64K, even though the client supports 1MB transfers.

In this case, the application sending the 256K write request would have it's request broken down into 4 separate 64K chunks and each chunk is processed synchronously (i.e. serially). In this situation, the Direct NFS client would likely outperform the 11.31 native client because it is able to schedule all 4 of these 64K I/Os asynchronously (i.e. in parallel).

So as you can see, the times where the Oracle Direct NFS client *might* outperform the native NFS client are when the application is attempting to perform I/O's larger than both the NFS client and server support. In that case, it's possible for the Oracle client to beat native 11.31, but it still depends a lot on the database configuration, workload, etc.

As I said earlier, the NFS lab is investigating the possibility of adding asynchronous direct I/O to the 11.31 client. In addition, I've asked NetApps to investigate the possibility of increasing their read/write size to 1MB (to match what HP, Sun and Linux have already done). They've agreed to investigate this. I don't have any information about when either of these changes will be available, but both are being investigated.

Again, if you're using an 11.31 (ONCplus B.11.31.04) NFS client and an NFS server that supports large I/O sizes then it's likely you won't see any performance difference between the Oracle Direct NFS client and the native 11.31 NFS client. Even if your NFS server only supports 64K request sizes, this only really matters if your database application is performing I/O requests larger than 64K.

Hope this helps,

Dave