R/W NFS between LAN-Connected HP-UX 11.11 environments - UDP or TCP Transport?

[Alzhy]

The environments are on Gigabit inter-connect. The NFS client will be pretty I/O intensive so the NFS server will practically be acting like as NAS Filer (network attached storage).

What transport protocol should I use?

UDP or TCP

My experience in the past (Linux and Solaris )has been to always use UDP for LAN NFS clients. I think most filers/NAS I've used in the past also uses UDP...

[David Lodge]

For NFS on a NAS box I'd use UDP. It's got less overhead than TCP (good with lots of data) and you don't really need the virtual circuit connections of TCP.

If it needs to be secure then I'd not use NFS at all :-)

[Alzhy]

Well reason I ask is the CIFS Server/Client solution does not seem to fit the bill. My CIFS Client Connection are encountering NFS retries, rpc errors..

[Steven E. Protter]

CIFS is as your error shows running on top of NFS client.

NFS is likely to get the same error.

Ideally, you would like a fiber connection to storaqe. This will provide you what you need for something that is i/o intensive. Network solution might not be appropriate here.

If you want decent file locking you want NFS with TCP, not UDP. Still, NFS 3 might not give you the locking you need.

Then you are faced with the need for NFS 4 and HP's desire to force you to upgrade to 11i v2 to get it.

I believe the enhancements to NFS 4 will help with the problems you are encountering.

I'm actually undecided on your core issue. The CIFS client is supposed to be a TCP client. One one wonder under the circumstances how well NFS UDP client will work.

My suggestion is radical. Try it both ways and see if either works. If not, call your HP rep and ask them why you can not have NFS 4 for HP-UX 11i v1.

SEP

[Alzhy]

Thanks SEP.

I know this has been asked before - but why would CIFS use NFS at all? I had a difficult time explaining to a client about this issue. The fact that a Windows server was serving out the share with no NFS involved at all - and yet you have CIFS client complaiing that it is an NFS issue!?

Right now.. the tests proved NFS seems to fit the bill. Tests were run and no NFS timeouts at all and even quotas and setuid/setgid files work fine with NFS. I use UDP since this seem to offer better throughput and performance than TCP...

[rick jones]

If your infrastructure supports it, consider using JumboFrames. The 9000 byte MTU will help lessen the number of IP datagram fragments required to carry the UDP datagram carrying the write reply or read request.

If the NICs in your systems support it, you could also consider using "large send" or TSO, which stands for Tcp Segmentation Offload - this allows the sending TCP to pretend that the MSS for the connection is >> the "real" MSS and then the NIC will resegment to the apropriate segment size. I believe this can be used with either standard 1500 byte MTU or JumboFrames.

TSO depends on CKO (ChecKsum Offload) and is only for TCP. I think the HP-UX docs (lanadmin etc) will talk about a "VMTU" or virtual MTU

[Alzhy]

I actually thought about this when I left the office.

TSO has been on my radar screen for the last year or so having been prodded by a colleauge already using it on 11i v2 environemnts (which was the first environment to support this IIRC).

We are actually using APA and last time I looked TSO did not support APA (yet).. this probably has changed already as my recent scan of the GIGETHER-01 package seem to make mention of TSO as already supportive of APA..

I'll keep this forum posted on the results of our enabling TSO in our environment. But right now, I guess I am on the right direction to use UDP as the NFS transport..

[RAC_1]

If you are using UDP, also look at increasing the no. of biod daemons. This would help a little bit. Increasing the number of biod damons is not for TCP.

[rick jones]

Minor nit - it would be APA that provides support for TSO, not the other way 'round. Still, APA would be a consideration in the equation.

I trust you have decent distribution of traffic across the links per your selection of load balancing algorithm?

[Dave Olker · Accepted solution]

My recommendation is to use TCP whenever possible. Certainly test with both if you have the time and resources to do so, but the combination of larger read/write sizes available with NFS V3 and the fact that vendors constantly improve their TCP/IP performance have resulted in the situation where it's very rare to find cases where UDP outperforms TCP, even in a LAN environment.

I've done quite a bit of testing with GigE connected systems, and I can get wire-speed NFS throughput over GigE using TCP. Also, TCP allows you to use features like TSO (previously mentioned by Rick Jones). However, there are a couple of known problems getting NFS to effectively use TSO, but these bugs are being addressed in an upcoming NFS patch.

I'd also like to address something RAC posted:

> If you are using UDP, also look at
> increasing the no. of biod daemons.
> This would help a little bit.
> Increasing the number of biod
> damons is not for TCP.

A couple of comments:

1. Increasing the number of biods beyond the default of 16 will almost never get you better performance. There may be very isolated cases where it will (mid/large multi-CPU clients with lots of asynchronous I/O), but in my experience 16 is more than adequate for most environments.

2. Biods are used equally by UDP and TCP. Why would changing the number of biods be a UDP-only consideration? Why is this "not for TCP"? Are you sure you're not confusing this with the number of nfsds on the server? Those are UDP-only, as TCP uses pools of kernel threads.

Regards,

Dave