Poor NFS write speeds

[Stephen Walton]

I am getting extremely slow writes to my K460 NFS server, HP-UX 10.20. I have tried everything I can find on the Web, to no avail: netperf shows no network speed issues. Varying the number of nfsd's from 4 to 32 didn't help. Changing rsize/wsize, NFS version (2 to 3 and back), async/sync, no change. To give some idea of the problem, a 'cp' of a 5 MB file takes 90 seconds to the K460, as compared to about 20 seconds for an 'rcp' of the same file. All systems are on a 100baseT switch. netperf shows a base speed of 60 Mbits/sec to the K460 for both TCP_STREAM and UDP_STREAM. It is not a client issue; writes to other systems including an ancient Pentium 150 running RedHat as well as to a Quantum SNAP server, are fine. The K460 has patch PHNE_25234 installed.

Help?

[Sajid_1]

hello,

If you compare NFS and rcp, then obviously 'rcp' will be faster. NFS is considered as the *slowest* among all file transaction methods (ftp, rcp, cpio, NFS ..etc).

What you can do is to compare NFS with NFS. Check other systems and it's settings. Check for any speed difference and then make the changes. The first thing I would do is to apply all latest patches to the system. Then the network traffic, concurrent processes, resource usage with Galnce etc.

gl,

[A. Clay Stephenson · Accepted solution]

Something is already wrong. 20 seconds for a 5MB rcp is a dog. It should be well under 2 secs on a 100MB FD connection.

Before you do anything else, ftp a file in both directions. FTP has the least overhead so that we can see the network transfer speed. It should be at least 3MB/s or so or you have a badly configured NIC, switch port, or a lousy cable.

[Stephen Walton]

Thanks for the quickie responses. There was an important typo in my original message: the 'rcp' test was on a 55 MB file, not a 5 MB file. Mea culpa. I should also have noted that I have done many other tests: K460 NFS client to/from Linux NFS server, K460 NFS client to Quantum SNAP server as NFS server, Linux NFS server to Linux NFS client. All show decent write speeds.

It is true that the K460 net connection may not be 100%: netperf clocks it at about 60 Megabits per second, while Pentium Linux to Pentium Linux comes in at over 90.

[Patrick Wallek]

What type of 100 Mb card do you have in the K460? The HPPB 100Mb cards will not perform as well as might be expected.

Another thing to check is to make sure that both your card and your switch are hard set at 100FD. Do not use auto-negotiate.

[Stephen Walton]

To answer everyone's questions: the card on the K460 is an HP brand HP-PB 100BaseTX card. I don't have the model number handy. Even though it is set to auto-negotiate, it seems to be fine. I've attached 'netperf' output to this message. "hpux" is our K460, and "p150" is a Pentium 150 running RedHat 7.2. The test system was an Athlon 550 running RedHat 7.3. All the speeds are quite high.

An FTP of a file from the Athlon to the K460 or vice versa runs at more than 5 Mbytes/sec. A 'cp' of that 55 MByte file I mentioned takes about 20 seconds to a file system NFS mounted from the Pentium 150, as compared to 90 seconds as I said before for the same file copied to a file system NFS mounted from the K460; the Athlon was the NFS client in both cases. Whatever the problem is, I can't find evidence that it is affecting any part of our networking other than NFS writes to the K460.

[A. Clay Stephenson]

Okay, now that the typo it is out of the way, I suppose that it time for me to think. I would first do an nfsstat -rc. Please post that. That should give you some clues.

One other thing that can help NFS performance is buffer cache; normally on 10.20 I run fixed buffer cache with bufpages set to 80000 (320MB) but heavy-duty NFS servers benefit from a larger buffer cache of maybe 800MB (if you have enough memory for this). You do not want to run large buffer caches with dynamic buffer cache on 10.20.

[Stephen Walton]

Here is the output of 'nfsstat -rc' on the machine in question:

Client rpc:
calls badcalls retrans badxid timeout wait newcred
22522710 8 746 0 737 0 0

And for good measure, the output of 'nfsstat -rs':

Server rpc:
calls badcalls nullrecv badlen xdrcall nfsdrun
52217256 0 9238408 0 0 52217256

If there's anything wrong there, I'd like to know, but I think it looks fine. The system in question has 512 MB RAM. The kernel is using the default settings for caches (I think; at least I haven't changed them): dynamic buffer caches with dbc_max_pct set to 50 and db_min_pct set to 5. The NFS Tuning doc recommends setting default_disk_ir to 1 if your system is protected by UPS (ours is); it is presently zero on my server. Might that help?

Admin note: the summary list for this group now shows that I've stated someone's posting solved this problem, which it hasn't. Obviously I did something wrong.

[Patrick Wallek]

Just an FYI for you, the "Magic Rabbit" icon appears by a posting after you have 8, 9 or 10 points to ANY response to your question. You have assigned 10 points to one response and 8 points to a couple of responses, so the system things your problem has been solved.

This is discussed in the points documentation, by the way.

[Stephen Walton]

SOLVED! I R'd TFM. In the HP-UX NFS tuning notes (which were 11.0/11i specific but somewhat applied to 10.x as well) were some kernel parameter suggestions. The key one did indeed turn out to be setting default_disk_ir to 1! With that simple change, my write speeds increased almost fivefold.

Thanks to everyone who took the time to read & respond.

[rick jones]

That setting default _disk_ir to one helped is goodness - do keep in mind though that it means you are depending on the disc to successfully transfer the contents of its write cache each time - and if there is a powerfailure in the middle of a write, that may not happen.

also, with the nfsv3 stuff, you might want to make sure your client is doing mostly cached writes - that setting default_disc_ir to 1 helped for v3 may imply that the client was not doing cached writes, or perhaps the client was not actually doing v3?

with v2, if you are willing to live with the window of having default_disc_ir 1, you may also consider async mounts - you would set that in the exports file.

bummer about having to use the HP-PB NIC though - the HSC NIC is able to support 100BT link-rate...