NFS over a WAN - moving files is slow

[RobertCarback]

We have a Linux server that mounts an HP-UX 11i filesystem via nfs over a WAN (Baltimore - Phila). The Linux server, every 2 minutes, does an 'ls' to get 50 of the names of the files that have been placed automatically in the nfs filesystem. The files are then moved (mv) to a local filesystem on the Linux server. The problem is that once a month, when there are around 1000 of these files (at a total of around 44MB -- 44K each) in the nfs filesystem, nfs is so slow that the 'ls' times out and gets 0 files instead of 50.

[spex]

Hi Robert,

Does the speed improve if you umount/mount the filesystem immediately before 'ls'?

PCS

[RobertCarback]

re: "Does the speed improve if you umount/mount the filesystem immediately before 'ls'?" --
I am unable to do this as it is a production server. This happens on the first of the month (when we process a lot of files)so I wouldn't be able to really test until 01/01/2007

[Dave Olker]

Hi Robert,

Are you using NFS Version 2 or 3 for this mount? Can you confirm whether the client is sending a READDIR or READDIRPLUS request to get the contents of the directory? Since your application seems to only care about the list of files, it might improve things dramatically if the client were always using READDIR. Especially over a WAN, a READDIR call/reply will be far more efficient than READDIRPLUS - in particular if there are 1000 or more files in the directory.

You should be able to verify this quickly by using the "nfsstat -c" command on the client and checking how many READDIR calls were made vs. READDIRPLUS calls. Please post the results here and if READDIRPLUS is being used we can talk about ways to reduce that or eliminate it.

Regards,

Dave

[RobertCarback]

Using Version 3. ran nfsstat --c and readdir and readdir+ are both '0 0%'

[RobertCarback]

On the Linux server nfstat -c shows --
readdir 420888 27%
readdirplus 1420 0%

[Dave Olker]

> On the Linux server nfstat -c shows --
> readdir 420888 27%
> readdirplus 1420 0%

The Linux system is the NFS client, so the HP-UX system not logging any READDIR or READDIRPLUS calls in the nfsstat -c output makes sense. The -c option only prints the client-side stats, so if the HP-UX box is the NFS server I wouldn't expect to see any stats.

Is the Linux client using UDP or TCP for the filesystem? The fact that the ls command "times out" would indicate that you're using "soft" mounts. Is this true?

My recommendations in this case would be:

1) Use TCP for the filesystem, not UDP
2) Use "hard" mounts, not soft
3) Disable READDIRPLUS on the HP-UX server

To do #3 you can use the following two adb commands:

# echo 'do_readdirplus/W 0' | adb -k -w /stand/vmunix /dev/mem

# echo 'do_readdirplus?W 0' | adb -k -w /stand/vmunix /dev/mem

The difference between these two commands is the "/" or "?" before the "W 0" option. One of the commands changes the running kernel in memory and the other changes the kernel on disk so the change will survive a reboot. If the kernel gets rebuilt for some reason (installing a kernel patch?) then you'll have to repeat this adb command as the do_readdirplus tunable will revert back to the default value of 1.

In order to verify that do_readdirplus is correctly turned off, you can use these two adb commands:


# echo 'do_readdirplus/D' | adb -k /stand/vmunix /dev/mem

# echo 'do_readdirplus?D' | adb -k /stand/vmunix /dev/mem

They should both show do_readdirplus set to 0.

Again, using TCP, hard, and disabling READDIRPLUS are the first 3 things I'd try. If those don't work we can look at other options.

Regards,

Dave

[RobertCarback]

1) Use TCP for the filesystem, not UDP
2) Use "hard" mounts, not soft
3) Disable READDIRPLUS on the HP-UX server


1.How can I determine whether it is using TCP or UDP?
2.How can I change to a "hard" mount?
3.Can I change readdirplus while users are on the system?

[Dave Olker]

> 1.How can I determine whether it is using TCP or UDP?

On most implementations you could issue the "nfsstat -m" command to tell you exactly how the filesystem is mounted. Unfortunately Linux doesn't support this.

One way to tell would be to log into the HP-UX system and use the "netstat -an | grep 2049" command. If you see TCP connections between your Linux client and the HP-UX system to port 2049 then you're using TCP.

Another way would be to issue the command "nfsstat -s" on the HP-UX system and check whether the system is servicing requests from "Connectionless oriented" or "Connection oriented" clients. Connectionless oriented = UDP. Connection oriented = TCP.


> 2.How can I change to a "hard" mount?

Are you using the "soft" mount option when you mount the filesystem? On most every NFS implementation the default is "hard" unless you specifically override this behavior and ask for "soft" semantics. If you've mounted the filesystem with "soft" then you'd need to unmount the filesystem and remount it without the "soft" option.

In most cases you can issue the "mount" command and it will display the NFS filesystems along with any mount options you specifically asked for in parenthesis.


> 3.Can I change readdirplus while users are on the system?

Yes. The change is dynamic. It takes effect immediately and is safe to use while users are on the system or NFS filesystem are exported.

Regards,

Dave

[RobertCarback]

1. netstat -an | grep 2049 shows mostly udp but 1 tcp --

tcp 0 0 *.2049 *.* LISTEN
udp 0 0 *.2049 *.*
udp 0 0 *.2049 *.*
udp 0 0 *.2049 *.*
udp 0 0 *.2049 *.*

nfsstat -s shows mostly connectionless ---

Connection oriented:
calls badcalls nullrecv
4855 0 0
badlen xdrcall dupchecks
0 0 1136
dupreqs
0
Connectionless oriented:
calls badcalls nullrecv
1545922 1 0
badlen xdrcall dupchecks
0 1 73834
dupreqs
2

2. The mount command looks as though it is 'hard' --

shsmxhp:/faxsystem on /usr/vsifax/ddf type nfs (rw,addr=192.168.102.24)

[Dave Olker · Accepted solution]

Hi Robert,

> 1. netstat -an | grep 2049 shows mostly udp but 1 tcp --
>
> tcp 0 0 *.2049 *.* LISTEN
> udp 0 0 *.2049 *.*
> udp 0 0 *.2049 *.*
> udp 0 0 *.2049 *.*
> udp 0 0 *.2049 *.*


The only TCP instance here is the listen socket, which means it's the nfsd daemon waiting for an inbound TCP connection. In other words, you have no TCP connections at this time.

> nfsstat -s shows mostly connectionless ---
>
> Connection oriented:
> calls badcalls nullrecv
> 4855 0 0
> badlen xdrcall dupchecks
> 0 0 1136
> dupreqs
> 0
> Connectionless oriented:
> calls badcalls nullrecv
> 1545922 1 0
> badlen xdrcall dupchecks
> 0 1 73834
> dupreqs
> 2


You're using UDP. No doubt about it.


> 2. The mount command looks as though it is 'hard' --
>
> shsmxhp:/faxsystem on /usr/vsifax/ddf type nfs (rw,addr=192.168.102.24)


This does appear to be a hard mount, but just to be sure you can mount the filesystem with the "hard" mount option if you wish.

The fact that you're using NFS over a WAN means this is an ideal candidate for TCP. UDP was really never designed for use in a WAN environment. The fact that your filesystem is using UDP tells me your Linux client may not even support TCP. If that's the case you may want to look at either getting the latest copy of nfsutils on this system or upgrading to a newer version of Linux that supports NFS/TCP filesystems.

What vendor and version of Linux are you running? Which version of the Linux kernel is this system running? What version of the nfsutils package is this Linux system running?

Regards,

Dave

[Todd Whitcher]

Hi Robert,

So it looks like your using UDP, which is a very bad idea over your WAN link.

Check your mount options on the linux server, on HP-UX its proto=tcp, and mount the NFS filesystem with TCP.

That should really make a difference.

[RobertCarback]

What vendor and version of Linux are you running? Which version of the Linux kernel is this system running? What version of the nfsutils package is this Linux system running?

Redhat Linux 3.2.3-20
Linux version 2.4.21-4.ELsmp

Is it actually called nfsutils? I don't see that

[Dave Olker]

> Is it actually called nfsutils? I don't see that

You can use the rpm tool to query the system for installed packages and look for the nfs-utils package:

# rpm -qa | grep nfs
nfs-utils-1.0.6-44EL

You can look for the latest nfs-utils package at the SourceForge NFS project:

http://sourceforge.net/projects/nfs

Before you do this you can always try forcibly mounting the filesystem with TCP using the syntax Todd provided. If you are not able to get a working TCP mount then I'd start looking at updating the nfs-utils package.

Regards,

Dave

[RobertCarback]

Thanks. I'm more HP-UX than Linux.

rpm -qa | grep nfs

nfs-utils-1.0.6-31EL

Is this up to date enough?
I will try the mount. Looks like the option in linux is to just mention 'type'. I will put in 'type,hard'

[Dave Olker]

You should also try to forcibly use TCP by using the "tcp" option, or whatever the equivalent option is for your flavor of Linux. The mount man page should document exactly how to request TCP semantics.

Regards,

Dave

[RobertCarback]

Sorry, I meant 'tcp', not 'type'

[John Collier]

Not that this will solve any of your problems, but I want to point out that you should keep in mind that you will NEVER see blazing quickness over a WAN link unless you are connected end-to-end over a T3 or more. Even a T3 would be much slower than your average Fast Ethernet segment.

Most of the WAN connections in Corp. America over a distance such as what you describe are going to be fractional T1s. The absolute best you can expect over a full T1 is only 1.54 Meg/Sec. When you consider that you are most likely not the only one using this link at any given time, it is not hard to understand how it can get bogged down.

It has been a long time since I have had to customize an NFS mount, but it memory serves correctly, I am fairly sure that you can adjust the packet size. If that is true, then I would think that adjusting it way down would be a good idea under the circumstances.

Of course, you have probably already thought of all of this, but since this is the one area that many people forget I thought I would include it. Perhaps it will be of value to someone searching the forums at a later time.