NFS poor performance

[joseph wholey]

I'm in a real pinch... Need help!!!! I'm running RHEL AS3 on two servers (they're identical HW/SW). My NFS client is experiencing horrible performance. The current mount options are "soft". That's it. To give an idea of horrible, it takes between 20-30 to open a 160MB file on the client. On the server side, it takes about 3 seconds for the same file. I understand that there will be a bit of a delay on the client side, but what I'm experiencing is unreasonable. These two servers are actually in the same rack, same subnet and connected to the same switch. There are no errors on the ports on the switch. Can anyone help on a step by step troubleshooting procedure to rule things in and out?

[Steven E. Protter]

Shalom Joseph,

Define unreasonable? If it is resulting in user complaints, then its really is a problem.

There is further analysis that can be done on the switch.

NFS problems in general come from two places. Poor server performance and network issues.

Since you've posted your Linux thread to HP-UX, I'll give you the right diagnostics. ethtool eth0 Are you getting the speed you expect? If not you have a network configuration problem and should correct it.

You may wish to look at /var/log/messages and see what is going on there. tcpdump might help you if there is a non NFS congestion issue that is makeing things slow.

SEP

[joseph wholey]

Steven, I've given this a little more thought... it was a client who complained that they were unable to open a 160MB file on the NFS client in a reasonable amount of time.
I did an ftp of that file between the NFS server and the client and transmission rate was appx 11MB/sec (it's a 100MB nic). So if we do the math, (160MB/sec)/(11MB/sec), we get appx 14.5 seconds.
It's taking the client appx 15-16 seconds to "vi -R ". As far as I can tell, this is not really an NFS performance issue as much as it is a network constraint issue. Am I missing something???? Please throw in you 2 cents. thx.

[Mel Burslan]

Just from recent experience, you say that your NIC is at 100MBs (assuming you also meant Full duplex) but did you check with the network folks that the switch port you are attaching to, is configured at the same speed and duplex settings ? This is the most commonly overlooked mistake in my vicinity.

Hope this helps

[joseph wholey]

Thanks for the advice... I know I'm set up correctly from the server perspective, and I did have the port on the switch checked and they said all was well... but I never did ask what speed they had it set at. However, 9-11MB/sec is a reasonable trasfer rate for a 100MB nic. But, I will do due dilligence and check that. thx again.

[Dave Olker]

Hi Joseph,

I agree that 9-11 MB/sec is about right for a 100MB NIC to run, especially for NFS traffic.

If you really want to test your NFS throughput I recommend you grab a copy of iozone (http://www.iozone.org) and use it to measure your throughput. Iozone is a very powerful and flexible benchmarking tool capable of generating almost any type of I/O requests. I use it constantly in my NFS performance role.

I recommend starting with a sequential WRITE test and then doing a sequential READ test to see what kinds of numbers you get before you start branching out into other tests (like random I/O, mmap, etc.). I also suggest using a record size of 32K since that's what most NFS clients use. Finally, be sure to unmount and remount your NFS filesystem between tests in order to nullify any benefits of caching (in other words, you want your NFS requests to go over-the-wire and not be satisfied locally from cache).

You can find the documentation for iozone on the iozone.org web site, but if you have any problems using it feel free to post your questions.

Regards,

Dave

[Al Licause]

11MB/s translates to roughly 88M bits/second which for a 100Mbps pipe is good. So more than likely you are not having any datalink or physical layer problems and the nic matches the network port for speed and duplex.

So now you need to better define the nfs problem. When you say it takes x number of seconds ( and since you only listed numbers and no units for those numbers, one can only assume it was 20-30 seconds in your first post ) to open a file.....you need to understand that for any connection there is a handshake then a file open if it is a file access and then the transfer if a file is being moved.

So when you say it takes 20-30 seconds to open a file, do you really mean it took that long to transfer the file or simply to open it ? If so, how long did it actually take to transfer the file ?

You might want to consider using tcpdump to monitor the connection from estabishment to file transfer completion and close. Compare this to the system that is working as per your expectations.

Look for duplicate acks, dropped packets, and delays in responses on either side.

You may need to adjust some of the mount parameters, like timeo, retrans and packet sizes both read and write. You might also want to mount the directory using tcp rather than udp to see if it makes any difference.

Try this and post your results.