totally bizarre NFS anomaly

[support_5]

Here is another snippet of interesting information:

I was looking in syslog for entries around the time when it started working. On the NFS server, I noticed these lines:

Sep 10 14:21:51 guam cmcld: lan2 recovered
Sep 10 14:21:51 guam cmcld: Subnet 172.22.242.0 switched from lan3 to lan2
Sep 10 14:21:51 guam cmcld: lan3 switched to lan2

Now, the NFS server and client are in a MC/ServiceGuard cluster arrangement. It would seem that for some reason, serviceguard changed the IP address for guam from being on LAN3, to LAN2. This could possibly be why it started working at about that time... (although that time is about half an hour earlier than when I first posted the message saying that it was working normally now. I don't recall it being that long from when I first noticed that it was working to when I posted the message saying it was working. 15 minutes is what I thought was about the time-frame.

(where it says in the forums, I posted at 04:50:52, that was actually 14:50:52 local time, so read that into the syslogs above.)

What do you think of that?

- Andy Gray

[Dave Olker]

Hi Andy,

To my knowledge, NFS doesn't have anything to do with SNMP. There may be some SNMP MIB statistics related to NFS or RPC, but I've never heard about an NFS problem that was resolved by restarting SNMP.

I find it very interesting that MC/ServiceGuard switched LAN interfaces on the NFS server around the time that things started working. This tells me there were problems with the network interface that probably manifested themselves in different ways that we haven't captured in this thread, but they were bad enough to have MC/SG switch the LAN over.

That being the case, you might have a defective piece of hardware on the server itself related to either LAN3 or LAN2. Could be a bad cable, NIC, or switch port. Any of these could cause packet corruption, and given the erratic behavior of typical corruption problems (and I've seen many of them) it doesn't surprise me that some packets get through while others hit a certain condition that triggers the UDP checksum failure.

Since both systems are connected to the same switch, it is unlikely a problem with the switch unless the switch port itself is having problems. I've seen them, but they're not as prevalent as cable or NIC problems.

In any case, it would be interesting to see what happens if the problem comes back if the server switched back to using LAN3.

The good news is you know how to reproduce the problem, should it reoccur. If it does, you can try switching LAN interfaces on the server and see if the problem goes away. If it does, you should start looking at the hardware of the interface generating the UDP checksum failures.

Regards,

Dave

[Dave Olker]

Hi Merijn,

There are several advantages of NFS/TCP over UDP, even in a LAN environment. They are all spelled out in the "TCP vs. UDP" chapter of my book "Optimizing NFS Performance" available from Prentice Hall. Some of the benefits are also listed in my NFS Performance white paper located at:

http://docs.hp.com/hpux/onlinedocs/1435/NFSPerformanceTuninginHP-UX11.0and11iSystems.pdf

In my opinion, the biggest reason to switch from UDP to TCP (aside from bizarre checksum problems) is if the NFS clients in your environment are logging timeout or retransmissions. You can check the "nfsstat -c" stats to see if your clients are logging any of these. If there are a very small number of timeouts/retrans then UDP is probably a good choice to continue using. If you see more than 5% of the total number of NFS calls resulting in timeouts/retrans then TCP may give you better results.

HP-UX 10.20 cannot do NFS/TCP, either as a client or a server. The first HP-UX release to support NFS/TCP is 11.0, where NFS/TCP was added via patches. 11i included support for NFS/TCP out of the box.

I'm sure AIX 5.X can do NFS/TCP but I don't know about 4.X. Also, the Linux support for NFS/TCP is very spotty. Some variants do, some don't, so you'll have to investigate your specific Linux versions to see if they support TCP for NFS or not.

Please let me know if you, or anyone else, have any other NFS-related questions.

Regards,

Dave

[support_5]

Hi all,

Thanks for the information and help. It's been invaluable. I will do some more testing shortly to try and see if I can duplicate the problem etc. I will update this with my results. May take some time though.

I the meantime, any further information you can impart with regarding NFS, it would be much appreciated, as the information given so far has been very useful.

Thanks again,

- Andrew Gray

[Dave Olker]

Hi Andrew,

I've written a few white papers on NFS, some of which are posted on http://docs.hp.com. They are:

NFS Performance Tuning for HP-UX 11.0 and 11i Systems

http://docs.hp.com/hpux/onlinedocs/1435/NFSPerformanceTuninginHP-UX11.0and11iSystems.pdf

Forcibly Unmounting NFS Filesystems

http://docs.hp.com/hpux/onlinedocs/3929/ForciblyUnmountingNFSFilesystems.pdf


I gave two presentations on NFS at this year's HP World conference. They are:


Introduction to NFS Version 4

ftp://198.151.251.239/pub/conference/hpworld2004/proceedings/3067.pdf

ONC/NFS Changes Planned for HP-UX

ftp://198.151.251.239/pub/conference/hpworld2004/proceedings/3068.pdf


I've also written a book on NFS titled "Optimizing NFS Performance: Tuning and Troubleshooting NFS on HP-UX". It is published by Prentice Hall and is available from Amazon.com:

http://www.amazon.com/exec/obidos/tg/detail/-/0130428167/qid=1095044325/sr=8-1/ref=sr_8_xs_ap_i1_xgl14/103-8048853-3867063?v=glance&s=books&n=507846

Hope this helps,

Dave

[support_5]

Great, I've just read some of the white-papers you have written and they are very good.

I have one more question though. The sysadmins and I are considering changing all our NFS mounted filesystems to move away from being listed in /etc/fstab (usually as soft mounted filesystems), to being automounted. What do you think of that idea? Also, since we are mixing 11.0 and 11.i environments, do you think it would be best to use the old automount rather than the new autofs? Or use autofs where we can (ie on 11.i boxes) but use automount on the 11.0 boxes?

What do you think would be the best way to go here? Any concerns/gotchas? Recommendations?

Thanks heaps!!

- Andrew Gray

[Dave Olker]

Hi Andy,

The legacy automounter only supports:

o NFS Version 2 mounts
o UDP mounts

NFS Version 2 has the following limitations, among others:

o 8K read/write sizes
o 2GB file size maximum
o Unsafe Asynchronous Writing

Unless you're OK with these limitations on your 11.0 clients, my recommendation would be to use AutoFS on all of your 11.0 and 11i systems.

On your 11i systems I strongly urge you to download the ONC 2.3 (i.e. Enhanced) version of AutoFS from Software Depot:

http://www.software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=ENHAUTO

This version of AutoFS is vastly superior to the ONC 1.2 version of AutoFS, which is the only version available for 11.0.

For 11.0 systems, my recommendation is to use the ONC 1.2 AutoFS, but to make it as bullet-proof as possible before implementing it. The way to do that is two-fold:

1. Install the latest ONC/NFS megapatch for 11.0 systems, as this contains all of the known ONC 1.2 AutoFS bug fixes. The latest patch available today is: PHNE_30377. Be sure to install any dependent patches as well.

2. Download and install the DeviceIDs bundle from Software Depot:

http://www.software.hp.com/portal/swdepot/displayProductInfo.do?productNumber=DEVICEID

This bundle allows the 11.0 NFS client to store device ID information in the /etc/mnttab file, which is essential for proper AutoFS behavior. I won't bore you with the reasons why, unless you really want them. (The DeviceID bundle is included as part of the Enhanced AutoFS bundle for 11i, so you don't need to download both for 11i systems).

Hope this helps,

Dave