NFS, automount works, manual mount fails

[Darren Henderson]

Have an HP-UX 11.11 system (boon) trying to connect to a FreeBSD 6.0-STABLE NFS v3 server (monhegan).

The server is exporting 4 ro file systems.

If I allow automount to do the work, changing to /net/monhegan works beautifully. All four filesystems are present and readable.

However, if I try to mount any of the file systems manually, I always get:

root@boon[/] mount -r -o soft monhegan:/var /monhegan
nfs mount: getaddr_nfs: monhegan: NFS service not responding(retry delay=5s)
nfs mount: retry: retrying(1) for: /monhegan after 5 seconds
nfs mount: retry: giving up on: /monhegan

Doing some packet sniffing on the server shows that automount is using a different process to connect up.

Essentially we see a v2 NFS call, a v2 port mapper call, a mountd port 712 call, etc etc

With a manual mount we see a v2 pormat call foolwed by two v2 mount calls, three v2 portmapper calls.

Not going to go into any more details with those but it does tell me that automounting is doing something that I'm not.

/etc/auto_master contains only:
/net -hosts -nosuid,soft

If I do a mount using -o nosuid,soft I get the same results as above.

Other threads indicated that nsswitch.conf could have a role in this due to the defaults assumed for services. On boon it only has

name: files

Adding services: files to it made no difference.

Normally, automount would be "good enough" for our purposes but we've had some issues getting data protector to work with automounted volumes and since there isn't a native agent.... kind of need the manual mounts to work correctly.

Have tried this with several HP-UX 11.11 clients with the same results.

[Christine Hartman]

do you have your exports file setup? and if so did you remember to exportfs -a?

[Christine Hartman]

for nfs mounts...make sure nfs is running (ps -ef |grep nfs)....also you have to setup your /etc/exports file and then run exportfs -a.....the man page on exports should explain file syntax, etc.

[Chan 007]

Hi,

Your nsswitch.conf should also have other entries like

passwd: files
group: files
hosts: files
networks: files.

Try checking your hosts file, resolvf.conf.

Cheers...007

[Darren Henderson]

FreeBSD doesn't have an exportfs command. Its exportfs file contains:

/ -ro db2 boon
/usr -ro db2 boon
/var -ro db2 boon
/data -ro db2 boon


Those directories are exported. This is known to be correct because the automount from boon is able to mount these and see their contents just fine. I just can't seem to duplicate that feat using manual mounts from boon.

[Darren Henderson]

Adding groups, passwd, networks, and services to nsswitch.conf had no effect.

I would expect that the autmount would fail if the problem was in that area - though autofs may rely on differnt system components I suppose.

It almost looks like an NFS version issue but specifying the version doesn't seem to have an effect (ie mount calls with -o vers=1, -o vers=2, and -o vers=3 all behave the same).

[Dave Olker]

Hi Darren,

Which automounter are you using to successfully mount the filesystem? Is it the legacy automounter or AutoFS? If you're using AutoFS, is it the ONC 1.2 version that shipped with 11.11 or the Enhanced AutoFS that is available from the http://software.hp.com site?

Once you are able to successfully mount the filesystem with the automounter, please issue the command: "nfsstat -m" and post the results here so that I can see exactly which mount options were used to successfully mount the filesystem.

Thanks!

Dave

[Darren Henderson]

AUTOFS is set to 0 in /etc/rc.config.d/nfsconf

The process that is running is:

/usr/lib/netsvc/fs/automount/automount -f /etc/auto_master

This is the default automount.

nfsstat -m doesn't return anything.

nfsstat -c output is attached. Not sure that it reveals anything usefull other than it appears to be relying on v2.

[Dave Olker]

Darren,

I'm not sure why "nfsstat -m" would fail to return anything, especially if the filesystem is currently mounted. Are you certain the filesystem was actually mounted when you issued the "nfsstat -m" command?

As for a manual mount, have you tried forcing the mount to use NFS version 2 and the UDP protocol via the "-o vers=2, proto=udp" options? Since you're using the legacy automounter, it can only use UDP (not TCP) and it can only use NFS Version 2 (not 3). Since your clients are running 11.11, their default behavior is to try using TCP and NFS V3, which may not work with your specific server.

Please let me know if a manual UDP/PV2 mount succeeds.

Regards,

Dave

[Darren Henderson]

Dave,

Same results.

root@boon[/] mount -o vers=2,proto=udp monhegan:/var /monhegan
nfs mount: getaddr_nfs: monhegan: NFS service not responding(retry delay=5s)
nfs mount: retry: retrying(1) for: /monhegan after 5 seconds
nfs mount: retry: giving up on: /monhegan

A script capture is attached. Shows a mount, then an attempt to do the nfs mount, another mount, a cd /net/monhegan, another mount, an ls of the mounted files, a nfsstat -m and an nfsstat -c

I don't know why the nfsstat -m is failing to display anything - have tried it from several machines with the same results.

[Dave Olker]

Darren,

Does this happen with all NFS servers or only the FreeBSD system? You said you have multiple 11.11 systems available. If you make one of them an NFS server temporarily (just export /tmp or something) do the other 11.11 clients mount from that system properly and display the mounts correctly with nfsstat -m? I'm trying to understand if this is something broken on the client or specific to this one server.

Regards,

Dave

[Darren Henderson]

Dave,

I've taken another box and done an

exportfs -v -i /tmp

And then from boon mounted it with

mount prirpt2:/tmp /monhegan

nfsstat -m on boon then reports
/monhegan from prirpt2:/tmp (Addr 192.168.46.132)
Flags: vers=3,proto=tcp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

Interestingly, if I umount /monhegan on boon and then mount the share via automount (cd /net/prirpt2/tmp) nfsstat -m returns nothing.

So, apparently, nfsstat -m doesn't work with automounted file systems. Odd. But another sign that automount is doing things differently then direct mounts.

[Darren Henderson]

One more data point, perhaps not directly applicable.

The FreeBSD system is able to mount file systems from the HP-UX systems just fine.

[Dave Olker]

Darren,

The "nfsstat -m" not working with legacy automounter is something specific to your systems. It works just fine on my 11.11 systems here in Roseville, CA.

What your test showed is the client is able to manually mount from another 11.11 system, and it used NFS V3 and TCP. This is what I hoped to see. It would seem to indicate a problem with the FreeBSD server.

Do you know if the FreeBSD server supports mounting via TCP? I remember we had a problem early in the HP-UX 11.0 days when we first added support for TCP where mounting from an 11.0 server would result in the same error messages you're seeing with FreeBSD. The problem turned out to be the fact that the /etc/services file on the NFS server was not updated with a TCP entry for nfsd/2049. Actually, the problem was that the NFS server was using NIS for it's services database and the NIS services table hadn't been updated with a NFS/TCP entry.

In any case, do you know if your FreeBSD server is configured to service TCP mounts? If not, can it be?

My suspicion is that the manual mount command is requesting a TCP daemon on your FreeBSD server and that is why the mount fails - even when you manually request a UDP mount. To prove this I would need a network trace of the failing mount request. Do mentioned previously you had the ability to capture network packet traces. Can you please collect a trace of the failing manual mount and either post it here or email it to me privately?

Regards,

Dave

[Darren Henderson]

A trace is on its way Dave.

FreeBSD 6 defaults to NFSv3. This is clean install, not an upgrade from a previous version so I believe the config files are correct modulo any bugs in the distributuion. I'll send along the current sockstat as well.

[Dave Olker]

Darren,

The issue of NFS V3 support is not important at this point. I'm more concerned about the issue of NFS/TCP support. A server can support NFS V3 without supporting TCP.

One way to tell would be to log into the 11.11 client and issue the command:

# rpcinfo -p

Does the rpcinfo output show both a UDP and TCP version of nfsd registered for both versions 2 and 3?

Dave

[Darren Henderson]

Oddly enough(?) it doesn't show anything for nfsd.

root@boon[/] rpcinfo -p monhegan
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100000 4 7 111 rpcbind
100000 3 7 111 rpcbind
100000 2 7 111 rpcbind
100005 1 udp 1016 mountd
100005 3 udp 1016 mountd
100005 1 tcp 712 mountd
100005 3 tcp 712 mountd
root@boon[/]

From the bsd system's perspective its the same...

root@monhegan[/] rpcinfo -p localhost
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100000 4 local 111 rpcbind
100000 3 local 111 rpcbind
100000 2 local 111 rpcbind
100005 1 udp 1016 mountd
100005 3 udp 1016 mountd
100005 1 tcp 712 mountd
100005 3 tcp 712 mountd
root@monhegan[/]

[Dave Olker]

Ummmm, I'm not a FreeBSD person by any means, but that doesn't look good. I've never seen an NFS implementation that didn't register it's nfsd daemons with the portmapper/rpcbind during startup.

Can you confirm that the server is configured as an NFS server properly? Are the nfsd daemons or threads (again, don't know how NFS is implemented on FreeBSD) are running?

Dave

[Darren Henderson]

I believe it is - doesn't seem that the automount would work if it wasnt. I'll double check and I will lookinto the rpc advertisement of nfsd.

I'm currently in a position where both of these boxes can be rebooted so I'll do that as well to clear up any funky issues that may be causing an issue.

The components are running:
root@monhegan[/etc/defaults] ps ax | grep nfs | grep -v grep
103 ?? IL 0:00.00 [nfsiod 0]
104 ?? IL 0:00.00 [nfsiod 1]
105 ?? IL 0:00.00 [nfsiod 2]
106 ?? IL 0:00.00 [nfsiod 3]
480 ?? Is 0:00.02 nfsd: master (nfsd)
481 ?? I 0:00.12 nfsd: server (nfsd)
482 ?? I 0:00.00 nfsd: server (nfsd)
483 ?? I 0:00.00 nfsd: server (nfsd)
484 ?? I 0:00.00 nfsd: server (nfsd)
root@monhegan[/etc/defaults] ps ax | grep mountd | grep -v grep
478 ?? Is 0:00.07 /usr/sbin/mountd -r
root@monhegan[/etc/defaults] ps ax | grep rpcbind | grep -v grep
411 ?? Ss 0:00.37 /usr/sbin/rpcbind
root@monhegan[/etc/defaults]

[Dave Olker · Accepted solution]

Hi Darren,

I looked at the failing trace and it looks like the mount is failing because the nfsd daemons/threads on the FreeBSD server are not registering with the portmapper.

Here are the 4 interesting packets in the trace (heavily edited for content). They are packets 10, 11, 12, 13 in the "manual" trace if you want to look for yourself:

No. Time Source Destination Protocol Info
10 11.668974 192.168.46.122 192.168.46.89 Portmap V2 GETPORT Call (Reply In 11) NFS(100003) V:3 TCP

Portmap GETPORT Call NFS(100003) Version:3 TCP
Program Version: 2
V2 Procedure: GETPORT (3)
Program: NFS (100003)
Version: 3
Proto: TCP (6)
Port: 0

No. Time Source Destination Protocol Info
11 11.669049 192.168.46.89 192.168.46.122 Portmap V2 GETPORT Reply (Call In 10) PROGRAM_NOT_AVAILABLE

Portmap GETPORT Reply Port:0 PROGRAM_NOT_AVAILABLE
Program Version: 2
V2 Procedure: GETPORT (3)
Port: 0

No. Time Source Destination Protocol Info
12 11.670473 192.168.46.122 192.168.46.89 Portmap V2 GETPORT Call (Reply In 13) NFS(100003) V:3 UDP

Portmap GETPORT Call NFS(100003) Version:3 UDP
Program Version: 2
V2 Procedure: GETPORT (3)
Program: NFS (100003)
Version: 3
Proto: UDP (17)
Port: 0

No. Time Source Destination Protocol Info
13 11.670541 192.168.46.89 192.168.46.122 Portmap V2 GETPORT Reply (Call In 12) PROGRAM_NOT_AVAILABLE

Portmap GETPORT Reply Port:0 PROGRAM_NOT_AVAILABLE
Program Version: 2
V2 Procedure: GETPORT (3)
Port: 0



The trace shows the client asking the FreeBSD server for the port number of the TCP nfsd, being told there is nothing registered, tries asking for the UDP nfsd, and is told there is nothing registered.

The client eventually gives up since it cannot confirm what port number to contact the server's nfsd daemon/thread on. I think if you can determine why the nfsd daemons/threads on the FreeBSD are failing to register with the portmapper then the mount will likely succeed.

Regards,

Dave

[Darren Henderson]

Thanks Dave, I will look into that.

Any clue as to why the automount does succeed?

[Dave Olker]

> I believe it is - doesn't seem that the automount would work if it wasnt.

The difference is that the old automounter never even bothers to ask the server what port to contact the nfsd daemon on. It just assumes the port number will be 2049 and starts sending NFS requests to that port.

The manual NFS mount command doesn't make this assumption, which is good because some implementations of NFS allow you to change the port number of NFS from 2049 to a different port. The mount command queries the remote portmapper asking what port to send NFS requests to and is told that NFS is effectively not running, which is why the mount fails.

If you look at the manual mount trace you will find that the manual mount command was able to successfully talk to the server's rpc.mountd daemon and acquire the filehandle for the mountpoint in question. It only failed once it tried to find out which port number to send actual NFS requests to.

Bottom line, you need to figure out why the FreeBSD server is not registering the nfs program with the local rpcbind/portmapper. Once that registration is successful you'll probably have no problem mounting filesystems via the 11.11 clients.

Hope this helps,

Dave

[Darren Henderson]

After the reboot everything appears to be working as expected and rpcinfo is now showing nfs services - so you were correct Dave.

I wish I had an explanation as to why rpcinfo was not reporting correctly before the reboot - most likely operator error. I may have cuased rpcbind on the bsd server to restart without restarting nfsd which is a no-no.

Thanks to everyone who took the time to read the posts.

-Darren

[Kursat Karaca]

I think there is FW between nfsclient and server. Automount makes GETPORT Call and that invokes FW's inspection engine to open dynamic ports for a period of time. After that period it closes. Manual mount does not make a GERPORT call so if FW had closed that ports you can not make nfs connection.

Cheers.