I/O error on NFS mount

[mohnkern]

We have 3 HP-UX boxes, one is 10.20 and the other two are 11.i boxes.

We're trying to do nfs mounts on all the boxes, and two of them we
don't seem to be having any issue with, however with one of them when
we attempted to do the mount, we get an error:

Nov 28 07:06:55 ohio vmunix: NFS getattr failed for server nas1: RPC:
(unknown e
rror code)

The entry in fstab looks like this:

: /home nfs rw,soft,suid 0 0

When I attempted mount /home, I get the following error:

$mount /home
NFS getattr failed for server nas1: RPC: (unknown error code)
nfs mount: mount_nfs: mount: /home: I/O error

This exact same entry works fine on the 11.i and 10.20 box.

export file on the server appears to be configured correctly.

Any help would be greatly appreciated.

[Sandman!]

Maybe the patch levels for NFS are different on the two 11i boxes. Compare the patches on the 11i and 10.20 boxes with the bad one. IMHO you may not have a NFS patch found on the other two.

~cheers

[mohnkern]

Is there a way to check the NFS patch levels on the two boxes rather than a complete list?

(Thanks for your help)

[Tim Nelson · Accepted solution]

swlist -l product|grep -i nfs

might be the initial place to start

[Steven E. Protter]

Shalom,

You should have two things done:

1) Regular quarterly patch set on all systems.
2) Build an NFS patch set from itrc.hp.com patch database and get it installed.

SEP

[mohnkern]

Patch levels match on the 11.i boxes:

$ swlist -l product|grep -i nfs
NFS B.11.11 ONC/NFS; Network-File System,Information Services,Utilities
PHKL_25238 1.0 11.00 NFS nfsd deadlock
PHKL_25993 1.0 thread nostop for NFS, rlimit, Ufalloc fix
PHKL_28185 1.0 Tunable;vxportal;vx_maxlink;DMAPI NFS hang
PHKL_29335 1.0 vx_nospace on NFS write.
PHKL_34595 1.0 VM NFS umount fix
PHNE_36168 1.0 ONC/NFS General Release/Performance Patch


[ohio_root]$ swlist -l product|grep -i nfs
NFS B.11.11 ONC/NFS; Network-File System,Information Services,Utilities
PHKL_25238 1.0 11.00 NFS nfsd deadlock
PHKL_25993 1.0 thread nostop for NFS, rlimit, Ufalloc fix
PHKL_28185 1.0 Tunable;vxportal;vx_maxlink;DMAPI NFS hang
PHKL_29335 1.0 vx_nospace on NFS write.
PHKL_34595 1.0 VM NFS umount fix
PHNE_36168 1.0 ONC/NFS General Release/Performance Patch


Here's the 10.20 box:

NFS B.10.20 ONC/NFS; Network-File System,Information Services,Utilities
PHCO_15339 B.10.00.00.AA fsclean(1M) support for NFSv3
PHCO_15340 B.10.00.00.AA /sbin/is_local_root support for NFSv3
PHCO_15341 B.10.00.00.AA mountall(1M) support for NFSv3 and CacheFS
PHCO_15343 B.10.00.00.AA umountall(1M) support for NFSv3
PHKL_17573 B.10.00.00.AA ki/nfs vasusage, ACE 2 bundle cumulative patch
PHKL_20126 B.10.00.00.AA NFS diskless client kernel patch (ACE 199912
PHNE_28886 B.10.00.00.AA NFS/NIS General Release/Performance Patch

[Dave Olker]

Hi,

The first thing I'd want to see is a network trace collected during the working mount and the failing mount. Collect a trace on the 11i box during the successful mount and a second trace on the 11i box during the failing mount. Hopefully the traces will give some indication of why the mount is failing.

It sounds like the NFS server in this case is some kind of NAS system. What type of system is it and what OS does it run?

Regards,

Dave

[mohnkern]

I ran nettl -tn 0x30800000 -e ns_ls_ip -size 1024 -tracemax 99999 -f /tmp/raw0 on the box while attempting to do a mount.

The contents of the trace file /tmp/raw0.TRC000 are attached.

the SAN is a NetAPP running their own OS version 7.0X10

[Dave Olker]

Ok, so I have the failing trace. I still need to see a trace of the working mount from the other 11i system to compare against this one.

This trace shows we sent a GETATTR to the NetApps box and in reply the filer sent back a FIN - asking to close the TCP connection. I see nothing obviously wrong in the GETATTR call we sent that would cause the server to forcibly close the connection, so I'd like the working trace to use as comparison to see if there is any difference between the GETATTR from the working 11i system and the failing 11i system that would explain why the NetApp box treats them differently.

Regards,

Dave

[mohnkern]

Thanks for all the help. The other box is a production box, let me see what I can do to get one for you.

[mohnkern]

While I'm waiting to see how we can do a mount and umount on the other 11.i machine I had a 10.20 machine that I knew we could do it on, so I did it there, and ran a trace.


Unfortunately it create two files, which even when gzipped are more than 1 MB in size. Is there a way I can extract the data I need to post?

[Dave Olker]

The only way to filter stuff out would be to format the traces and apply a filter. I'd prefer to see the raw traces as it lets me decide how to filter them to pull out what I want to see.

If you cannot attach them to the ITRC thread then just send the trace files to me directly (dave.olker@hp.com) and I'll have a look.

Dave

[mohnkern]

I emailed the files to you, if you don't get them, let me know.

[mohnkern]

I found an 11.i machine where we could duplicate the mount, so I ran it, emailed you the trace results.

(again too large to post)

[Dave Olker]

Hi Scott,

I looked at both traces and the GETATTR calls are pretty much identical (not completely identical because the names of the systems are different, etc.) so I don't think it's anything inherent to the NFS call itself causing the problem.

One thing I did notice different is the working system was using a TCP connection that was already established. The failing system is trying to establish a new TCP connection to the NetApp server for this mount and it's that connection that fails.

One way to determine if this is a TCP problem would be to see if you can mount the desired filesystem on the 11i client using UDP (via the -o proto=udp option). Let me know if the client can successfully mount with UDP and we'll go from there.

Regards,

Dave

[mohnkern]

I attempted:

mount -o proto=udp

from both systems (the one that was working, and the one that wasn't)

The results --

On the machine that was working, mount worked via udp as well.

From the non working machine --

[ohio_root]$mount -o proto=udp /home
NFS server nas1 not responding still trying
NFS server nas1 not responding still trying

Letting it continue to run, just to see what I get, but I expect it'll just keep saying the same thing.

[Dave Olker]

Has this specific NFS client ever been able to successfully mount this specific filesystem from this specific server or is this a new attempt to mount this filesystem by this client? Are you absolutely certain the NetApp box is configured to allow access from this NFS client?

Dave

[mohnkern]

This HP-UX box hasn't ever been configured to do mounts from this SAN.

We did a solid search, and couldn't find anywhere where the configuration entries for this non-working box are different from the two working ones.

[mohnkern]

We pulled up the log file from the nas unit and found this:

he messages file on nas1 contains this message:

Thu Nov 29 16:31:27 GMT [mnt_assist:warning]: Flushing "root" cache entry for
140.90.91.138 on path /vol/awips_home (xid 0)

140.90.91.138 is the machine that's not able to mount.

[Dave Olker]

I did a quick search of the NetApp knowledgebase for the "Flushing cache entry" message and I got this:

_________________________________


Title: Hosts are denied nfs mounts if the reverse DNS name lookup fails.

Description: If a host entry does not resolve in a reverse DNS, it is neither granted access nor added to the access cache.

Workaround: Add entries in /etc/hosts, or in DNS, which provide reverse name mappings.

Notes: A tell tale sign of this bug is messages of the form:
Thu Jul 15 11:22:26 PDT [mnt_assist:warning]: Client 10.41.70.239 (xid 0) fails to resolve via gethostbyaddr_r()
Thu Jul 15 11:22:26 PDT [mnt_assist:warning]: Flushing cache entry for 10.41.70.239 on path /vol/archvol2/archrlse4 (xid 0)

_________________________________


Are you *positive* this NetApp box is able to successfully resolve both hostname and IP address for the new 11i system? The fact that this has never worked leads me to believe it's something on the NetApp side causing the problem.

Regards,

Dave

[mohnkern]

Wow, congratulations on pointing us the right direction.

Indeed, tne NAS unit couldn't look up the host name of the non mounting machine.

However, it wasn't because there were entries in /etc/hosts for the two working machines (we checked for that).

Instead, it turned out that the two working machines were in NIS and the one non working machine wasn't.

We're seeing one other mount problem (on an NFS mount), but I'll work on it for awhile, and see if we can fix it before coming back.

Again, many thanks.

[Dave Olker]

Happy to help. Let me know if you need help with any other NFS mounting issues.

Regards,

Dave