topic Re: NFS error: RPC: Server can't decode arguments in Operating System - HP-UX

NFS error: RPC: Server can't decode arguments

J Andersen — Thu, 31 May 2007 07:50:11 GMT

I am experiencing some problems with NFS on a newly installed HP-UX server (rx2620, HP-UX 11.31). The server mounts 4 filesystems via NFS, 2 from a Linux server, 2 from a NAS. The NAS mounts seem OK (Noone has complained yet), but the Linux ones have been giving constant problems - at first anything that accessed those disks would simply hang after a while, but then I noticed that the NAS disks use UDP and the Linux ones use TCP, so I changed the Linux disks to mount via UDP. Now people can use the disks, but they see the following message:

NFS write failed for server bldsrv: error 11 (RPC: Server can't decode arguments)

This is when they log in or out (the home directories are on one of the Linux disks). What is this about? And, of course, how do I fix it?

Re: NFS error: RPC: Server can't decode arguments

Steven E. Protter — Thu, 31 May 2007 11:36:47 GMT

Shalom,

Checklist:

Parameters and error messages in /var/log/messages on the Linux hosts.

There may be problems there that will contribute to the solution or may be found to be the cause of the problem.

Check the ITRC patch database for patches, there may be something wrong with the 11.31 NFS client. Its pretty new software.

SEP

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Fri, 01 Jun 2007 02:31:19 GMT

These are the messages in /var/adm/syslog/syslog.log:

Jun 1 13:19:15 elk vmunix: WARNING: VxVM vxio V-5-0-23 Open on an spurious volume device (0x3000005) encountered. This device may be valid, but has not yet been configured in the kernel.

Jun 1 13:19:25 elk vmunix: NFS write failed for server bldsrv: error 11 (RPC: Server can't decode arguments)
Jun 1 13:19:25 elk vmunix: NFS write error on host bldsrv: I/O error.
Jun 1 13:19:25 elk vmunix: (file handle: 1000002 3a0002 2000000 33820700 a4e44b5f a800700)

Jun 1 13:19:43 elk vmunix: tunopen(): CLONEOPEN not allowed

The first one, about the 'spurious volume', and the last one, about 'tunopen()' only occur occasionally, the three about NFS repeat all the time.

I only found 3 patches that refer to NFS and 11.31: PHSS_36077, PHSS_36544 and PHSS_36545, none of which seem relevant.

Re: NFS error: RPC: Server can't decode arguments

David Piller — Fri, 01 Jun 2007 02:57:46 GMT

Hi,

just experienced the same problem, and received from support - may help, may not, I am not the responsible for this area:

Possible workaround is to mount the NFS filesystem in Version 2 an the 11.31 client:

$mount -F nfs -o vers=2,rsize=4096,wsize=4096 nfs-server:/share /local

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Fri, 01 Jun 2007 04:33:58 GMT

Hi, Steven and David,

I appreciate your help - I tried to assign points, but it doesn't seem to work, I'm not sure why. But thanks.

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Fri, 01 Jun 2007 04:52:10 GMT

Hi David,

I tried out your suggestion, but that just gives me:

NFS getattr failed for server bldsrv: error 9 (RPC: Program/version mismatch)
nfs mount: /home1: I/O error

The strange thing is, the server should support vers 2; this is what rpcinfo says:

# rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 1024 status
100024 1 tcp 32768 status
100011 1 udp 966 rquotad
100011 2 udp 966 rquotad
100011 1 tcp 969 rquotad
100011 2 tcp 969 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100021 1 udp 2549 nlockmgr
100021 3 udp 2549 nlockmgr
100021 4 udp 2549 nlockmgr
100021 1 tcp 40496 nlockmgr
100021 3 tcp 40496 nlockmgr
100021 4 tcp 40496 nlockmgr
100005 1 udp 1008 mountd
100005 1 tcp 1011 mountd
100005 2 udp 1008 mountd
100005 2 tcp 1011 mountd
100005 3 udp 1008 mountd
100005 3 tcp 1011 mountd

Re: NFS error: RPC: Server can't decode arguments

Dave Olker — Fri, 01 Jun 2007 10:32:38 GMT

Hi Jan,

I think the reason you're not able to assign points is because you submitted your question as a user "J Andersen" and you have since been using a login of "Jan Andersen". I think you might have two different user profiles and since they don't match, the database doesn't consider you the author of this thread.

As for your NFS problem, I'd like to see a network trace of the failing requests. Please use wireshark or ethereal to collect a trace on the 11.31 client when it mounts the filesystem and generates the traffic that results in the "can't decode" error so that I can see exactly what is going to the server.

David - I really don't like the answer HP support gave about using NFS v2 to solve any problem unless it's working around a specific hardware problem in your environment. NFS v3 and v4 should work in any environment that v2 does, and if you have a problem that can only be fixed by v2 then I want to know more about it because either it's a defect in 11.31 or it's a bogus solution.

Regards,

Dave

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Tue, 05 Jun 2007 05:54:01 GMT

Hi, sorry, I won't be able to do anythin about this until torrow, I think - just in case you were waiting for my response.

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Wed, 13 Jun 2007 08:36:14 GMT

OK, I'm back - does anybody know where I can find a wireshark for 11.31? I can only find it for 11.23

Re: NFS error: RPC: Server can't decode arguments

Dave Olker — Wed, 13 Jun 2007 10:34:57 GMT

Hi Jan,

The 11.23 version of Wireshark will work just fine on 11.31 systems. I'm running it on all of my 11.31 boxes.

Dave

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Thu, 14 Jun 2007 08:45:14 GMT

How do I install the thing? I downloaded a file called ixWireshark_A.08.00-0.99.5.001_HP-UX_B.11.23_IA+PA.depot, but swinstall says 'Not compatible'.

Re: NFS error: RPC: Server can't decode arguments

Dave Olker — Thu, 14 Jun 2007 10:43:49 GMT

You need to go to the Options->Change Options screen in swinstall and select the "Allow installation of incompatible software" and try again.

Dave

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Fri, 15 Jun 2007 03:12:22 GMT

OK, I got past that one - but now it get another one that makes no sense to me:

Could not apply the software selection "ixWireshark,r=A.08.00-0.99.5.001,a=HP-UX_B.11.23_IA/PA,v=HP" because it refers to a bundle or to a product, subproduct or fileset within a bundle and attempts to select the item or items all failed. The messages below show the items which could not be selected and the errors generated:

Could not apply the software selection "Wireshark,r=A.08.00-0.99.5.001,a=HP-UX_B.11.23_IA/PA,v=HP" because a different variant of this same product has already been selected.

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Mon, 18 Jun 2007 07:15:36 GMT

Hi Dave

I have run the Wireshark trace, but I can see there is a limit of 1MB on uploading files. Did you want to see the trace?

Re: NFS error: RPC: Server can't decode arguments

Dave Olker — Mon, 18 Jun 2007 09:22:46 GMT

Hi Jan,

Sure, you can email the trace to me at dave.olker@hp.com.

Please also provide the IP address of the remote NFS system. In other words, if you collected the trace on the client then provide the IP address of the server, and vice versa.

Regards,

Dave

Re: NFS error: RPC: Server can't decode arguments

Dave Olker — Mon, 18 Jun 2007 10:35:34 GMT

Hi Jan,

Thanks for sending the trace. Can you please tell me about the Linux server? What OS is it running? Which version of the kernel? Which version of the nfs-utils package? I'd like to try reproducing this problem on my systems so I need to know as much as possible about the Linux server.

I wonder if you could try a couple experiments to see if we can find something that works:'

1) Try using an rsize/wsize value of 1K just to see if small transfers make it across the wire OK.

2) Try using the rsize/wsize of 1K and TCP to see if that allows TCP to start working.

Let me know how these tests go.

Regards,

Dave

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Mon, 18 Jun 2007 11:02:24 GMT

Hi Dave,

That was quick! The Linux server is a Redhat AS 3, the kernel is 2.4.21-4.ELsmp. Nfs-utils is 1.0.5-3.

I will try the experiments tomorrow and let you know.

Re: NFS error: RPC: Server can't decode arguments

Dave Olker — Mon, 18 Jun 2007 11:17:52 GMT

Also, can you tell me the exact syntax you use to export the filesystem on the Linux server? The actual contents of the /etc/exports file would be great.

Re: NFS error: RPC: Server can't decode arguments

Dave Olker — Mon, 18 Jun 2007 11:32:32 GMT

Hi Jan,

Nevermind - I just reproduced the problem on my systems. Let me bang away at this for a bit and see if I can figure it out.

It seems to have something to do with negotiating read and write sizes between the two systems. When I use this syntax:

# mount -o proto=udp :/

I get the error. When I use this synatx:

# mount -o proto=udp,rsize=32768,wsize=32768 :/

I don't see the error. However, when I issue the "nfsstat -m" command after mounting with what I thought was 32K read/write size the client reports that the server returned 8K as the negotiated read/write size.

So, something is fishy with the read/write size negotiations. To work around the problem you can use the "rsize/wsize" mount options and set them to 8K. I'll see if I can get to the bottom of this.

By the way, I'm traveling on business this week (in Las Vegas presenting at the HP Technology Forum) so it might take me a few days to work on this. In the mean time, the workaround of hard-coding the rsize/wsize should get you working.

Regards,

Dave

Re: NFS error: RPC: Server can't decode arguments

J Andersen — Tue, 19 Jun 2007 03:01:15 GMT

Hi Dave,

It seems to have made a difference, though I have only just tried the changes. It may have solved our problem, which is great.

I would have ginven you 10 out of 10 for this one, but unfortunately I can't seem to find the ID that I created this thread from; perhaps you can add them manually ;-)