NFS error: RPC: Server can't decode arguments

[J Andersen]

I am experiencing some problems with NFS on a newly installed HP-UX server (rx2620, HP-UX 11.31). The server mounts 4 filesystems via NFS, 2 from a Linux server, 2 from a NAS. The NAS mounts seem OK (Noone has complained yet), but the Linux ones have been giving constant problems - at first anything that accessed those disks would simply hang after a while, but then I noticed that the NAS disks use UDP and the Linux ones use TCP, so I changed the Linux disks to mount via UDP. Now people can use the disks, but they see the following message:

NFS write failed for server bldsrv: error 11 (RPC: Server can't decode arguments)

This is when they log in or out (the home directories are on one of the Linux disks). What is this about? And, of course, how do I fix it?

[Steven E. Protter]

Shalom,

Checklist:

Parameters and error messages in /var/log/messages on the Linux hosts.

There may be problems there that will contribute to the solution or may be found to be the cause of the problem.

Check the ITRC patch database for patches, there may be something wrong with the 11.31 NFS client. Its pretty new software.

SEP

[J Andersen]

These are the messages in /var/adm/syslog/syslog.log:

Jun 1 13:19:15 elk vmunix: WARNING: VxVM vxio V-5-0-23 Open on an spurious volume device (0x3000005) encountered. This device may be valid, but has not yet been configured in the kernel.

Jun 1 13:19:25 elk vmunix: NFS write failed for server bldsrv: error 11 (RPC: Server can't decode arguments)
Jun 1 13:19:25 elk vmunix: NFS write error on host bldsrv: I/O error.
Jun 1 13:19:25 elk vmunix: (file handle: 1000002 3a0002 2000000 33820700 a4e44b5f a800700)

Jun 1 13:19:43 elk vmunix: tunopen(): CLONEOPEN not allowed

The first one, about the 'spurious volume', and the last one, about 'tunopen()' only occur occasionally, the three about NFS repeat all the time.

I only found 3 patches that refer to NFS and 11.31: PHSS_36077, PHSS_36544 and PHSS_36545, none of which seem relevant.

[David Piller]

Hi,

just experienced the same problem, and received from support - may help, may not, I am not the responsible for this area:

Possible workaround is to mount the NFS filesystem in Version 2 an the 11.31 client:

$mount -F nfs -o vers=2,rsize=4096,wsize=4096 nfs-server:/share /local

[J Andersen]

Hi, Steven and David,

I appreciate your help - I tried to assign points, but it doesn't seem to work, I'm not sure why. But thanks.

[J Andersen]

Hi David,

I tried out your suggestion, but that just gives me:

NFS getattr failed for server bldsrv: error 9 (RPC: Program/version mismatch)
nfs mount: /home1: I/O error

The strange thing is, the server should support vers 2; this is what rpcinfo says:

# rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 1024 status
100024 1 tcp 32768 status
100011 1 udp 966 rquotad
100011 2 udp 966 rquotad
100011 1 tcp 969 rquotad
100011 2 tcp 969 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100021 1 udp 2549 nlockmgr
100021 3 udp 2549 nlockmgr
100021 4 udp 2549 nlockmgr
100021 1 tcp 40496 nlockmgr
100021 3 tcp 40496 nlockmgr
100021 4 tcp 40496 nlockmgr
100005 1 udp 1008 mountd
100005 1 tcp 1011 mountd
100005 2 udp 1008 mountd
100005 2 tcp 1011 mountd
100005 3 udp 1008 mountd
100005 3 tcp 1011 mountd

[Dave Olker]

Hi Jan,

I think the reason you're not able to assign points is because you submitted your question as a user "J Andersen" and you have since been using a login of "Jan Andersen". I think you might have two different user profiles and since they don't match, the database doesn't consider you the author of this thread.

As for your NFS problem, I'd like to see a network trace of the failing requests. Please use wireshark or ethereal to collect a trace on the 11.31 client when it mounts the filesystem and generates the traffic that results in the "can't decode" error so that I can see exactly what is going to the server.

David - I really don't like the answer HP support gave about using NFS v2 to solve any problem unless it's working around a specific hardware problem in your environment. NFS v3 and v4 should work in any environment that v2 does, and if you have a problem that can only be fixed by v2 then I want to know more about it because either it's a defect in 11.31 or it's a bogus solution.

Regards,

Dave

[J Andersen]

Hi, sorry, I won't be able to do anythin about this until torrow, I think - just in case you were waiting for my response.

[J Andersen]

OK, I'm back - does anybody know where I can find a wireshark for 11.31? I can only find it for 11.23

[Dave Olker]

Hi Jan,

The 11.23 version of Wireshark will work just fine on 11.31 systems. I'm running it on all of my 11.31 boxes.

Dave

[J Andersen]

How do I install the thing? I downloaded a file called ixWireshark_A.08.00-0.99.5.001_HP-UX_B.11.23_IA+PA.depot, but swinstall says 'Not compatible'.

[Dave Olker]

You need to go to the Options->Change Options screen in swinstall and select the "Allow installation of incompatible software" and try again.

Dave

[J Andersen]

OK, I got past that one - but now it get another one that makes no sense to me:

Could not apply the software selection "ixWireshark,r=A.08.00-0.99.5.001,a=HP-UX_B.11.23_IA/PA,v=HP" because it refers to a bundle or to a product, subproduct or fileset within a bundle and attempts to select the item or items all failed. The messages below show the items which could not be selected and the errors generated:

Could not apply the software selection "Wireshark,r=A.08.00-0.99.5.001,a=HP-UX_B.11.23_IA/PA,v=HP" because a different variant of this same product has already been selected.

[J Andersen]

Hi Dave

I have run the Wireshark trace, but I can see there is a limit of 1MB on uploading files. Did you want to see the trace?

[Dave Olker]

Hi Jan,

Sure, you can email the trace to me at dave.olker@hp.com.

Please also provide the IP address of the remote NFS system. In other words, if you collected the trace on the client then provide the IP address of the server, and vice versa.

Regards,

Dave

[Dave Olker]

Hi Jan,

Thanks for sending the trace. Can you please tell me about the Linux server? What OS is it running? Which version of the kernel? Which version of the nfs-utils package? I'd like to try reproducing this problem on my systems so I need to know as much as possible about the Linux server.

I wonder if you could try a couple experiments to see if we can find something that works:'

1) Try using an rsize/wsize value of 1K just to see if small transfers make it across the wire OK.

2) Try using the rsize/wsize of 1K and TCP to see if that allows TCP to start working.

Let me know how these tests go.

Regards,

Dave

[J Andersen]

Hi Dave,

That was quick! The Linux server is a Redhat AS 3, the kernel is 2.4.21-4.ELsmp. Nfs-utils is 1.0.5-3.

I will try the experiments tomorrow and let you know.

[Dave Olker]

Also, can you tell me the exact syntax you use to export the filesystem on the Linux server? The actual contents of the /etc/exports file would be great.

[Dave Olker]

Hi Jan,

Nevermind - I just reproduced the problem on my systems. Let me bang away at this for a bit and see if I can figure it out.

It seems to have something to do with negotiating read and write sizes between the two systems. When I use this syntax:

# mount -o proto=udp :/

I get the error. When I use this synatx:

# mount -o proto=udp,rsize=32768,wsize=32768 :/

I don't see the error. However, when I issue the "nfsstat -m" command after mounting with what I thought was 32K read/write size the client reports that the server returned 8K as the negotiated read/write size.

So, something is fishy with the read/write size negotiations. To work around the problem you can use the "rsize/wsize" mount options and set them to 8K. I'll see if I can get to the bottom of this.

By the way, I'm traveling on business this week (in Las Vegas presenting at the HP Technology Forum) so it might take me a few days to work on this. In the mean time, the workaround of hard-coding the rsize/wsize should get you working.

Regards,

Dave

[J Andersen]

Hi Dave,

It seems to have made a difference, though I have only just tried the changes. It may have solved our problem, which is great.

I would have ginven you 10 out of 10 for this one, but unfortunately I can't seem to find the ID that I created this thread from; perhaps you can add them manually ;-)

[Dave Olker]

Hi Jan,

> It seems to have made a difference, though I
> have only just tried the changes. It may have
> solved our problem, which is great.

As it turns out, this was a problem another customer already reported and we've coded a solution for. The formal fix will be included in the initial 11.31 NFS patch, which is scheduled for September.

The workaround listed in the bug report was exactly as I gave you - use the rsize and wsize mount options to request an 8K read and write size for these mounts. Hopefully this is a workaround you can use in your environment until the fix is officially released in September.

By the way, I'd recommend you go back to using TCP for your mounts instead of UDP. TCP has shown to be much more reliable on both HP-UX and Linux platforms for many years so I always recommend customers use TCP whenever possible. You'll still need to use the rsize/wsize mount options with TCP because the request size negotiation is still an issue with TCP mounts, just as it is with UDP, and will be addressed in the September patch.

> I would have ginven you 10 out of 10 for
> this one, but unfortunately I can't seem to
> find the ID that I created this thread
> from

That's OK. It's the thought that counts. :)

[Dennis Handly]

>I would have ginven you 10 out of 10 ...

Actually you could have given Dave 10 points for each answer.

>but unfortunately I can't seem to find the ID that I created this thread from;

I'm not sure if the moderators can help you recover? Try adding a thread to:
Your questions regarding ITRC Forums
http://forums1.itrc.hp.com/service/forums/categoryhome.do?categoryId=296

[J Andersen]

I´ll look forward to the patchset in September, then. Now I´ve got another problem on another machine - same setup, different (and harder) problem. I´ll post it in another thread.

Thanks again for all your help!