High connectionless oriented stats in nfsstat

[Nick Batch]

We have several rp3440 boxes mounting file systems via nfs from an HP 8420 server. The nfs mount options returned via nfsstat -m indicate that we are using tcp as the protocol. And yet nfsstat -c from any of our client machines shows high levels of calls in both connection oriented AND connectionless oriented sections. In addition the connectionless section shows many badxiods and timemouts whereas the connection oriented section looks entirely healthy and all the servers seem to be behaving properly. Can anyone give a clue as to why this might be??

nfsstat -m output is:
/apps/BMIS11i/oracle from nraerp7:/apps/BMIS11i/oracle (Addr 172.16.9.37)
Flags: vers=3,proto=tcp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/apps/BMIS11i/apps from nraerp7:/apps/BMIS11i/apps (Addr 172.16.9.37)
Flags: vers=3,proto=tcp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

Relevant section of nfsstat -c are:
Connection oriented:
calls badcalls badxids
9880658 317 10
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 308 0
interrupts
10
Connectionless oriented:
calls badcalls retrans
46487817 973571 55
badxids timeouts waits
18793 971268 0
newcreds badverfs timers
0 0 16
toobig nomem cantsend
0 0 0
bufulocks
0

[Steven E. Protter]

Shalom,

I believe NFS v4 which requires 11.23 or higher would solve this problem.

NFS connections are not being terminated in a normal fashion, which is pretty typical for NFS v3 and lower.

I'd make sure the network is stable and not worry too much unless there are user complaints.

SEP

[Dave Olker]

Hi Nick,

Before I get to your question, I must address something that Steven brings up time and time again in these threads, regardless of how many times I correct it.

> I believe NFS v4 which requires 11.23 or
> higher would solve this problem.

NFS v4 is NOT, I repeat, NOT available on 11.23. It shipped with 11.31 (11i v3). That is the first HP-UX release to support NFS v4, and for the foreseeable future - the only release of HP-UX that will support NFS v4.

Steven - please, please stop posting that NFS v4 is available on 11.23. It's not.

NFS v4 wouldn't necessarily address Nick's problem because it's likely something other than his current NFS/TCP mounts is causing the Connectionless counters to increment. The only thing NFS v4 has going for it as it relates to Nick's problem is that v4 doesn't support UDP. However, if all of the NFS v3 mounts are using TCP and this system is still logging Connectionless statistics then they're coming from somewhere else.

Also, before I get to Nick, I have to ask:

> NFS connections are not being terminated in
> a normal fashion, which is pretty typical
> for NFS v3 and lower.

What does this mean? How does NFS v3 and lower terminate connections in a non-normal fashion? And why would *any* termination of an NFS/TCP connection result in UDP statistics being incremented on the client?


Ok, now to Nick's original question. The appearance of UDP statistics in nfsstat could be the result of many things:

1) Perhaps this system was using UDP for NFS mounts in the past and you've only recently switched to TCP. If you haven't reset these statistics (via nfsstat -z) you could be seeing remnants of old UDP mount points that are no longer in place.

You could try resetting the statistics with "nfsstat -z" and see if they continue to increment even when there are no NFS/UDP mounts.


2) Are you running an older version of automounter on the system? The older automounter used UDP exclusively, so even if it never mounts anything - if it is configured and gets triggered for some reason it could cause some Connectionless statistics to get logged even during a failed mount attempt.


3) Are you using the HP CIFS Client product on your system? The CIFS Client leverages the NFS client protocol to work its magic, so if you're using the CIFS Client to access shares on Windows systems you might see the nfsstat counters increment based on that traffic.


As for the individual counters themselves (badxids, timeouts, etc.) until we know why the global Connectionless counters are being incremented in the first place I'm not going to worry about why these specific counters are incrementing.

The good news is your TCP statistics are clean and that's the protocol you're using for your current NFS mounts.

Let me know if your system continues to increment the Connectionless counters even after they are reset. If so, let me know what OS you're running, which automounter you're running (if any), and whether you're running the HP CIFS Client product to access Windows filesystems.

Regards,

Dave

[Nick Batch]

Thanks for the info gents. Dave, in reply to your questions:

1) I don't think we have ever had any non nfs v3 mounts on these boxes, so I don't think it's that. Also, I am running a soak test of the setup now, and I can see the connectionless stats and the connection oriented stats going up at about the same rate in terms of calls! However, on the nfs server box, its only logging connection oriented stats via nfsstat -sr as you would expect.

2) Discovered that we are running automount but oddly only on 3 out the 4 client boxes - and the one that isn't running automount is running up connectionless calls at a higher rate than the others. But the client box without automount seems to be configured as an nfs server for some reason - so it is running some nfsd's and no biod's. But again, it seems to be behaving normally from an application point of view. Also the auto_master file just has the default entry:
/net -hosts -nosuid,soft

3) Don't think we are using CIFS at all - certainly no windows shares in the config.

[Dave Olker]

Hi Nick,

1) What OS are these systems running? What is a "soak" test? If you still see UDP requests incrementing on the system at the moment, I'd be curious if any UDP NFS requests show up in a network trace collected on the client system.

Can you try collecting a network trace on the client logging the Connectionless requests and see if you find any UDP NFS requests showing up in the trace? If so, the trace might show the process ID of the process generating the NFS requests - even if they're being sent to the loopback interface.


2) Which version of automounter are you running? Old legacy automounter? ONC 1.2 AutoFS? ONC 2.3 AutoFS? Are you actually using automounter on these systems? If not, can you try disabling the automounter on the systems and see if the Connectionless counters stop incrementing?

Regards,

Dave

[Nick Batch]

We're running 11.11 and our application is oracle e-business suite. We are testing mounting the ebiz binaries via nfs shares instead of having local copies. The soak test is running a bunch of simulated user sessions (about 1000) via mercury load runner. So its not really "soaking" the network or nfs but its imposing a reasonable load on the whole system.

I'll need to get a friendly unix bloke with root pwd to help with the trace so will have a go in the morning...but it sounds like that is the best way to track it down.

The automounter isn't being used - not sure exactly which one it is. Can you tell from these details:
/usr/lib/netsvc/fs/automount/automount:
auto_main.c $Date: 2004/04/21 14:59:52 $Revision: r11.11/3 PATCH_11.11 (PHNE_30378)
auto_look.c $Date: 2002/04/16 10:51:48 $Revision: r11.11/2 PATCH_11.11 (PHNE_25627)
auto_node.c $Date: 2004/04/21 15:00:08 $Revision: r11.11/3 PATCH_11.11 (PHNE_30378)
auto_mount.c $Date: 2002/12/19 16:29:29 $Revision: r11.11/1 PATCH_11.11 (PHNE_28103)

Cheers

Nick

[Dave Olker]

Rather than continue to speculate on the origin of these NFS requests, I'll wait for the network trace to see if it captures any.

> The automounter isn't being used

Do you mean it's not running at all or it's running but you're not using it for mounting filesystems? Even if it's running and only has the typical

/net -hosts

map entry, it could still be generating UDP traffic if some application (an Oracle soaker, for example) were to traverse the path /net/.

If you're not using automounter for mounting anything but it's still running, I suggest you stop it and then change the "AUTOMOUNT" variable to 0 in the /etc/rc.config.d/nfsconf file to make sure it doesn't get restarted during the next system boot.

Regards,

Dave

[Nick Batch]

Ok, things maybe becoming a bit clearer. Trace file is attached and it shows lots of NLM traffic going over UDP. We are running apache on the nfs client boxes, and I guess the apache processes are trying to make use of file locking...? Sure enough when I stop apache the connectionless numbers stop rising.

The other problem we may have is that there is a firewall between clients and server and I think it only has ports 111 and 2049 open currently which doesn't sound like enough. Could that explain the large number of badcalls in the connectionless stats? Having said that, everything appears to be working normally, so apache isn't really complaining - which seems odd.

[Dave Olker]

Hi Nick,

If there is a firewall between these systems it appears to be letting NLM requests through because I see in the trace there are many NLM calls between:

Source: 172.16.9.27(B) Dest: 172.16.9.37(B)

And these requests are answered back with good replies from the remote system. So at least the file locking to server nraerp7 appears to be working.

Do you know what kind of system nraerp7 is? What OS is it running? The reason I ask is HP recently released patches for 11.11 and 11.23 that allows you to specify fixed port numbers for rpc.lockd, rpc.statd, and rpc.mountd so that these daemons start on the same port number each time. This makes configuring an NFS server behind a firewall much simpler and straight forward.

If you think the firewall is the problem and the NFS server is an HP-UX system running 11.11 or higher, you can fix the port numbers of the other RPC daemons and open those select port numbers in the firewall. That might help reduce the badcalls and badxids.

Regards,

Dave

[Nick Batch]

Dave, thanks again for the info. We are running hp-ux 11.11 on clients and servers and patched to dec-06 gold pack on both, so we could look to fix the port nums I guess.

Our firewall config only restricts connections into the server (if that makes sense) but does not restrict any comms that are initiated to run in the other direction. Maybe this explains with the lockd/statd traffic is not being restricted?

The other odd thing is that the nfs server isn't showing any connectionless traffic. Output from nfsstat -sr as follows:
Connection oriented:
calls badcalls nullrecv
18807022 0 0
badlen xdrcall dupchecks
0 0 402663
dupreqs
0

Connectionless oriented:
calls badcalls nullrecv
3 0 0
badlen xdrcall dupchecks
0 0 0
dupreqs
0
If the nfs server were receiving udp traffic from the clients, shouldn't its connectionless numbers be increasing?

I'm still not quite sure if we actually have a problem! Everything seems to be running fine, even under fairly heavy load. Just be nice to explain all the connectionless numbers...

Nick

[Dave Olker]

Hi Nick,

NLM traffic should not increment the nfsstat counters on the server, or the client for that matter. These counters should only be incrementing if there is actual *NFS* traffic, and NLM traffic is not the same. So the fact that there are file locks happening between this client and server would not show up in these counters.

Can you please paste an entire "nfsstat -c" output here so I can see exactly which client-side counters are being incremented? That might make it easier for me to go through the trace and find a matching request. Also, if you could send me the *raw* trace file, as opposed to the formatted one, it would make it much easier for me to analyze the trace.

If you are not comfortable posting it here, feel free to email it to me at dave.olker@hp.com.

Regards,

Dave