NFS Issue

[shasha_1]

We have 2 HP-UX 11.23 servers and have exported 2 filesystems and nfs-mounted them on the other machine. Now, the users are complaining about the slow performance even when they're just doing ls or moving from one directory to another.

I looked at the /var/adm/syslog/syslog.log file on the nfs client and this is what I saw:

Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1643]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1651]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1652]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1655]: nfsd 4 0 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1660]: nfsd 4 2 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1652]: nfsd 2 3 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1662]: nfsd 2 2 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1661]: nfsd 2 1 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1659]: nfsd 1 1 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1665]: nfsd 1 2 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1651]: nfsd 1 3 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1643]: nfsd 4 3 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1656]: nfsd 1 0 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1653]: nfsd do_one bind 3
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1658]: nfsd 4 1 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1657]: nfsd 2 0 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1653]: Return from t_optmgmt(XTI_DISTRIBUTE) 0
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1678]: nfsd 3 0 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1679]: nfsd 3 1 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1653]: nfsd 3 3 sock 4
Jul 20 00:34:16 wbscsp01 /usr/sbin/nfsd[1680]: nfsd 3 2 sock 4
Jul 26 23:48:59 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 26 23:48:59 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 26 23:54:41 wbscsp01 vmunix: NFS server WBSCSP02 ok
Jul 26 23:54:41 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 26 23:54:41 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 26 23:54:41 wbscsp01 vmunix: NFS server WBSCSP02 ok
Jul 27 00:16:48 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 27 00:16:52 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 27 00:22:22 wbscsp01 vmunix: NFS server WBSCSP02 ok
Jul 27 00:22:22 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 27 00:22:22 wbscsp01 vmunix: NFS server WBSCSP02 not responding still trying
Jul 27 00:22:22 wbscsp01 vmunix: NFS server WBSCSP02 ok


I then ran nfsstat -cr and the output is this:

Client rpc:
Connection oriented:
calls 4242403
badcalls 454 badxids
83
timeouts 0 newcreds 0
badverfs 0
timers 0 cantconn 368 nomem 0 interrupts
83

Connectionless oriented:
calls 5925 badcalls 0 retrans 0
badxids 5 timeouts 0 waits 0
newcreds 0 badverfs 0 timers 0
toobig 0 nomem 0 cantsend 0
bufulocks 0


and output from nfsstat -m:

/bscs from WBSCSP02:/bscs (Addr 10.10.5.10)
Flags: vers=3,proto=tcp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All:srtt=0(0ms),dev=0(0ms),cur=0(0ms)

/bscs/WORK from WBSCSP02:/bscs/WORK (Addr 10.10.5.10)
Flags: vers=3,proto=tcp,auth=unix,hard,intr,link,
symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt=0 (0ms), dev=0 (0ms), cur=0(0ms)


Is there something wrong with NFS based on these outputs?

[Pete Randall]

What are your NIC settings? Something like this could result from a speed/duplex mismatch between the NIC and the switch. Are you running 100Mb? Are both the card and the switch forced to full duplex?

I'm attaching the procedure (borrowed extensively from Bill Hassell) to check and set your NIC.


Pete

[shasha_1]

Both the switch and the NIC on the server has been set to 100Half-Duplex, is there any issue with that?

[Kent Ostby]

I believe this is caused by running additional NFSD clients without stopping the ones that were already running.

This is unsupported and causes the problem above.

The "fix" as it were would be to do:

/sbin/init.d/nfs.server stop

followed by:

/sbin/init.d/nfs.server start

[Pete Randall]

Kent has most likely nailed the issue but I would like to see both your switch port and your NIC set 100FD if possible. At half duplex, it has to wait half the time: i.e. if one end is talking the other has to wait before it can talk. Full duplex should provide a significant improvement in overall network throughput.


Pete

[Rick Garland]

Bounce the NFS services on the NFS server.

Apply the latest NFS patches. These are included in the Jul 05 patch bundle

[Rick Garland]

Also, can you change the duplex to 100 Full Duplex?