NFS Issue

[Cliff Lim Kok Hwee]

Gd Afternoon Forum,

I am currently having a issue with NFS.

The NFS server is named nfsserver and nfs client is nfsclient for simple reference.

I do not have access to nfsserver but nfsclient I do. I performed a ping of nfsserver and successful.

Currently I am having the following error reported in the dmesg:

NFS server (pid959@/net) not responding still trying
NFS server (pid959@/net) ok
NFS server (pid959@/net) not responding still trying
NFS server (pid959@/net) ok
NFS lookup failed for server nfsserver: RPC: Unable to receive
NFS lookup failed for server nfsserver: RPC: Unable to receive

I am trying to access directory /net on nfsclient which is linked directly to nfsserver.

Please advise me.

thanks/cliff

[Ralph Grothe]

Probably you have a somewhat congested that network between NFS server and client.
Have a look if there are many retransmissions taking place.
Look at the output of nfsstat.

e.g.

nfsstat -cr

also look at the soft round trip time of your NFS mounts

nfsstat -m

[Ralph Grothe]

Ah, these commands will probably hang
because you cannot even get RPC through.

Is your DNS name resolution properly working for the NFS server (maybe you lost connection to your DNS server?)

[Cliff Lim Kok Hwee]

Hi Ralph,

I performed nslookup nfsserver from nfsclient and is being resolved.

Performed the following command,

#nfsstat -cr

Client rpc:
Connection oriented:
calls badcalls badxids
0 0 0
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 0 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
8747634 602 6306
badxids timeouts waits
3521 5603 0
newcreds badverfs timers
0 0 17384
toobig nomem cantsend
0 0 0
bufulocks
0

# nfsstat -m

/tmp_mnt/net/nfsserver/opt from nfsserver:/opt (Addr 16.149.104.101)
Flags: vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5
Lookups: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)
Reads: srtt= 10 ( 25ms), dev= 4 ( 20ms), cur= 3 ( 60ms)
Writes: srtt= 93 (232ms), dev= 31 (155ms), cur= 27 (540ms)
All: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)

/tmp_mnt/net/nfsserver/rls from nfsserver:/rls (Addr 16.149.104.101)
Flags: vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5
Lookups: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)
Reads: srtt= 7 ( 17ms), dev= 4 ( 20ms), cur= 2 ( 40ms)
Writes: srtt= 23 ( 57ms), dev= 4 ( 20ms), cur= 4 ( 80ms)
All: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)

/tmp_mnt/net/nfsserver/home from nfsserver:/home (Addr 16.149.104.101)
Flags: vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5
Lookups: srtt= 8 ( 20ms), dev= 6 ( 30ms), cur= 4 ( 80ms)
All: srtt= 8 ( 20ms), dev= 21 (105ms), cur= 11 (220ms)

Query:

Commands run from nfsclient.

- How do I interprete the command nfsstat -cr for retransmission occurence?

- From nfsstat -m, I realised that the /net filesystem is not being displayed. Only displaying /opt,/rls and /home

Thanks/cliff

[Ralph Grothe]

You do have a few UDP retransmission,
but related to the total number of calls these seem to me rather neglectable.
However, they might have occured since when your RPC troubles started.

You maybe should reset the nfsstat counters by issueing

nfsstat -z | tee nfsstat_till.$(date +%Y%m%d)

Note, you need to be root on your NFS client to run this command.
Then watch if the counters of retrans and badxids grow.

Meanwhile I highly recommend you read the excellent NFS performance guide from Dave Olker.
He is the HP-UX NFS guru, and will most likely be responding to this post if his time permits

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,952,00.html

[Ralph Grothe]

The shared filesystem isn't appearing?

Check what the NFS server has eported to your client.

showmount -e

[Cliff Lim Kok Hwee]

Ralph,

Command run at nfsclient

showmount -e 16.149.104.101
export list for 16.149.104.101:
/home (everyone)
/opt (everyone)
/rls (everyone)

Correct me if I am wrong, looks like /net is not being exported to nfsclient?

Pity I cannot access nfsserver.

thanks/cliff

[Ralph Grothe]

Ask the admin of the NFS server to check for the export you are missing.
Might be that it was only exported manually and not put in /etc/exports,
and then got lost during a restart of the NFS server maybe due to a reboot.

[Ralph Grothe]

On the other hand from your "nfsstat -m" dump I can see that those shares were mounted through automounter from so called hosts mappings.
These usually get mounted under
/net// when you cd into them.
The old automounter is providing the symlinks from /tmp_mnt/net/* to /net/*
So did you really expect an export called 'net', or is it just a clashing of names?

[Cliff Lim Kok Hwee]

Ralph,

Thanks for highlighting. It should be as u mention. Not aspecting /net filesystem to be exported.

As such issue still remains....

So when normally will NFS lookup failed for server nfsserver: RPC: Unable to receive

NFS server (pid959@/net) not responding still trying
NFS server (pid959@/net) ok


Thanks/cliff

[Cliff Lim Kok Hwee]

Hi Ralph,

I can cd to /net/nfsserver/opt,/net/nfsserver/home and /net/nfsserver/rlm and performing listing etc.

But the error message appear always in dmesg.

thanks/cliff

[Jan Sladky]

hi Cliff

at first check whether all nfs server daemons
are running

rpcinfo -p server (either or clinet or server)

rcpbind (portmap), rpc.nfsd, rpc.mountd

if not start them

(on HP-UX is it done via
/sbin/init.d/nfs.core
/sbin/init.d/nfs.server)

on Linux something similar in /usr/sbin

then try mount server via IP (not name)

post results

br Jan

[Ralph Grothe]

Have the nfsstat retrans counters increased?

[Cliff Lim Kok Hwee]

Gd Afternoon Ralph/Forum,

I am checking the following parameter retrans thru the command,

#nfsstat -cr

Client rpc:
Connection oriented:
calls badcalls badxids
0 0 0
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 0 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
14469277 786 9902
badxids

timeouts waits
4998 8812 0
newcreds badverfs timers
0 0 25598
toobig nomem cantsend
0 0 0
bufulocks
0


The parameter retrans seemed to have increase from 6000+ to 9000+.

But with regards to performing nfs to the /net there is no problem.

Need your advise./ cliff

[Cliff Lim Kok Hwee]

I am also posting the nfs client daemon processes currently running on nfsclient

# ps -ef |grep rpc


root 948 1 0 Nov 7 ? 0:00 /usr/sbin/rpc.lockd
root 894 1 0 Nov 7 ? 0:00 /usr/sbin/rpcbind
root 942 1 0 Nov 7 ? 0:00 /usr/sbin/rpc.statd
root 1333 1 0 Nov 7 ? 1:13 /opt/dce/sbin/rpcd
root 1740 1 0 Nov 7 ? 0:00 /usr/sbin/rpc.mountd

#ps -ef |grep biod


root 27827 1 0 Nov 18 ? 0:00 /usr/sbin/biod 16
root 27838 1 0 Nov 18 ? 0:00 /usr/sbin/biod 16
root 27832 1 0 Nov 18 ? 0:00 /usr/sbin/biod 16

#ps -ef |grep nfs


root 27891 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 899 0 0 Nov 7 ? 0:00 nfskd
root 27890 1 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27893 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27894 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27895 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27899 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27904 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27905 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27903 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27897 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27900 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27898 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27902 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27901 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27896 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27889 1 0 Nov 18 ? 0:00 /usr/sbin/nfsd 16
root 27892 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 17320 0 0 Nov 9 ? 0:15 nfsktcpd