topic Re: NFS Issue in Operating System - HP-UX

NFS Issue

Cliff Lim Kok Hwee — Tue, 16 Nov 2004 03:12:18 GMT

Gd Afternoon Forum,

I am currently having a issue with NFS.

The NFS server is named nfsserver and nfs client is nfsclient for simple reference.

I do not have access to nfsserver but nfsclient I do. I performed a ping of nfsserver and successful.

Currently I am having the following error reported in the dmesg:

NFS server (pid959@/net) not responding still trying
NFS server (pid959@/net) ok
NFS server (pid959@/net) not responding still trying
NFS server (pid959@/net) ok
NFS lookup failed for server nfsserver: RPC: Unable to receive
NFS lookup failed for server nfsserver: RPC: Unable to receive

I am trying to access directory /net on nfsclient which is linked directly to nfsserver.

Please advise me.

thanks/cliff

Re: NFS Issue

Ralph Grothe — Tue, 16 Nov 2004 03:17:59 GMT

Probably you have a somewhat congested that network between NFS server and client.
Have a look if there are many retransmissions taking place.
Look at the output of nfsstat.

e.g.

nfsstat -cr

also look at the soft round trip time of your NFS mounts

nfsstat -m

Re: NFS Issue

Ralph Grothe — Tue, 16 Nov 2004 03:22:08 GMT

Ah, these commands will probably hang
because you cannot even get RPC through.

Is your DNS name resolution properly working for the NFS server (maybe you lost connection to your DNS server?)

Re: NFS Issue

Cliff Lim Kok Hwee — Tue, 16 Nov 2004 03:35:08 GMT

Hi Ralph,

I performed nslookup nfsserver from nfsclient and is being resolved.

Performed the following command,

#nfsstat -cr

Client rpc:
Connection oriented:
calls badcalls badxids
0 0 0
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 0 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
8747634 602 6306
badxids timeouts waits
3521 5603 0
newcreds badverfs timers
0 0 17384
toobig nomem cantsend
0 0 0
bufulocks
0

# nfsstat -m

/tmp_mnt/net/nfsserver/opt from nfsserver:/opt (Addr 16.149.104.101)
Flags: vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5
Lookups: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)
Reads: srtt= 10 ( 25ms), dev= 4 ( 20ms), cur= 3 ( 60ms)
Writes: srtt= 93 (232ms), dev= 31 (155ms), cur= 27 (540ms)
All: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)

/tmp_mnt/net/nfsserver/rls from nfsserver:/rls (Addr 16.149.104.101)
Flags: vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5
Lookups: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)
Reads: srtt= 7 ( 17ms), dev= 4 ( 20ms), cur= 2 ( 40ms)
Writes: srtt= 23 ( 57ms), dev= 4 ( 20ms), cur= 4 ( 80ms)
All: srtt= 7 ( 17ms), dev= 3 ( 15ms), cur= 2 ( 40ms)

/tmp_mnt/net/nfsserver/home from nfsserver:/home (Addr 16.149.104.101)
Flags: vers=2,proto=udp,auth=unix,soft,intr,dynamic,rsize=8192,wsize=8192,retrans=5
Lookups: srtt= 8 ( 20ms), dev= 6 ( 30ms), cur= 4 ( 80ms)
All: srtt= 8 ( 20ms), dev= 21 (105ms), cur= 11 (220ms)

Query:

Commands run from nfsclient.

- How do I interprete the command nfsstat -cr for retransmission occurence?

- From nfsstat -m, I realised that the /net filesystem is not being displayed. Only displaying /opt,/rls and /home

Thanks/cliff

Re: NFS Issue

Ralph Grothe — Tue, 16 Nov 2004 04:15:17 GMT

You do have a few UDP retransmission,
but related to the total number of calls these seem to me rather neglectable.
However, they might have occured since when your RPC troubles started.

You maybe should reset the nfsstat counters by issueing

nfsstat -z | tee nfsstat_till.$(date +%Y%m%d)

Note, you need to be root on your NFS client to run this command.
Then watch if the counters of retrans and badxids grow.

Meanwhile I highly recommend you read the excellent NFS performance guide from Dave Olker.
He is the HP-UX NFS guru, and will most likely be responding to this post if his time permits

http://h21007.www2.hp.com/dspp/tech/tech_TechDocumentDetailPage_IDX/1,1701,952,00.html

Re: NFS Issue

Ralph Grothe — Tue, 16 Nov 2004 04:21:04 GMT

The shared filesystem isn't appearing?

Check what the NFS server has eported to your client.

showmount -e

Re: NFS Issue

Cliff Lim Kok Hwee — Tue, 16 Nov 2004 04:32:14 GMT

Ralph,

Command run at nfsclient

showmount -e 16.149.104.101
export list for 16.149.104.101:
/home (everyone)
/opt (everyone)
/rls (everyone)

Correct me if I am wrong, looks like /net is not being exported to nfsclient?

Pity I cannot access nfsserver.

thanks/cliff

Re: NFS Issue

Ralph Grothe — Tue, 16 Nov 2004 04:56:59 GMT

Ask the admin of the NFS server to check for the export you are missing.
Might be that it was only exported manually and not put in /etc/exports,
and then got lost during a restart of the NFS server maybe due to a reboot.

Re: NFS Issue

Ralph Grothe — Tue, 16 Nov 2004 05:07:24 GMT

On the other hand from your "nfsstat -m" dump I can see that those shares were mounted through automounter from so called hosts mappings.
These usually get mounted under
/net// when you cd into them.
The old automounter is providing the symlinks from /tmp_mnt/net/* to /net/*
So did you really expect an export called 'net', or is it just a clashing of names?

Re: NFS Issue

Cliff Lim Kok Hwee — Tue, 16 Nov 2004 05:14:13 GMT

Ralph,

Thanks for highlighting. It should be as u mention. Not aspecting /net filesystem to be exported.

As such issue still remains....

So when normally will NFS lookup failed for server nfsserver: RPC: Unable to receive

NFS server (pid959@/net) not responding still trying
NFS server (pid959@/net) ok

Thanks/cliff

Re: NFS Issue

Cliff Lim Kok Hwee — Tue, 16 Nov 2004 05:21:55 GMT

Hi Ralph,

I can cd to /net/nfsserver/opt,/net/nfsserver/home and /net/nfsserver/rlm and performing listing etc.

But the error message appear always in dmesg.

thanks/cliff

Re: NFS Issue

Jan Sladky — Tue, 16 Nov 2004 06:35:59 GMT

hi Cliff

at first check whether all nfs server daemons
are running

rpcinfo -p server (either or clinet or server)

rcpbind (portmap), rpc.nfsd, rpc.mountd

if not start them

(on HP-UX is it done via
/sbin/init.d/nfs.core
/sbin/init.d/nfs.server)

on Linux something similar in /usr/sbin

then try mount server via IP (not name)

post results

br Jan

Re: NFS Issue

Ralph Grothe — Tue, 16 Nov 2004 06:54:05 GMT

Have the nfsstat retrans counters increased?

Re: NFS Issue

Cliff Lim Kok Hwee — Fri, 19 Nov 2004 01:27:09 GMT

Gd Afternoon Ralph/Forum,

I am checking the following parameter retrans thru the command,

#nfsstat -cr

Client rpc:
Connection oriented:
calls badcalls badxids
0 0 0
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 0 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
14469277 786 9902
badxids

timeouts waits
4998 8812 0
newcreds badverfs timers
0 0 25598
toobig nomem cantsend
0 0 0
bufulocks
0

The parameter retrans seemed to have increase from 6000+ to 9000+.

But with regards to performing nfs to the /net there is no problem.

Need your advise./ cliff

Re: NFS Issue

Cliff Lim Kok Hwee — Fri, 19 Nov 2004 01:35:35 GMT

I am also posting the nfs client daemon processes currently running on nfsclient

# ps -ef |grep rpc

root 948 1 0 Nov 7 ? 0:00 /usr/sbin/rpc.lockd
root 894 1 0 Nov 7 ? 0:00 /usr/sbin/rpcbind
root 942 1 0 Nov 7 ? 0:00 /usr/sbin/rpc.statd
root 1333 1 0 Nov 7 ? 1:13 /opt/dce/sbin/rpcd
root 1740 1 0 Nov 7 ? 0:00 /usr/sbin/rpc.mountd

#ps -ef |grep biod

root 27827 1 0 Nov 18 ? 0:00 /usr/sbin/biod 16
root 27838 1 0 Nov 18 ? 0:00 /usr/sbin/biod 16
root 27832 1 0 Nov 18 ? 0:00 /usr/sbin/biod 16

#ps -ef |grep nfs

root 27891 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 899 0 0 Nov 7 ? 0:00 nfskd
root 27890 1 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27893 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27894 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27895 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27899 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27904 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27905 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27903 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27897 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27900 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27898 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27902 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27901 27890 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27896 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 27889 1 0 Nov 18 ? 0:00 /usr/sbin/nfsd 16
root 27892 27891 0 Nov 18 ? 0:01 /usr/sbin/nfsd 16
root 17320 0 0 Nov 9 ? 0:15 nfsktcpd