nfs block on bdf

[Todd McDaniel_1]

Hello, I have a problem that crops up every once in a great while...

it appears that my nfs mount is hanging and i dont know why.

I ran glance (G) on a pid for a bdf that hung and get "Block ON: NFS", in the last column...

When I run "truss/tusc bdf" command, it stops at the point of the NFS mount.

Any help is appreciated.

[Michael Steele_2 · Accepted solution]

Try some evaluation first. Please attach:

nfsstat -rc
nfsstat -s
netstat -m
netstat -s -p udp
vmstat -n 5 5

grep -i num_nfsd /etc/rc.config.d/nfsconf

tail -20 /etc/rc.config.d/nfsconf

[Todd McDaniel_1]

-m is not supported.

here are some of what you requested.

# netstat -s -p udp
udp:
0 incomplete headers
0 bad checksums
0 socket overflows


# tail -20 /etc/rc.config.d/nfsconf
#For the 11.0 Release line both AUTOFS and the old Automount
#are delivered. In order to invoke the AUTOFS instead of
#you must set the AUTOFS flag to 1.
#
#/usr/sbin/automount is now a script that sources in this file
#Depending on the variable AUTOFS, either AUTOFS or the old
#automount process will execute. The nfs.client start script
#will also use this variable to start the appropriate process
#during the boot sequence.
#AUTOFS= 0 - use the old automount process.
# 1 - use the new AutoFS.
#AUTOMOUNT_OPTIONS= - options to the AutoFS automount command
#AUTOMOUNTD_OPTIONS= - options to the AutoFS automountd daemon
#
#The AUTOMOUNT flag still needs to be set for either the old
#automount or new AutoFS to be started by the nfs.client script.
#
AUTOFS=0
AUTOMOUNT_OPTIONS=""
AUTOMOUNTD_OPTIONS=""

___________________________________________
Should I change the autoFS to 1? All my boxes are set to 0, to use the old method for automount.
___________________________________________
# nfsstat -rc

Client rpc:
Connection oriented:
calls badcalls badxids
99651 ... 1137 ... 0
timeouts newcreds badverfs
1136 ... 0 ... 0
timers cantconn nomem
0 ... 0 ... 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
70794 ... 0 ... 12
badxids timeouts waits
12 ... 8 ... 0
newcreds badverfs timers
0 ... 0 ... 23
toobig nomem cantsend
0 ... 0 ... 0
bufulocks
0

[Kevin Wright]

run rpcinfo -p , what do you get? Is the server OK. do other clients have this same problem?

nfsstat -m is not supported? What version of unix is this?

[Victor BERRIDGE]

Also you could start a nettl (1M) trace on client and server, wait for the issue again and use netfmt (1M) to format the traces...

Good luck
All the best
Victor

[Todd McDaniel_1]

yes, -m is supported, I must have mistyped it.

My NFS server pushing out the mount point is fine as well...

What is weird now is that everything is fine?!?>!?! not sure why....

I can't tell why it went wrong nor why it cleared up after about an hour of problems.

here is the requested info anyway.

# nfsstat -m
/appl/finapps/appscds from chdcfas4.sbc.com:/appl/finapps/appscds (Addr 132.201
.57.81)
Flags: vers=3,proto=tcp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsi
ze=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)


# rpcinfo -p chdcfas4.sbc.com
program vers proto port service
100000 4 tcp 111 rpcbind
100000 3 tcp 111 rpcbind
100000 2 tcp 111 rpcbind
100000 4 udp 111 rpcbind
100000 3 udp 111 rpcbind
100000 2 udp 111 rpcbind
100024 1 tcp 56675 status
100024 1 udp 53704 status
100021 1 tcp 56676 nlockmgr
100021 1 udp 53707 nlockmgr
100021 3 tcp 56677 nlockmgr
100021 3 udp 53708 nlockmgr
100021 4 tcp 56678 nlockmgr
100021 4 udp 53709 nlockmgr
100020 1 udp 4045 llockmgr
100020 1 tcp 4045 llockmgr
100021 2 tcp 56679 nlockmgr
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100005 1 udp 53888 mountd
100005 3 udp 53888 mountd
100005 1 tcp 56729 mountd
100005 3 tcp 56729 mountd
1073741824 2 tcp 62179

[Michael Steele_2]

Nothing there.

Is this from the client side?

Need 'nfsstat -s'.

Check for symbollic links and remove them:

cd /dir;
find . -xdev -type l -exec rm -rf {} \;

[Todd McDaniel_1]

Michael,

the problem resolved itself. I don't think that will matter but here it is.


# netstat -s
tcp:
1059356094 packets sent
3529477245 data packets (2665904024 bytes)
6046 data packets (2750882 bytes) retransmitted
1849797837 ack-only packets (66075070 delayed)
936 URG only packets
15794 window probe packets
24952255 window update packets
3078197 control packets
1377156191 packets received
1986749892 acks (for 2665676021 bytes)
22914 duplicate acks
0 acks for unsent data
4089401603 packets (1622077476 bytes) received in-sequence
0 completely duplicate packets (0 bytes)
1689 packets with some dup, data (2075219 bytes duped)
254124 out of order packets (359631053 bytes)
0 packets (0 bytes) of data after window
18361 window probes
15667397 window update packets
391 packets received after close
0 segments discarded for bad checksum
0 bad TCP segments dropped due to state change
846845 connection requests
621736 connection accepts
1468581 connections established (including accepts)
1481132 connections closed (including 13148 drops)
4201 embryonic connections dropped
1985315441 segments updated rtt (of 1985315441 attempts)
3451 retransmit timeouts
25 connections dropped by rexmit timeout
15794 persist timeouts
23686 keepalive timeouts
22900 keepalive probes sent
0 connections dropped by keepalive
8 connect requests dropped due to full queue
130316 connect requests dropped due to no listener
udp:
0 incomplete headers
0 bad checksums
0 socket overflows
ip:
763713502 total packets received
0 bad IP headers
2116 fragments received
0 fragments dropped (dup or out of space)
0 fragments dropped after timeout
0 packets forwarded
0 packets not forwardable
icmp:
54056 calls to generate an ICMP error message
0 ICMP messages dropped
Output histogram:
echo reply: 53543
destination unreachable: 418
source quench: 0
routing redirect: 0
echo: 0
time exceeded: 0
parameter problem: 0
time stamp: 0
time stamp reply: 0
address mask request: 0
address mask reply: 95
0 bad ICMP messages
Input histogram:
echo reply: 132102
destination unreachable: 1991
source quench: 760
routing redirect: 0
echo: 53543
time exceeded: 51
parameter problem: 0
time stamp request: 0
time stamp reply: 0
address mask request: 95
address mask reply: 0
53638 responses sent
igmp:
0 messages received
0 messages received with too few bytes
0 messages received with bad checksum
0 membership queries received
0 membership queries received with incorrect fields(s)
0 membership reports received
0 membership reports received with incorrect field(s)
0 membership reports received for groups to which this host belongs
0 membership reports sent