cp on nfs mount sleeps

[Scott Van de Workeen]

To anyone that can assist with this. I am losing my mind. I have a DS20E, attached to my corporate network (like 55 other tru64 servers), and when I do a cp from an NFS mounted directory to this server, the cp process does a little then just sits there. No other system on my network does this. I have tried an ee0 card, and a tu1 card, verified the speed and duplex settings as matching the Cisco port settings of 100/Full, and yet it still does this.
This system is on NIS, and the NFS mounts are all automounted.

Does anyone have any thoughts on this?

Please Help!

[Victor Semaska_3 · Accepted solution]

When the cp is in progress do a '# /usr/sbin/netstat -i' and see if the error counts are going up.

Does '# /bin/nfsstat' show anything unusual.

Vic

[Alexey Borchev]

I would try:
1) Ping with variouse packetsize. From this DS20E to NFS server and back - from NFS server towards this DS20E.
2) FTP same file from same dir.
3) Maybe, rcp - again, same file from same dir.
4) Try cp, ftp, rcp this file in both directions.

[Scott Van de Workeen]

I have tried this copy process in both directions now. It is fast from the 5.1b server to the NFS server, but slow from the NFS server to the 5.1B server.

ftp is fast. mv responds the same as cp.

Here is a twist. I have 6 other 5.1B servers, and they all are doing this now. I have 15 5.1A servers, and they do not have this issue.

I have 12 4.0D - 4.0G systems and they don't do this either. It seems that this is limited to 5.1B.

If it makes any difference, the NFS servers are Sun V480's running Solaris 2.9 (12/03), and the disk is mounted from an EMC Clarion SAN array.

[Scott Van de Workeen]

This is what I get from nfsstat during the copy:

# nfsstat

Server rpc:
tcp: calls badcalls nullrecv badlen xdrcall creates
0 0 0 0 0 0
udp: calls badcalls nullrecv badlen xdrcall
0 0 0 0 0

Server nfs:
calls badcalls badprog badproc badvers badargs
0 0 0 0 0 0
unprivport weakauth
0 0

Server nfs V2: (0 out of 0 calls)
null getattr setattr root lookup readlink read
0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
wrcache write create remove rename link symlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
mkdir rmdir readdir statfs
0 0% 0 0% 0 0% 0 0%

Server nfs V3: (0 out of 0 calls)
null getattr setattr lookup access readlink read
0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
write create mkdir symlink mknod remove rmdir
0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
rename link readdir readdir+ fsstat fsinfo pathconf
0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
commit
0 0%

Client rpc:
tcp: calls badxids badverfs timeouts newcreds
0 0 0 0 0
creates connects badconns inputs avails interrupts
0 0 0 0 0 0
udp: calls badxids badverfs timeouts newcreds retrans
57983 1 0 2417 0 2417
badcalls timers waits
1 5340 0

Client nfs:
calls retrans badcalls nclget nclsleep ndestroys ncleans
57983 0 1 57983 0 20 57963

Client nfs V2: (341 out of 57983 calls)
null getattr setattr root lookup readlink read
0 0% 150 43% 0 0% 0 0% 75 21% 9 2% 0 0%
wrcache write create remove rename link symlink
0 0% 0 0% 0 0% 0 0% 0 0% 0 0% 0 0%
mkdir rmdir readdir statfs
0 0% 0 0% 2 0% 105 30%

Client nfs V3: (57642 out of 57983 calls)
null getattr setattr lookup access readlink read
0 0% 69 0% 6 0% 123 0% 41 0% 4 0% 16742 29%
write create mkdir symlink mknod remove rmdir
40483 70% 4 0% 0 0% 0 0% 0 0% 2 0% 0 0%
rename link readdir readdir+ fsstat fsinfo pathconf
0 0% 0 0% 14 0% 10 0% 57 0% 9 0% 48 0%
commit
30 0%
I don't see any issues, but maybe I am blind.....

[Victor Semaska_3]

udp: calls badxids badverfs timeouts newcreds retrans
57983 1 0 2417 0 2417

Looks like you got timeouts and retransmissions. Did you zero out the counts with '# nfsstat -z' before trying the copy?

How about the '# netstat -i'? What are the counts for Ierrs & Oerrs?

Vic

[Scott Van de Workeen]

netstat -i reports 0 Ierrs and 0 Oerrs...

You are correct I did not clear the tables prior to the copy. But I will say that the numbers were the same after the cp test as they were before it.


Thanks for all of these suggestions.

[Alexey Borchev]

I would check up NFS mount settings -
NFS v2 or v3, TCP or UDP.
1) Is it the same on 5.1B and 5.1A systems?
2) Maybe, try to fiddle with NFS v2 or v3, TCP or UDP?

[Victor Semaska_3]

I can't remember the specifics but I did have problems with automount once. Also, I think on V5.1B automount should be replaced with autofs. I've never used it.

How about taking automount out of the equation for now and manually mount the NFS disk to see what happens.

Alexey's suggestion of trying the different protocols is a good idea. In the mount command it's the '-o proto=tcp' or '-o proto=udp' option. The mount command should look something like this:

# mount -t nfs -o proto=udp|tcp :/ /

Vic

[Scott Van de Workeen]

Vic,
I will try your suggestions. I may not get to try them until after 5pm EST.....

[Scott Van de Workeen]

I have tried the suggestions of proto=udp, proto=tcp on manual nfs mounts, changed the system to use autofs and autofsmount, and nothing changes the situation.

I did some traceroutes to the NFS servers and they take an average of 4 miliseconds , where other systems on the same subnet/switch never reach 1 milisecond.

I am at a total loss as to what to try next.

[Victor Semaska_3]

Sorry, running out of ideas myself.

One thing to try is export a directory from one of the other V5.1B (or V5.1A) systems to see if that works. This is to see if it has anything to do with the Sun server.

Last thing I can think of is have networking put a sniffer on the network and try to determine where it's stalling. We once had NFS problems because of something (I think) called Packeteer. Any network changes recently?

Vic

[Alexey Borchev]

The observation on traceroute times (1 ms vs. 4 ms.) may give a klue.

I would try to plug this system into another port (preferably, into another switch).

And check traceroute times and NFS speed again.

[Fabrice Rafart_1]

Hi,

You also can try to use truss on cp and see ...

/usr/opt/svr4/usr/bin/truss -o /tmp/truss.out cp ...

Regards,

[Al Licause]

You mentioned at least one system have a 100Mbps interface. Do any of the other systems, in particular the server, have gigabit interfaces ?

If so, on the gigabit side, check to see if flow control has specifically been enabled on the switch. Don't leave this up to auto negotiation....make sure it has been specifically enabled in both directions.

We have seen a few similar cases recently in which flow control has remedied such situations.