topic Re: NFS write very slow NFS read fast in Operating System - Linux

NFS write very slow NFS read fast

Laurent Laperrousaz — Wed, 28 Sep 2005 11:27:15 GMT

I have a share on a Linux server which is used for a large DATA storage.
It worked perfectly until we moved our servers to a new location.
We had some network topology changes but I cn't explain why now all the clients that try to write on the NFS share are very slow ! in the mean time reading is still fast...

I use on the client side:
merlin.lusis:/TANGO/LIVRAISON_HP /home/hector/merlin nfs user,async,rsize=8192,wsize=8192,hard 0 0
and on the server side:
/TANGO/LIVRAISON_HP/ *(async,rw,no_root_squash,insecure,no_wdelay)

Have any idea ?
Thanx

Re: NFS write very slow NFS read fast

Steven E. Protter — Wed, 28 Sep 2005 12:56:52 GMT

What is the configuration of the volume that is exported via NFS. If its raid 5 one would expect slow write performance due to how many disks every write needs to go to.

Further, run some nfsstat commands during write operations. You might see something useful you can post here.

I think your export is cool except the no_root_squash is a little dangerous, but that is not a performance issue.

SEP

Re: NFS write very slow NFS read fast

Donny Jekels — Wed, 28 Sep 2005 13:51:42 GMT

please post your nfsstat output to this message.

also capture top output during writes

also would need nfs version number and linux kernel version or distro.

Re: NFS write very slow NFS read fast

Laurent Laperrousaz — Wed, 28 Sep 2005 13:52:34 GMT

no it's just a fast IDE disk with no raid at all!

I will try nfsstat on both sides and post the results

The no_root_squash is to be changed I agree!

Re: NFS write very slow NFS read fast

Ivan Ferreira — Wed, 28 Sep 2005 13:54:27 GMT

Try doing a put and get using ftp. Verify the transfer speed and compare them.

Also, verify the netstat -ni output, check for packet errors and collisions. You may be having a speed negotiation problem.

Re: NFS write very slow NFS read fast

Steven E. Protter — Wed, 28 Sep 2005 13:59:15 GMT

I'd also run sar on the disk itself, sar -d and see what the data looks like during a write. IDE performance has come up lately but SCSI still out does it at the low end.

The key for Sherlock Holme's here is that things worked perfectly until we moved our servers to a new location.

Perhaps its a network bottleneck. What has changed about the location? Routers different? Firewall different? Something changed and it may not be your system.

SEP

Re: NFS write very slow NFS read fast

Laurent Laperrousaz — Wed, 28 Sep 2005 14:35:36 GMT

Server nfs v3:
null getattr setattr lookup access readlink
15 0% 274155 15% 1658 0% 1238931 69% 16212 0% 426 0%
read write create mkdir symlink mknod
63351 3% 90650 5% 1193 0% 124 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
1982 0% 262 0% 234 0% 0 0% 49 0% 5349 0%
fsstat fsinfo pathconf commit
81156 4% 8 0% 87 0% 3152 0%

lookup are high but I don't know what they correspond to!

SEP: that's the point what has changed! we changed the network mask to enlarge the LAN:
mask was 255.255.255.0 and is now 255.255.248.0
the client and the server are on the same network (no router, the client is a HP-UX V11.0 or a linux box mandrake 10.1)
I experimented the same tranfers using samba and then it works perfectly and fast.
So it is not a network issue.

Re: NFS write very slow NFS read fast

Ivan Ferreira — Wed, 28 Sep 2005 15:07:42 GMT

That nfsstat output is not very helpfull, use the -c option. Also, is not always good a big rsize and wsize.

Follow the instructions here:

http://nfs.sourceforge.net/nfs-howto/performance.html

And here:

http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.aix.doc/aixbman/prftungd/nfscliprfmon.htm

Re: NFS write very slow NFS read fast

Laurent Laperrousaz — Wed, 28 Sep 2005 17:13:31 GMT

here are the client side nfsstat result.
concerning the buffer size I used the size preconised by the nfs.sourceforge.net!
and indeed itt improved a bit the time to write a file on the nfs share.

hector@hpisim:/home/hector>nfsstat -cn

Client nfs:
calls badcalls clgets
1036825 170 1036825
cltoomany
0
Version 2: (152 calls)
null getattr setattr
0 0% 69 45% 0 0%
root lookup readlink
0 0% 81 53% 0 0%
read wrcache write
0 0% 0 0% 0 0%
create remove rename
0 0% 0 0% 0 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 1 0% 1 0%
Version 3: (1036673 calls)
null getattr setattr
0 0% 170817 16% 591 0%
lookup access readlink
806884 77% 5774 0% 425 0%
read write create
22694 2% 26095 2% 130 0%
mkdir symlink mknod
10 0% 0 0% 0 0%
remove rmdir rename
124 0% 9 0% 3 0%
link readdir readdir+
0 0% 5 0% 1931 0%
fsstat fsinfo pathconf
34 0% 1 0% 44 0%
commit
1102 0%

Re: NFS write very slow NFS read fast

Laurent Laperrousaz — Thu, 29 Sep 2005 07:54:43 GMT

I finally found what happened:
We had a big number of packet receive errors on the eth0 device. That has been solved by rebooting the server...
I thought that the problem was on the wire but after changing it, it was still bad.
Only a hardware reset on the device caused a good behaviour of the network interface.
Still I don't understand what happened, I 'm still worrying why it happened (and how)

Re: NFS write very slow NFS read fast

Bill Thorsteinson — Thu, 29 Sep 2005 09:01:50 GMT

I have run into that problem several times.
It results when the duplex on the connection
is Half Duplex on one side and Full Duplex on
the other. Transfers work well in one
direction only.

The solution I recommend is to enable
autonegotiation at the switch and server.

If you must have the duplex hard-coded, then
load the driver with the appropriate options
just before configuring it. Don't load in
/etc/modules, instead use a pre-up entry
in /etc/network/interfaces to load the
module when you are configuring it.
It appears that some switches will fall back
to Half-Duplex if there is no traffic
for a while after the line falls back
even if they are hard coded to 100FD.

Re: NFS write very slow NFS read fast

Ivan Ferreira — Thu, 29 Sep 2005 10:44:54 GMT

But when I asked if you had errors in netstat -ni you said no, and that where no hardware errors, and that other transfers methods worked fine.

If the problem was the hardware, why other transfer methods was not affected?. The protocol maybe?

Re: NFS write very slow NFS read fast

rick jones — Thu, 29 Sep 2005 12:01:07 GMT

Duplex mismatches only result in lost packets when both sides of the wire try to talk at the same time. If the applications in question are single-packet request/response - eg ping - then one will not see the "problem" with a duplex mismatch.

It is possible that link-level errors were only detected at the switch, in which case netstat would not show much. It may also be the case that the link-level errrors detected by the driver were not propagated up to netstat? Ethtool might be a good thing to use in addition to netstat.

If there were packet losses, one would expect to see retransmissions recorded by the upper layer protocol(s) - TCP, or NFS in this case.

Some boilerplate on duplex:

How Autoneg is supposed to work:

When both sides of the link are set to autoneg, they will "negotiate"
the duplex setting and select full duplex if both sides can do
full-duplex.

If one side is hardcoded and not using autoneg, the autoneg process
will "fail" and the side trying to autoneg is required by spec to use
half-duplex mode.

If one side is using half-duplex, and the other is using full-duplex,
sorrow and woe is the usual result.

So, the following table shows what will happen given various settings
on each side:

Auto Half Full

Auto Happiness Lucky Sorrow

Half Lucky Happiness Sorrow

Full Sorrow Sorrow Happiness

Happiness means that there is a good shot of everything going well.
Lucky means that things will likely go well, but not because you did
anything correctly :) Sorrow means that there _will_ be a duplex
mis-match.

When there is a duplex mismatch, on the side running half-duplex you
will see various errors and probably a number of late collisions. On
the side running full-duplex you will see things like FCS errors.
Note that those errors are not necessarily conclusive, they are simply
indicators.

Re: NFS write very slow NFS read fast

Laurent Laperrousaz — Fri, 30 Sep 2005 04:38:06 GMT

For Ivan:
I did not say I ran netstat -in but right, I tried transfers with samba that did not appear to be slow.
That is why for a while I stopped checking the low level...
I was wrong.

Thanx to everyone for their contributions

I close the thread

Re: NFS write very slow NFS read fast

Laurent Laperrousaz — Fri, 30 Sep 2005 04:40:44 GMT

By rechecking the frame errors with "ifconfig" I finally discovered that the origin of my problem was related to IP protocol negociation.

Lots of useful information were provided here.

Thanx again

Laurent