Operating System - Linux
1829575 Members
3422 Online
109992 Solutions
New Discussion

Re: NFS write very slow NFS read fast

 
Laurent Laperrousaz
Regular Advisor

NFS write very slow NFS read fast

I have a share on a Linux server which is used for a large DATA storage.
It worked perfectly until we moved our servers to a new location.
We had some network topology changes but I cn't explain why now all the clients that try to write on the NFS share are very slow ! in the mean time reading is still fast...

I use on the client side:
merlin.lusis:/TANGO/LIVRAISON_HP /home/hector/merlin nfs user,async,rsize=8192,wsize=8192,hard 0 0
and on the server side:
/TANGO/LIVRAISON_HP/ *(async,rw,no_root_squash,insecure,no_wdelay)

Have any idea ?
Thanx
14 REPLIES 14
Steven E. Protter
Exalted Contributor

Re: NFS write very slow NFS read fast

What is the configuration of the volume that is exported via NFS. If its raid 5 one would expect slow write performance due to how many disks every write needs to go to.

Further, run some nfsstat commands during write operations. You might see something useful you can post here.

I think your export is cool except the no_root_squash is a little dangerous, but that is not a performance issue.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Donny Jekels
Respected Contributor

Re: NFS write very slow NFS read fast

please post your nfsstat output to this message.

also capture top output during writes

also would need nfs version number and linux kernel version or distro.
"Vision, is the art of seeing the invisible"
Laurent Laperrousaz
Regular Advisor

Re: NFS write very slow NFS read fast

no it's just a fast IDE disk with no raid at all!

I will try nfsstat on both sides and post the results

The no_root_squash is to be changed I agree!
Ivan Ferreira
Honored Contributor

Re: NFS write very slow NFS read fast

Try doing a put and get using ftp. Verify the transfer speed and compare them.

Also, verify the netstat -ni output, check for packet errors and collisions. You may be having a speed negotiation problem.
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Steven E. Protter
Exalted Contributor

Re: NFS write very slow NFS read fast

I'd also run sar on the disk itself, sar -d and see what the data looks like during a write. IDE performance has come up lately but SCSI still out does it at the low end.

The key for Sherlock Holme's here is that things worked perfectly until we moved our servers to a new location.

Perhaps its a network bottleneck. What has changed about the location? Routers different? Firewall different? Something changed and it may not be your system.

SEP
Steven E Protter
Owner of ISN Corporation
http://isnamerica.com
http://hpuxconsulting.com
Sponsor: http://hpux.ws
Twitter: http://twitter.com/hpuxlinux
Founder http://newdatacloud.com
Laurent Laperrousaz
Regular Advisor

Re: NFS write very slow NFS read fast

Server nfs v3:
null getattr setattr lookup access readlink
15 0% 274155 15% 1658 0% 1238931 69% 16212 0% 426 0%
read write create mkdir symlink mknod
63351 3% 90650 5% 1193 0% 124 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
1982 0% 262 0% 234 0% 0 0% 49 0% 5349 0%
fsstat fsinfo pathconf commit
81156 4% 8 0% 87 0% 3152 0%

lookup are high but I don't know what they correspond to!

SEP: that's the point what has changed! we changed the network mask to enlarge the LAN:
mask was 255.255.255.0 and is now 255.255.248.0
the client and the server are on the same network (no router, the client is a HP-UX V11.0 or a linux box mandrake 10.1)
I experimented the same tranfers using samba and then it works perfectly and fast.
So it is not a network issue.

Ivan Ferreira
Honored Contributor

Re: NFS write very slow NFS read fast

That nfsstat output is not very helpfull, use the -c option. Also, is not always good a big rsize and wsize.

Follow the instructions here:

http://nfs.sourceforge.net/nfs-howto/performance.html

And here:

http://publib.boulder.ibm.com/infocenter/pseries/index.jsp?topic=/com.ibm.aix.doc/aixbman/prftungd/nfscliprfmon.htm
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
Laurent Laperrousaz
Regular Advisor

Re: NFS write very slow NFS read fast

here are the client side nfsstat result.
concerning the buffer size I used the size preconised by the nfs.sourceforge.net!
and indeed itt improved a bit the time to write a file on the nfs share.

hector@hpisim:/home/hector>nfsstat -cn

Client nfs:
calls badcalls clgets
1036825 170 1036825
cltoomany
0
Version 2: (152 calls)
null getattr setattr
0 0% 69 45% 0 0%
root lookup readlink
0 0% 81 53% 0 0%
read wrcache write
0 0% 0 0% 0 0%
create remove rename
0 0% 0 0% 0 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 1 0% 1 0%
Version 3: (1036673 calls)
null getattr setattr
0 0% 170817 16% 591 0%
lookup access readlink
806884 77% 5774 0% 425 0%
read write create
22694 2% 26095 2% 130 0%
mkdir symlink mknod
10 0% 0 0% 0 0%
remove rmdir rename
124 0% 9 0% 3 0%
link readdir readdir+
0 0% 5 0% 1931 0%
fsstat fsinfo pathconf
34 0% 1 0% 44 0%
commit
1102 0%
Laurent Laperrousaz
Regular Advisor

Re: NFS write very slow NFS read fast

I finally found what happened:
We had a big number of packet receive errors on the eth0 device. That has been solved by rebooting the server...
I thought that the problem was on the wire but after changing it, it was still bad.
Only a hardware reset on the device caused a good behaviour of the network interface.
Still I don't understand what happened, I 'm still worrying why it happened (and how)
Bill Thorsteinson
Honored Contributor

Re: NFS write very slow NFS read fast

I have run into that problem several times.
It results when the duplex on the connection
is Half Duplex on one side and Full Duplex on
the other. Transfers work well in one
direction only.

The solution I recommend is to enable
autonegotiation at the switch and server.

If you must have the duplex hard-coded, then
load the driver with the appropriate options
just before configuring it. Don't load in
/etc/modules, instead use a pre-up entry
in /etc/network/interfaces to load the
module when you are configuring it.
It appears that some switches will fall back
to Half-Duplex if there is no traffic
for a while after the line falls back
even if they are hard coded to 100FD.
Ivan Ferreira
Honored Contributor

Re: NFS write very slow NFS read fast

But when I asked if you had errors in netstat -ni you said no, and that where no hardware errors, and that other transfers methods worked fine.

If the problem was the hardware, why other transfer methods was not affected?. The protocol maybe?
Por que hacerlo dificil si es posible hacerlo facil? - Why do it the hard way, when you can do it the easy way?
rick jones
Honored Contributor

Re: NFS write very slow NFS read fast

Duplex mismatches only result in lost packets when both sides of the wire try to talk at the same time. If the applications in question are single-packet request/response - eg ping - then one will not see the "problem" with a duplex mismatch.

It is possible that link-level errors were only detected at the switch, in which case netstat would not show much. It may also be the case that the link-level errrors detected by the driver were not propagated up to netstat? Ethtool might be a good thing to use in addition to netstat.

If there were packet losses, one would expect to see retransmissions recorded by the upper layer protocol(s) - TCP, or NFS in this case.

Some boilerplate on duplex:

How Autoneg is supposed to work:

When both sides of the link are set to autoneg, they will "negotiate"
the duplex setting and select full duplex if both sides can do
full-duplex.

If one side is hardcoded and not using autoneg, the autoneg process
will "fail" and the side trying to autoneg is required by spec to use
half-duplex mode.

If one side is using half-duplex, and the other is using full-duplex,
sorrow and woe is the usual result.

So, the following table shows what will happen given various settings
on each side:

Auto Half Full

Auto Happiness Lucky Sorrow

Half Lucky Happiness Sorrow

Full Sorrow Sorrow Happiness

Happiness means that there is a good shot of everything going well.
Lucky means that things will likely go well, but not because you did
anything correctly :) Sorrow means that there _will_ be a duplex
mis-match.

When there is a duplex mismatch, on the side running half-duplex you
will see various errors and probably a number of late collisions. On
the side running full-duplex you will see things like FCS errors.
Note that those errors are not necessarily conclusive, they are simply
indicators.
there is no rest for the wicked yet the virtuous have no pillows
Laurent Laperrousaz
Regular Advisor

Re: NFS write very slow NFS read fast

For Ivan:
I did not say I ran netstat -in but right, I tried transfers with samba that did not appear to be slow.
That is why for a while I stopped checking the low level...
I was wrong.

Thanx to everyone for their contributions

I close the thread
Laurent Laperrousaz
Regular Advisor

Re: NFS write very slow NFS read fast

By rechecking the frame errors with "ifconfig" I finally discovered that the origin of my problem was related to IP protocol negociation.

Lots of useful information were provided here.

Thanx again

Laurent