topic Re: NFS timeout in Operating System - HP-UX

NFS timeout

Michael Allmer — Thu, 18 Jan 2007 09:47:36 GMT

We are NFS mounting a share from a VMS systems to a HPUX. The issue is when we try and do a listing on a sub directory we get an PRC timeout. We can attempt again, gettting a timeout yet again, the thrid time we get the listing.
The subdir has over 6000 files. other directories that have few do not behave this way.
Looking for Ideas?

Thanks

Re: NFS timeout

Pete Randall — Thu, 18 Jan 2007 09:55:24 GMT

Mike,

You might try raising the timeout value with the "mount -o timeo=" command. The default is specified in the kernel. Try doubling it and see if this helps.

Pete

Re: NFS timeout

Dave Olker — Thu, 18 Jan 2007 11:11:11 GMT

Hi Mike,

What OS is the HP-UX client running?

# uname -a

What transport protocol is the mount using? What NFS version is the mount using?

# nfsstat -m

Please post the output from those two commands so I can at least see how the NFS mount is configured and what OS you're running.

Regards,

Dave

Re: NFS timeout

Michael Allmer — Thu, 18 Jan 2007 11:25:05 GMT

HP-UX greta B.11.00 A 9000/800

/camdsk from lijms1:/camdsk (Addr 170.126.128.4)
Flags: vers=3,proto=udp,auth=unix,soft,intr,link,devs,rsize=8192,wsize=8192,r
etrans=5
Lookups: srtt= 7 ( 17ms), dev= 4 ( 20ms), cur= 2 ( 40ms)
Reads: srtt= 8 ( 20ms), dev= 4 ( 20ms), cur= 3 ( 60ms)
All: srtt= 7 ( 17ms), dev= 4 ( 20ms), cur= 2 ( 40ms)

Re: NFS timeout

Dave Olker — Thu, 18 Jan 2007 11:34:36 GMT

Ok, so you're running 11.0 and using UDP. It is possible the problem is your network is so busy that it's dropping too many packets for the READDIRPLUS to finish successfully in the large directory.

There are a few options but the first one I'd suggest is using TCP for the mount. If your network is congested and you're dropping packets, TCP should work better in that environment.

11.0 does support TCP but you need to be at a certain NFS patch level and then you have to enable TCP support. Can you tell me what is the latest NFS patch installed on your system:

# swlist -l product | grep -iE 'onc|nfs'

and the contents of your /etc/rc.config.d/nfsconf file:

# cat /etc/rc.config.d/nfsconf

so I can see which patches are installed and if TCP support is enabled on the system.

Regards,

Dave

Re: NFS timeout

Michael Allmer — Thu, 18 Jan 2007 11:47:29 GMT

Would this be a client issue or a server issue? I mount other NFS shares from an HP-UX server that does not act this way.

Re: NFS timeout

Dave Olker — Thu, 18 Jan 2007 12:06:58 GMT

That's what I'm trying to determine.

If your client is mounting filesystems using UDP it's either because it hasn't been configured to use TCP or because your server doesn't support TCP. If you want a way of quickly checking if your NFS server supports TCP try this:

# rpcinfo -p

Looking at the output of the rpcinfo command we should be able to see if the VMS server is registering a TCP version of NFS.

Once we know whether both client and server support TCP, the goal will be to get this filesystem mounted with TCP and see if that resolves your timeouts. If TCP doesn't resolve it then we'll start looking elsewhere, but TCP is the first thing to check.

Regards,

Dave

Re: NFS timeout

Michael Allmer — Thu, 18 Jan 2007 12:38:36 GMT

The attached file has the output requested.

Re: NFS timeout

Dave Olker — Thu, 18 Jan 2007 12:49:38 GMT

Ok, it looks like your server does support TCP, you have the latest 11.0 NFS patch installed but TCP is not enabled on the client.

You need to use the following commands to enable NFS/TCP support:

# /sbin/init.d/nfs.client stop
# /usr/sbin/setoncenv NFS_TCP 1
# /sbin/init.d/nfs.client start

This should stop the NFS client services, enable TCP support and then re-start the NFS client stuff. After these commands you should be able to mount the filesystem from this VMS server via TCP.

If the VMS filesystem is still mounted you'll need to unmount it and re-mount it to change from UDP to TCP. To confirm which protocol is in use after you remount the filesystem you can issue the command:

# nfsstat -m

and you should see TCP as the protocol. Once you've confirmed the filesystem is mounted with TCP try the directory listing again and see if it behaves any better.

Regards,

Dave

Re: NFS timeout

Michael Allmer — Thu, 18 Jan 2007 14:19:04 GMT

I will make the changes tonight, and let you know how it goes.

Thank you for the information.

Mike

Re: NFS timeout

Michael Allmer — Thu, 18 Jan 2007 23:13:38 GMT

I made the changes suggested by Dave.

All mounts are connected via tcp

The first listing of the subdirectories resulted in a fast rpc timeout return.

Ran the ll again, I got the rpc timeout a third an fourth time.

cd to nfs share then ran ll and got list.

cd to large directory containing 6000+ files and ran ll.

It finally returned results.

entry in /etc/fstab is;
lijms1:/camdsk /camdsk nfs rw,suid,soft,retry=2,retrans=8,timeo=30 0 0

Do you think hard mounting this will resolve this, or is there some other underlining problem.

Re: NFS timeout

Michael Allmer — Fri, 19 Jan 2007 00:39:01 GMT

Since I will not be able to monitor this for the rest of the week, I will close this thread.

Since this nfs mount is only needed for a script, I have setup the following to overcome this for now.

I hard mounted the share.

I then created to shell scripts; one to mount the other to umount the share.

I have scheduled these on cron around the other script.

While I still get the follwoing after geting a list of files, it does come back.

NFS server lijms1 not responding still trying
NFS server lijms1 not responding still trying
NFS server lijms1 ok

Thank you for all the assistance.

Re: NFS timeout

Michael Allmer — Fri, 19 Jan 2007 00:39:59 GMT

Will look for answers agian sometime.

Re: NFS timeout

Dave Olker — Fri, 19 Jan 2007 02:16:53 GMT

Hi Mike,

Now that you're using TCP, I think this is your new problem:

> entry in /etc/fstab is;
> lijms1:/camdsk /camdsk nfs
> rw,suid,soft,retry=2,retrans=8,timeo=30

With TCP mounts you definitely want to get rid of all the mount options and just use the defaults. The retry, retrans, and timeo options will wreak havoc with TCP mounts.

My recommendation is to remove all the mount options from the /etc/fstab entry and just use the defaults. On 11.0 systems, the only mount options I typically change are the "rsize" and "wsize" options, since those default to 8K on 11.0 and 32K on every release after 11.0. I wouldn't have suggested 32K reads and writes with UDP in your environment (you're having enough problems with 8K reads and writes) but now that you've switched to TCP if you get rid of the other mount options you can try experimenting with larger rsize/wsize mount options as well. You may find the combination of TCP, 32K reads/writes and default for all other mount options actually increases your performance significantly.

I suggest you do a quick test to see if removing all the mount options will work. Try this:

# mkdir /nfstest
# mount lijms1:/camdsk /nfstest

Then try issuing the commands in the /nfstest directory. If it works as desired then you can try this:

# umount /nfstest
# mount -o rsize=32768,wsize=32768 lijms1:/camdsk /nfstest

See if the new rsize/wsize gives even better performance. It may have no effect on the performance of reading the directory but it may improve things for reading and writing actual files.

Once you decide on a "good" set of mount options you can modify your /etc/fstab entry for the real mount point accordingly. Then just unmount/remount with the new options and you're done.

Regards,

Dave