NFS Mount write issue...

[jmckinzie]

I have an nfs mount that I can see however cannot write to.

IE,
I mount nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11

to:
/pvcstst

now, I do an ll and it locks up.

I have experimented with hard/soft and timeout mount options to no avail.

I know I cannot write because I wrote a mkfile script that tried to write a file of a particular size and it just freezes the session.

Any ideas would be great.

Thanks,

Jody

[Steven E. Protter · Accepted solution]

Shalom Jody,

Take a look at the actual permissions and ownership of the folder being exported.

Take a look at the /etc/exports files for options that may prevent write access.

Also take a look at /etc/fstab on the exporting system, the exporting filesystem may be read only.

SEP

[jmckinzie]

I have checked both of these...
here is the output on servers:


Client: cfcap01d

fstab
nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11 /pvcstst nfs bg,soft,timeo=30,retrans=2,rw,nosuid,intr 0 0


server: nasfsa01

/vol/fcrr_v1/pvcs11 -access=galaxy:cfcap01d.nam.nsroot.net,root=gay

[Jaime Bolanos Rojas.]

Jody,

You might want to re-run throught this manual, just to make sure that everything is well configured on the system.

http://docs.hp.com/en/5991-1153/ch02s01.html

Regards,

jaime.

[A. Clay Stephenson]

If ll doesn't work then you have a more serious problem than noot be able to write.

You need to start at a more fundamental level. I would umount the NFS filesystem on the client and then exportfs -u the file system on the server and then umount the actual filesystem and examine the mode (permissions) of the mountpoint itself. Are those extremely restrictive? You can then mount the filesystem and examine the modes of the directories on this filesystem. Are they reasonable?

I also assume that you have looked for an installed the latest NFS patches on both the client and server.

[A. Clay Stephenson]

YUou should also examine the permission of the client's NFS mountpoint to see if it is set extremely restrictively as well.

[jmckinzie]

I have checked all these things...
They look ok.

I have also learned that I can do an ll before i try to create a file but, once a file creation is attempted, the ll command hangs...even after i kill the file creation process.

Any ideas?

[A. Clay Stephenson]

Next steps,

On the NFS server, somewhere in the exported filesystem:

echo "Test File" > mynewfile
chmod 644 mynewfile

On the NFS client (while things are still ok, ie, you haven't tried to create an nfs file) -- and cd'ed to wherever mynewfile directory is:

Can you ls mynewfile?
Can you cat mynewfile?

Now, let's try to update the file (note that this will be a write() but not a creat():
echo "Line Two" >> mynewfile
Is that ok?
Can you cat mynewfile now?

Finally, we will creat() a new file under NFS:
echo "Test File 2" > mynewfile2

It would probably help if you identified the NFS Server and Client OS's.

[A. Clay Stephenson]

By the way, the following commands assume that you are the same user (and have the same UID) on both systems, If not you should "chmod 666 mynewfile" after creating it on the NFS server.

echo "Line Two" >> mynewfile
Is that ok?
Can you cat mynewfile now?

This also brings up another question, how are you managing users? NIS? LDAP? or simply passwd files?

[jmckinzie]

I can write to the created file.
The problem is coming when I am trying to write large files.

We are using keon that updates local passwd files to manage users.

I don't believe it is a NIS issue.

[jmckinzie]

I can write to the nfs mount however, after it write 1024 bytes or 1MB it stops writing.

Any ideas as to why?

Everything else seems to be working just fine.

Please help!!!

-TIA

[A. Clay Stephenson]

There are 3 orders of magnitude difference between the sizes of your two "large" files although neither would really be considered large.

This sounds vaguely like network connectivity problems so let's take NFS out of the picture and try good old ftp.

Pick a sizeable file, e.g. cp /stand/vmunix /var/tmp # I don't want yo to clobber the real /stand/vmunix

Now,
ftp /var/tmp/vmunix in both directions. Do the transfer rates reflect approximately 50% of your Bandwidth? ie. If your are using a 100MiBit/s link are your transfer rates via ftp about 5MiByte/s? In both directions? If so, your network is fine.

Now confess, what OS is your NFS server and client running.

[jmckinzie]

I am running HPUX 11.11 on my client however, the server is a network applicance host.

It is not an HP host however, I have several other HP hosts mounted to it via NFS.

Any ideas on what to try?

I can't do the FTP thing because we do not have ftp enabled on any of our networks.

SFTP is not enabled either.

[jmckinzie]

Anyone have any other ideas to try on this?

-TIA

[OldSchool]

can you export the folder in question to one of the other HPs? If so, can you duplicate the problem from one of them?

On the NFS server: any quotas or disk space issues?

[A. Clay Stephenson]

Have you looked for and applied the latest NFS related patches? If so, which ones?

Post the output of nfsstat -c and nfsstat -m.

[jmckinzie]

I cannot recreate the problem and I have other clients mounted to the exact same directory, which eliminates the NFS server...
Here is the other information requested.


NFS patches:

swlist -l product | grep -i nfs
NFS B.11.11 ONC/NFS; Network-File System,Information Services,Utilities
PHKL_25238 1.0 11.00 NFS nfsd deadlock
PHKL_25993 1.0 thread nostop for NFS, rlimit, Ufalloc fix
PHKL_28185 1.0 Tunable;vxportal;vx_maxlink;DMAPI NFS hang
PHKL_29335 1.0 vx_nospace on NFS write.
PHKL_30151 1.0 NFS binary overwrite hang
PHNE_32477 1.0 ONC/NFS General Release/Performance Patch
# nfsstat -c
##################################
Client rpc:
Connection oriented:
calls badcalls badxids
22761 0 0
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 0 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
636 402 1610
badxids timeouts waits
0 2012 0
newcreds badverfs timers
0 0 17
toobig nomem cantsend
0 0 0
bufulocks
0

Client nfs:
calls badcalls clgets
22875 6 22875
cltoomany
0
Version 2: (0 calls)
null getattr setattr
0 0% 0 0% 0 0%
root lookup readlink
0 0% 0 0% 0 0%
read wrcache write
0 0% 0 0% 0 0%
create remove rename
0 0% 0 0% 0 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 0 0% 0 0%
Version 3: (22875 calls)
null getattr setattr
0 0% 4815 21% 1140 4%
lookup access readlink
3623 15% 1023 4% 19 0%
read write create
8123 35% 2506 10% 87 0%
mkdir symlink mknod
0 0% 0 0% 0 0%
remove rmdir rename
71 0% 0 0% 33 0%
link readdir readdir+
1 0% 13 0% 1131 4%
fsstat fsinfo pathconf
261 1% 29 0% 0 0%
commit
0 0%

#####################################

nfsstat -m

##########################
# nfsstat -m
/pvcstst from nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11 (Addr 169.163.247.219)
Flags: vers=3,proto=udp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
Lookups: srtt= 44 (110ms), dev= 17 ( 85ms), cur= 14 (280ms)
Reads: srtt= 17 ( 42ms), dev= 8 ( 40ms), cur= 6 (120ms)
Writes: srtt= 3 ( 7ms), dev= 3 ( 15ms), cur= 1 ( 20ms)
All: srtt= 44 (110ms), dev= 17 ( 85ms), cur= 14 (280ms)

/prod from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/prod (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/dv from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/dv (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/var/opt/cae from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/caeinstall (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/var/opt/caelog from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/caelog (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/var/mail from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/mail (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/etc/opt/cae from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/caedata (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home91 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home91 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home71 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home71 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home11 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home11 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home04 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home04 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home03 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home03 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home02 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home02 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/opt/lsf from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/prod/lsf (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/sysadm from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/sysadm (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/test2 from nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11 (Addr 169.163.247.219)
Flags: vers=3,proto=udp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
Lookups: srtt= 25 ( 62ms), dev= 3 ( 15ms), cur= 4 ( 80ms)
Writes: srtt= 4 ( 10ms), dev= 4 ( 20ms), cur= 2 ( 40ms)
All: srtt= 26 ( 65ms), dev= 3 ( 15ms), cur= 4 ( 80ms)

Thanks,

[Dave Olker]

Hi Jody,

It looks like the mount point you're having problems with is UDP using 32K read/write sizes. Have you tried using TCP for this mount point? Have you tried using a small rsize/wsize value?

I'd first try using TCP to see if this is a UDP-related problem. I'd then try mounting the filesystem with -rsize=512 and -wsize=512 to see if using a very small read/write request size allows this to work. If either of those suggestions work then it points to a network problem similar to what Clay was pointing to.

You can also check the lanadmin statistics of the interface card on the client to see if any errors are being logged. I would also like to see the "netstat -s" output on the client to see if it's logging any errors at the transport layer.

Regards,

Dave

[A. Clay Stephenson]

I agree with Dave, try using the NFS mount options proto=tcp,rsize=512,wsize=512.

If this fixes the problem then your network connection is suspect. Have you checked the speed/duplex settings on both your HP-UX client and the corresponding switch port?
Unless this is a Gigabit/s connection, I would hard-set both ends of the connection.
Surprisingly, if one end is running half-duplex and the other end is running full-duplex, things will almost work well and chances are, you wouldn't even notice a problem using telnet, for example.

[Dave Olker]

Hi Jody,

I realize you've closed this thread but I just wanted to comment. The fact that lowering the rsize/wsize to 1024 caused things to work is not really a solution - it just highlights the fact that you've got an underlying network problem in your environment.

NFS should be able to work just fine with 32K read/write sizes - in fact your nfsstat output shows every other NFS mount on this client was using 32K read/write sizes and apparently working just fine.

Are you now using 1024 rsize/wsize with UDP or TCP? There are known problems with certain network interface cards and UDP traffic. Did you switch to TCP or just drop the rsize/wsize values? Just curious.

Again, I'm glad your system is working, but I really think you're only hiding the problem by dropping the rsize/wsize as a permanent solution. The small rsize/wsize was a good proof point, and it's a test I've used numerous times with customers, but I urge you to investigate your networking layer a bit closer before you consider this issue resolved. Otherwise you're bound to run into a new problem with some other network application down the road caused by the same underlying problem.

Regards,

Dave

[jmckinzie]

I have reopened this thread to help troubleshoot an underlying problem.

What is the difference in the TCP vs. UDP connection as far as NFS is concerned.

[A. Clay Stephenson]

Essentially the same as it is for any other network service. Udp is a stateless protocol that can lose packets and there is no assurance that the packets will be delivered in order. Tcp, on the other hand, is a stateful protocol that assures that the packets are delivered in order. Essentially, by forcing NFS to use tcp protocol, you are moving the burden of packet handling to other mechanisms.

It was my intent by having you do the proto=tcp,wsize=512,rsize=512 at one whack to hopefully get you to a working state. I assummed that you would then break it by changing the values one at a time until the problems reaapeared. That would then be another clue.

At this point, you have fixed the symptoms but I'm not sure that you have fixed the problem.

[Dave Olker]

My preference was to try using TCP with standard 32K read/write sizes first to see if this problem is limited to UDP - seeing as the majority of the NFS mounts on this client use TCP and don't appear to have the problem - and then try UDP with small read/write sizes to see if the small read/write sizes allow UDP to suddenly work.

So what are you using now? UDP with small sizes? TCP with small sizes? TCP with 32K read/write sizes? Can you tell me if TCP with 32K read/write sizes work and only UDP fails?

Regards,

Dave

[jmckinzie]

I am currently using udp with small packet size. Here is my entry in /etc/fstab

nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/home8111 /pvcs nfs bg,proto=udp,soft,rw,nosuid,rsize=1024,wsize=1024,intr 0 0

When I try to change the udp in proto to tcp, i receive the below message, but, when I changed it back to udp, it works fine.

Any ideas?

# nfs mount: retry: retrying(1) for: /pvcs after 5 seconds
nfs mount: retry: giving up on: /pvcs

[Dave Olker]

Are you absolutely certain that the specific NFS server you're mounting from supports TCP? Looking back at your nfsstat output I didn't see any TCP mounts from this specific IP address.

If this is a NetApp box, you may have to configure it to support TCP mounts depending on the version of OnTap it's running.

Are other NFS clients able to mount filesystems from this filer using TCP? Do you have admin rights on the NetApp box where you can use the filer admin tools to verify NFS/TCP support is enabled?

Regards,

Dave