topic Re: NFS Mount write issue... in Operating System - HP-UX

NFS Mount write issue...

jmckinzie — Tue, 18 Jul 2006 13:51:06 GMT

I have an nfs mount that I can see however cannot write to.

IE,
I mount nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11

to:
/pvcstst

now, I do an ll and it locks up.

I have experimented with hard/soft and timeout mount options to no avail.

I know I cannot write because I wrote a mkfile script that tried to write a file of a particular size and it just freezes the session.

Any ideas would be great.

Thanks,

Jody

Re: NFS Mount write issue...

Steven E. Protter — Tue, 18 Jul 2006 13:56:16 GMT

Shalom Jody,

Take a look at the actual permissions and ownership of the folder being exported.

Take a look at the /etc/exports files for options that may prevent write access.

Also take a look at /etc/fstab on the exporting system, the exporting filesystem may be read only.

SEP

Re: NFS Mount write issue...

jmckinzie — Tue, 18 Jul 2006 14:01:34 GMT

I have checked both of these...
here is the output on servers:

Client: cfcap01d

fstab
nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11 /pvcstst nfs bg,soft,timeo=30,retrans=2,rw,nosuid,intr 0 0

server: nasfsa01

/vol/fcrr_v1/pvcs11 -access=galaxy:cfcap01d.nam.nsroot.net,root=gay

Re: NFS Mount write issue...

Jaime Bolanos Rojas. — Tue, 18 Jul 2006 14:03:47 GMT

Jody,

You might want to re-run throught this manual, just to make sure that everything is well configured on the system.

http://docs.hp.com/en/5991-1153/ch02s01.html

Regards,

jaime.

Re: NFS Mount write issue...

A. Clay Stephenson — Tue, 18 Jul 2006 14:16:02 GMT

If ll doesn't work then you have a more serious problem than noot be able to write.

You need to start at a more fundamental level. I would umount the NFS filesystem on the client and then exportfs -u the file system on the server and then umount the actual filesystem and examine the mode (permissions) of the mountpoint itself. Are those extremely restrictive? You can then mount the filesystem and examine the modes of the directories on this filesystem. Are they reasonable?

I also assume that you have looked for an installed the latest NFS patches on both the client and server.

Re: NFS Mount write issue...

A. Clay Stephenson — Tue, 18 Jul 2006 14:25:29 GMT

YUou should also examine the permission of the client's NFS mountpoint to see if it is set extremely restrictively as well.

Re: NFS Mount write issue...

jmckinzie — Tue, 18 Jul 2006 14:52:56 GMT

I have checked all these things...
They look ok.

I have also learned that I can do an ll before i try to create a file but, once a file creation is attempted, the ll command hangs...even after i kill the file creation process.

Any ideas?

Re: NFS Mount write issue...

A. Clay Stephenson — Tue, 18 Jul 2006 15:07:34 GMT

Next steps,

On the NFS server, somewhere in the exported filesystem:

echo "Test File" > mynewfile
chmod 644 mynewfile

On the NFS client (while things are still ok, ie, you haven't tried to create an nfs file) -- and cd'ed to wherever mynewfile directory is:

Can you ls mynewfile?
Can you cat mynewfile?

Now, let's try to update the file (note that this will be a write() but not a creat():
echo "Line Two" >> mynewfile
Is that ok?
Can you cat mynewfile now?

Finally, we will creat() a new file under NFS:
echo "Test File 2" > mynewfile2

It would probably help if you identified the NFS Server and Client OS's.

Re: NFS Mount write issue...

A. Clay Stephenson — Tue, 18 Jul 2006 15:14:06 GMT

By the way, the following commands assume that you are the same user (and have the same UID) on both systems, If not you should "chmod 666 mynewfile" after creating it on the NFS server.

echo "Line Two" >> mynewfile
Is that ok?
Can you cat mynewfile now?

This also brings up another question, how are you managing users? NIS? LDAP? or simply passwd files?

Re: NFS Mount write issue...

jmckinzie — Tue, 18 Jul 2006 16:06:32 GMT

I can write to the created file.
The problem is coming when I am trying to write large files.

We are using keon that updates local passwd files to manage users.

I don't believe it is a NIS issue.

Re: NFS Mount write issue...

jmckinzie — Tue, 18 Jul 2006 21:27:56 GMT

I can write to the nfs mount however, after it write 1024 bytes or 1MB it stops writing.

Any ideas as to why?

Everything else seems to be working just fine.

Please help!!!

-TIA

Re: NFS Mount write issue...

A. Clay Stephenson — Tue, 18 Jul 2006 22:06:59 GMT

There are 3 orders of magnitude difference between the sizes of your two "large" files although neither would really be considered large.

This sounds vaguely like network connectivity problems so let's take NFS out of the picture and try good old ftp.

Pick a sizeable file, e.g. cp /stand/vmunix /var/tmp # I don't want yo to clobber the real /stand/vmunix

Now,
ftp /var/tmp/vmunix in both directions. Do the transfer rates reflect approximately 50% of your Bandwidth? ie. If your are using a 100MiBit/s link are your transfer rates via ftp about 5MiByte/s? In both directions? If so, your network is fine.

Now confess, what OS is your NFS server and client running.

Re: NFS Mount write issue...

jmckinzie — Tue, 18 Jul 2006 22:55:51 GMT

I am running HPUX 11.11 on my client however, the server is a network applicance host.

It is not an HP host however, I have several other HP hosts mounted to it via NFS.

Any ideas on what to try?

I can't do the FTP thing because we do not have ftp enabled on any of our networks.

SFTP is not enabled either.

Re: NFS Mount write issue...

jmckinzie — Wed, 19 Jul 2006 09:51:58 GMT

Anyone have any other ideas to try on this?

-TIA

Re: NFS Mount write issue...

OldSchool — Wed, 19 Jul 2006 10:05:16 GMT

can you export the folder in question to one of the other HPs? If so, can you duplicate the problem from one of them?

On the NFS server: any quotas or disk space issues?

Re: NFS Mount write issue...

A. Clay Stephenson — Wed, 19 Jul 2006 10:14:04 GMT

Have you looked for and applied the latest NFS related patches? If so, which ones?

Post the output of nfsstat -c and nfsstat -m.

Re: NFS Mount write issue...

jmckinzie — Wed, 19 Jul 2006 11:14:55 GMT

I cannot recreate the problem and I have other clients mounted to the exact same directory, which eliminates the NFS server...
Here is the other information requested.

NFS patches:

swlist -l product | grep -i nfs
NFS B.11.11 ONC/NFS; Network-File System,Information Services,Utilities
PHKL_25238 1.0 11.00 NFS nfsd deadlock
PHKL_25993 1.0 thread nostop for NFS, rlimit, Ufalloc fix
PHKL_28185 1.0 Tunable;vxportal;vx_maxlink;DMAPI NFS hang
PHKL_29335 1.0 vx_nospace on NFS write.
PHKL_30151 1.0 NFS binary overwrite hang
PHNE_32477 1.0 ONC/NFS General Release/Performance Patch
# nfsstat -c
##################################
Client rpc:
Connection oriented:
calls badcalls badxids
22761 0 0
timeouts newcreds badverfs
0 0 0
timers cantconn nomem
0 0 0
interrupts
0
Connectionless oriented:
calls badcalls retrans
636 402 1610
badxids timeouts waits
0 2012 0
newcreds badverfs timers
0 0 17
toobig nomem cantsend
0 0 0
bufulocks
0

Client nfs:
calls badcalls clgets
22875 6 22875
cltoomany
0
Version 2: (0 calls)
null getattr setattr
0 0% 0 0% 0 0%
root lookup readlink
0 0% 0 0% 0 0%
read wrcache write
0 0% 0 0% 0 0%
create remove rename
0 0% 0 0% 0 0%
link symlink mkdir
0 0% 0 0% 0 0%
rmdir readdir statfs
0 0% 0 0% 0 0%
Version 3: (22875 calls)
null getattr setattr
0 0% 4815 21% 1140 4%
lookup access readlink
3623 15% 1023 4% 19 0%
read write create
8123 35% 2506 10% 87 0%
mkdir symlink mknod
0 0% 0 0% 0 0%
remove rmdir rename
71 0% 0 0% 33 0%
link readdir readdir+
1 0% 13 0% 1131 4%
fsstat fsinfo pathconf
261 1% 29 0% 0 0%
commit
0 0%

#####################################

nfsstat -m

##########################
# nfsstat -m
/pvcstst from nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11 (Addr 169.163.247.219)
Flags: vers=3,proto=udp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
Lookups: srtt= 44 (110ms), dev= 17 ( 85ms), cur= 14 (280ms)
Reads: srtt= 17 ( 42ms), dev= 8 ( 40ms), cur= 6 (120ms)
Writes: srtt= 3 ( 7ms), dev= 3 ( 15ms), cur= 1 ( 20ms)
All: srtt= 44 (110ms), dev= 17 ( 85ms), cur= 14 (280ms)

/prod from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/prod (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/dv from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/dv (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/var/opt/cae from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/caeinstall (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/var/opt/caelog from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/caelog (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/var/mail from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/mail (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/etc/opt/cae from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/caedata (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home91 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home91 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home71 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home71 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home11 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home11 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home04 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home04 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home03 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home03 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/home02 from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/home02 (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/opt/lsf from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/prod/lsf (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/sysadm from nastpa05v01cfc:/vol/nastpa05v01cfc_v1/sysadm (Addr 161.127.237.22)
Flags: vers=3,proto=tcp,auth=unix,soft,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
All: srtt= 0 ( 0ms), dev= 0 ( 0ms), cur= 0 ( 0ms)

/test2 from nasfsa01.fsa.citicorp.com:/vol/fcrr_v1/pvcs11 (Addr 169.163.247.219)
Flags: vers=3,proto=udp,auth=unix,hard,intr,link,symlink,devs,rsize=32768,wsize=32768,retrans=5
Lookups: srtt= 25 ( 62ms), dev= 3 ( 15ms), cur= 4 ( 80ms)
Writes: srtt= 4 ( 10ms), dev= 4 ( 20ms), cur= 2 ( 40ms)
All: srtt= 26 ( 65ms), dev= 3 ( 15ms), cur= 4 ( 80ms)

Thanks,

Re: NFS Mount write issue...

Dave Olker — Wed, 19 Jul 2006 11:40:34 GMT

Hi Jody,

It looks like the mount point you're having problems with is UDP using 32K read/write sizes. Have you tried using TCP for this mount point? Have you tried using a small rsize/wsize value?

I'd first try using TCP to see if this is a UDP-related problem. I'd then try mounting the filesystem with -rsize=512 and -wsize=512 to see if using a very small read/write request size allows this to work. If either of those suggestions work then it points to a network problem similar to what Clay was pointing to.

You can also check the lanadmin statistics of the interface card on the client to see if any errors are being logged. I would also like to see the "netstat -s" output on the client to see if it's logging any errors at the transport layer.

Regards,

Dave

Re: NFS Mount write issue...

A. Clay Stephenson — Wed, 19 Jul 2006 12:30:58 GMT

I agree with Dave, try using the NFS mount options proto=tcp,rsize=512,wsize=512.

If this fixes the problem then your network connection is suspect. Have you checked the speed/duplex settings on both your HP-UX client and the corresponding switch port?
Unless this is a Gigabit/s connection, I would hard-set both ends of the connection.
Surprisingly, if one end is running half-duplex and the other end is running full-duplex, things will almost work well and chances are, you wouldn't even notice a problem using telnet, for example.

Re: NFS Mount write issue...

Dave Olker — Thu, 20 Jul 2006 00:08:46 GMT

Hi Jody,

I realize you've closed this thread but I just wanted to comment. The fact that lowering the rsize/wsize to 1024 caused things to work is not really a solution - it just highlights the fact that you've got an underlying network problem in your environment.

NFS should be able to work just fine with 32K read/write sizes - in fact your nfsstat output shows every other NFS mount on this client was using 32K read/write sizes and apparently working just fine.

Are you now using 1024 rsize/wsize with UDP or TCP? There are known problems with certain network interface cards and UDP traffic. Did you switch to TCP or just drop the rsize/wsize values? Just curious.

Again, I'm glad your system is working, but I really think you're only hiding the problem by dropping the rsize/wsize as a permanent solution. The small rsize/wsize was a good proof point, and it's a test I've used numerous times with customers, but I urge you to investigate your networking layer a bit closer before you consider this issue resolved. Otherwise you're bound to run into a new problem with some other network application down the road caused by the same underlying problem.

Regards,

Dave