topic Re: NFS File locking does not work correct. in Operating System - HP-UX

NFS File locking does not work correct.

RolandH — Thu, 21 Aug 2003 15:02:13 GMT

Hi Forumers,

I have several HP-UX 11.11 WS. All have the same configurationm (same patch revision).
I test the file locking with a perl script. On some systems it works on some not. Does not know why. If I lock a local file it works. If I try to do it over a nfs mount it fails.
On one system I have installed the newest nfs patches. It fails again.
If I look in the nfsstats (nfsstat -c) I see that the badcalls and badixds is increasing (Connectionless oriented = udp).

Where could be the problem? Need help - I'm frustrated !!!

Roland
PS: Have to go now will be back tomorrow

Re: NFS File locking does not work correct.

Massimo Bianchi — Thu, 21 Aug 2003 15:29:33 GMT

Hi Roland,
check if on the problematic server rpc.lock is properly working and restart it if in doubt.

I had some issue when cluster software was installed, because we needed the patest patch to have it proper working.

Massimo

Re: NFS File locking does not work correct.

A. Clay Stephenson — Thu, 21 Aug 2003 17:25:10 GMT

First, you should de a man perlfunc and examine the flock section. It specifically tells you that some versions of flock cannot lock things over the network. You can use Perl's fcntl function for more robust locking.

Next, make sure that lockd and statd are running on the boxes in question.

Finally, examine my 2 minute Perl script that does work under Perl 5.6.1 on (and across NFS) on HP-UX 10.20, 11.0, and 11.11 using flock().
Mine works without changes even across NFS mounts because I'm using an automounted home directory but you can simply change the filename if using hard NFS mounts. Note that for locking to work you must play by the rules and make sure that the file is opened for
writing (or update) even if you are only reading from the file. That's the reason for my opening the file with '+<' eventhough I am simply reading from the file.

Re: NFS File locking does not work correct.

RolandH — Fri, 22 Aug 2003 06:32:24 GMT

To Massimo:

I'm not in doubt that rpc.lock daemon is running. As I said before: Same systems with same patch revision. One work, one not.
I've checked this. Can I debug the rpc.lock daemon?

To Clay:
See my Perl script. I think I have made everythink fine and I use fcntl.
But I use this only for testing, because the main application is a binary and this binary locks a file. If the perl script works the application can start (means nfs locking is functional).

Thanx Guys! Any other ideas!!!
(Slowly, I think it could be a network thing)

Roland

Re: NFS File locking does not work correct.

RolandH — Fri, 22 Aug 2003 06:53:42 GMT

To Massimo:

What patch have you installed?

Roland

Re: NFS File locking does not work correct.

Massimo Bianchi — Fri, 22 Aug 2003 07:24:32 GMT

Hi,
to enalbe error logging:

lockd recognizes the following options and command-line arguments:

-l log_file Log any errors to the named log file
log_file. Errors are not logged if the -l
option is not specified.

Information logged to the file includes date
and time of the error, host name, process ID
and name of the function generating the
error, and the error message.

"quote from man rpc.lockd"

for the patch: we had to install the ONC/NFS patch, latest is

PHNE_28567

and requires a reboot.

Massimo

Re: NFS File locking does not work correct.

RolandH — Wed, 27 Aug 2003 10:50:08 GMT

Have had installed the patch. No change !! The problem is still there.
The log of the rpc.lockd shows no errors. ???

Any other ideas !!

Roland

Re: NFS File locking does not work correct.

Massimo Bianchi — Wed, 27 Aug 2003 10:55:44 GMT

How are the nfs mount done ?

via mount_nfs or via automounter ?

in the latter: automountd (V2) or autofs (V3)?

Massimo

Re: NFS File locking does not work correct.

RolandH — Wed, 27 Aug 2003 11:07:12 GMT

autofs(V3) !!!

Re: NFS File locking does not work correct.

Todd Whitcher — Wed, 27 Aug 2003 11:13:44 GMT

Hi Roland,

99 % of the time locking fails due to name resolution issues. The name server either can not resolve the hostname of the client, it has a bogus record for that client ( or other clients ) in its sm or sm.bak directory or there is a issue w/ shortname vs. FQDN names being resovled.

Attached is a file locking program you can use to test outside of your application.

cc -o locktest locktest.c

Then

./locktest /nfs_mount/file_name
Open OK
Locking...OK

If you do not see the "OK" after The "Locking..." then you have
* a file locking problem. The typical symptom would be that the program
* will not complete, but will just hang afetr the "Locking...".

So now what:

An easy step to test clear up bad host records is this:

On the NFS server and the NFS client:
1.
kill rpc.statd then kill rpc.lockd on both systems.

2. Remove the /var/statmon/sm and /var/statmon/sm.bak directories

3. Restart rpc.statd. and rpc.lockd ( statd is always started first ) on both systems

/usr/sbin/rpc.statd
/usr/sbin/rpc.lockd

( wait 60 seconds for the rpc.lockd grace period )

Test the locks using the locktest program.

If the locks fail, get a network trace along with lockd / statd debug on the NFS SERVER and the NFS CLIENT

Nettl commands:
nettl -tn 0x32800000 -e ns_ls_ip -s 512 -tm 900000 -f lcktrc

Then send rpc.statd and rpc.lockd the SIGUSR2 signal to toggle debug on.

Kill -17 pid_of_rpc.statd
kill -17 pid_of_rpc.lockd

( debug goes to /var/adm/rpc.statd.log rpc.lockd.log )

You can view the nettl traces with netfmt or view them with ethereal ( www.ethereal.com )
Ethereal is more user friendly.

For netfmt do this:

Create a filter file with the IP of the remote system in it ( on nfs server, put the nfs client and visa versa )

$ cat filter
filter ip_saddr ip_of_remote
filter ip_daddr ip_of_remote

format with netfmt

$ netfmt -Nnlf lcktrc.TRCXXX -c filter > lcktrc.out

You can vi the lcktrc.out file.

If you still have issues, I'd suggest logging a case via the Response Center.

Hope this helps,

Todd

Re: NFS File locking does not work correct.

Todd Whitcher — Wed, 27 Aug 2003 11:17:17 GMT

Hi Roland,

Correction in my first paragraph, NFS server not NAME server.

Hi Roland,

99 % of the time locking fails due to name resolution issues. The NFS SERVER either can not resolve the hostname of the client, it has a bogus record for that client ( or other clients ) in its sm or sm.bak directory or there is a issue w/ shortname vs. FQDN names being resovled.

Re: NFS File locking does not work correct.

RolandH — Wed, 27 Aug 2003 12:32:32 GMT

Hi Todd,

thanx for your reply.
We see a light at the end of the tunnel.
Sadly, the server is not in our responsibilty but the server team has a maintanance time at the weekend and they will do remove the /var/statmon/sm* dirs then and reboot the system. So we will be smarter at monday.

I will give you points, if we are sure that is solving the problem.

Regards
Roland

Re: NFS File locking does not work correct.

Todd Whitcher — Wed, 27 Aug 2003 12:39:37 GMT

Hi Roland,

Also, verify name resolution for those clients is working as you expect. Be sure to do forward and reverse lookup checks w/ nslookup.

Re: NFS File locking does not work correct.

RolandH — Thu, 28 Aug 2003 06:58:02 GMT

Hi Todd,

maintanace was a little bit earlier as expected.

Thanx a lot for your reply. It was a name resolution issue on the remote site. They have had wrong entries there. !!!! Damn !!!!

But a small note to your c programm. You have forgotten the unlocking function =:-}.

Have a nice day
and 10 points for ***YOUR MAGIC ANSWER***

Roland

Re: NFS File locking does not work correct.

Todd Whitcher — Thu, 28 Aug 2003 10:43:02 GMT

Good News,

We have a few of those locking programs around, I think that one just hangs if it can't get a lock.

Todd