NFS File locking does not work correct.

[RolandH]

Hi Forumers,

I have several HP-UX 11.11 WS. All have the same configurationm (same patch revision).
I test the file locking with a perl script. On some systems it works on some not. Does not know why. If I lock a local file it works. If I try to do it over a nfs mount it fails.
On one system I have installed the newest nfs patches. It fails again.
If I look in the nfsstats (nfsstat -c) I see that the badcalls and badixds is increasing (Connectionless oriented = udp).

Where could be the problem? Need help - I'm frustrated !!!

Roland
PS: Have to go now will be back tomorrow

[Massimo Bianchi]

Hi Roland,
check if on the problematic server rpc.lock is properly working and restart it if in doubt.

I had some issue when cluster software was installed, because we needed the patest patch to have it proper working.

Massimo

[A. Clay Stephenson]

First, you should de a man perlfunc and examine the flock section. It specifically tells you that some versions of flock cannot lock things over the network. You can use Perl's fcntl function for more robust locking.

Next, make sure that lockd and statd are running on the boxes in question.

Finally, examine my 2 minute Perl script that does work under Perl 5.6.1 on (and across NFS) on HP-UX 10.20, 11.0, and 11.11 using flock().
Mine works without changes even across NFS mounts because I'm using an automounted home directory but you can simply change the filename if using hard NFS mounts. Note that for locking to work you must play by the rules and make sure that the file is opened for
writing (or update) even if you are only reading from the file. That's the reason for my opening the file with '+<' eventhough I am simply reading from the file.

[RolandH]

To Massimo:

I'm not in doubt that rpc.lock daemon is running. As I said before: Same systems with same patch revision. One work, one not.
I've checked this. Can I debug the rpc.lock daemon?

To Clay:
See my Perl script. I think I have made everythink fine and I use fcntl.
But I use this only for testing, because the main application is a binary and this binary locks a file. If the perl script works the application can start (means nfs locking is functional).

Thanx Guys! Any other ideas!!!
(Slowly, I think it could be a network thing)


Roland

[RolandH]

To Massimo:

What patch have you installed?

Roland

[Massimo Bianchi]

Hi,
to enalbe error logging:

lockd recognizes the following options and command-line arguments:

-l log_file Log any errors to the named log file
log_file. Errors are not logged if the -l
option is not specified.

Information logged to the file includes date
and time of the error, host name, process ID
and name of the function generating the
error, and the error message.


"quote from man rpc.lockd"


for the patch: we had to install the ONC/NFS patch, latest is

PHNE_28567

and requires a reboot.

Massimo

[RolandH]

Have had installed the patch. No change !! The problem is still there.
The log of the rpc.lockd shows no errors. ???

Any other ideas !!

Roland

[Massimo Bianchi]

How are the nfs mount done ?

via mount_nfs or via automounter ?

in the latter: automountd (V2) or autofs (V3)?

Massimo

[RolandH]

autofs(V3) !!!

[Todd Whitcher · Accepted solution]

Hi Roland,

99 % of the time locking fails due to name resolution issues. The name server either can not resolve the hostname of the client, it has a bogus record for that client ( or other clients ) in its sm or sm.bak directory or there is a issue w/ shortname vs. FQDN names being resovled.

Attached is a file locking program you can use to test outside of your application.

cc -o locktest locktest.c

Then

./locktest /nfs_mount/file_name
Open OK
Locking...OK

If you do not see the "OK" after The "Locking..." then you have
* a file locking problem. The typical symptom would be that the program
* will not complete, but will just hang afetr the "Locking...".


So now what:

An easy step to test clear up bad host records is this:

On the NFS server and the NFS client:
1.
kill rpc.statd then kill rpc.lockd on both systems.

2. Remove the /var/statmon/sm and /var/statmon/sm.bak directories

3. Restart rpc.statd. and rpc.lockd ( statd is always started first ) on both systems

/usr/sbin/rpc.statd
/usr/sbin/rpc.lockd

( wait 60 seconds for the rpc.lockd grace period )

Test the locks using the locktest program.

If the locks fail, get a network trace along with lockd / statd debug on the NFS SERVER and the NFS CLIENT


Nettl commands:
nettl -tn 0x32800000 -e ns_ls_ip -s 512 -tm 900000 -f lcktrc

Then send rpc.statd and rpc.lockd the SIGUSR2 signal to toggle debug on.

Kill -17 pid_of_rpc.statd
kill -17 pid_of_rpc.lockd

( debug goes to /var/adm/rpc.statd.log rpc.lockd.log )


You can view the nettl traces with netfmt or view them with ethereal ( www.ethereal.com )
Ethereal is more user friendly.

For netfmt do this:

Create a filter file with the IP of the remote system in it ( on nfs server, put the nfs client and visa versa )

$ cat filter
filter ip_saddr ip_of_remote
filter ip_daddr ip_of_remote

format with netfmt

$ netfmt -Nnlf lcktrc.TRCXXX -c filter > lcktrc.out

You can vi the lcktrc.out file.

If you still have issues, I'd suggest logging a case via the Response Center.

Hope this helps,

Todd

[Todd Whitcher]

Hi Roland,

Correction in my first paragraph, NFS server not NAME server.

Hi Roland,

99 % of the time locking fails due to name resolution issues. The NFS SERVER either can not resolve the hostname of the client, it has a bogus record for that client ( or other clients ) in its sm or sm.bak directory or there is a issue w/ shortname vs. FQDN names being resovled.

[RolandH]

Hi Todd,

thanx for your reply.
We see a light at the end of the tunnel.
Sadly, the server is not in our responsibilty but the server team has a maintanance time at the weekend and they will do remove the /var/statmon/sm* dirs then and reboot the system. So we will be smarter at monday.

I will give you points, if we are sure that is solving the problem.


Regards
Roland

[Todd Whitcher]

Hi Roland,

Also, verify name resolution for those clients is working as you expect. Be sure to do forward and reverse lookup checks w/ nslookup.

[RolandH]

Hi Todd,

maintanace was a little bit earlier as expected.

Thanx a lot for your reply. It was a name resolution issue on the remote site. They have had wrong entries there. !!!! Damn !!!!

But a small note to your c programm. You have forgotten the unlocking function =:-}.

Have a nice day
and 10 points for ***YOUR MAGIC ANSWER***


Roland

[Todd Whitcher]

Good News,

We have a few of those locking programs around, I think that one just hangs if it can't get a lock.

Todd