HPUX/LINUX NFS flie lock problem

[clarkem]

I have a linux (Suse 10.1) NAS (Networked Attached Storage) pc, with an HP j5000 workstation running HPUX 10.20 mounting a directory on the NAS, via NFS. I can mount the linux directory from the j5000, using the automounter. Everthing looks ok, except that file locking doesn't work - I get the error -fcntl: No locks available.

Using direct NFS and autofs mounts results in the file lock hanging up.

This is also the case with hpux11.11. I've tried the standard fix, adding the insecure_locks directive to the linux export arguments, but no joy.

The problem is only hpux mounting a linux directory, file locking is ok in all other permutations of linux-linux, hpux-hpux and linux-hpux.

Any help/tips appreciated.

[Steven E. Protter]

Shalom,

Please check the Suse device. It is probably using NFS version 4.

The locking mechanism's available are not available to earlier nfs clients.

As far as I recall, the latest nfs client available for 11i v1 is Version 3.

You are probably going to need to configure the Linux device to support Version 3 clients. The 10.20 box, which is out of support needs to use NFS version 3,not 2.

SEP

[clarkem]

SEP - thanks for the reply -examining etc/sysconfig/nfs, the SUSE box is certainly running NFSv4. However, the novell site states:

"The NFSv3 and NFSv4 protocols are not compatible. A NFSv4 client cannot access a NFSv3 server, and vice versa. However, in order to simplify migrations from NFSv3 to NFSv4, both NFSv3 and NFSv4 services are launched by the command: rpc.nfsd."

So, if novell are correct, the suse box must be utilising V3 in order to talk to the HPUX box.

[Peter Godron]

Hi,
if the linus lockd is coded like Solaris you may have a HP Coding problem:
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1052530

[clarkem]

Thanks Peter - forgot to mention I added the "-C" parameter already. No difference.

[Dave Olker]

Hi Clark,

This problem should have nothing to do with the version of NFS your running or the -C option.

NFS clients and servers always negotiate to the highest supported protocol version available on both client and server so even if a server supported versions 2,3,4 and the client only supported version 2 they would still negotiate down to 2.

The -C option is only a factor in rare cases involving blocked record locks between multiple processes where one process is trying to cancel a pending blocked lock. If locking is not working at all then that isn't the problem.

The fact that you're getting an ENOLCKS error tells me it could be a couple of different possibilities:

1) rpc.lockd/rpc.statd are not running on both systems

2) Your local kernel file lock table is full (defaults to 200, so can fill up if there are many applications running on the HP-UX system holding file locks)

3) There is a configuration problem on the Suse box that is simply denying the lock completely.


The best data you could collect for this problem would be a debug rpc.lockd log file on the HP-UX box and an Ethereal trace taken on the client or server during the failed lock request.

To start debug rpc.lockd logging on the HP-UX system, simply send a SIGUSR2 signal (i.e. kill -17 ) to the running rpc.lockd process. This will toggle debug on and start logging to the /var/adm/rpc.lockd.log file.

While debug logging is running, start an ethereal network trace on either the client or server and then reproduce the problem. The debug rpc.lockd log file and the network trace will hopefully tell us why the lock is failing, or at least give us ideas of where to look next.

Regards,

Dave

[TwoProc]

I'm thinking that you're out of "nflocks" on your NAS server. Try increasing that and see if it goes away.

[rick jones]

I cannot vouch for this completely, but a recent, similar situation went past my eyeballs. The first step was to make sure that nfs-utils-1.0.6-114 was installed. That may or may not be the correct version in your case. This person found it at rpm.pbone.net.

Next, it seems that it was necessary to:


And one last issue, for those following the story. It turns out that
SLES NFS is a bit more security conscious than the average NFS server.
The "no_root_squash" and "no_auth_nlm" settings were easy to identify,
and they made most of the applications behave properly.

However, even with those settings, attempting to lock a file within a
directory that was not universally accessible (say, with permissions
770) failed. The last required setting is "no_subtree_check".


Now, this was for a SLES 9 server, but perhaps the same principles will apply.

BTW, did we (HP) even port NFS v3 back to 10.20? I thought that stopped at v2, but then it is so old that my DIMM memory could easily have bitrotted on that one.

[Dave Olker]

Hi Rick,

> BTW, did we (HP) even port NFS v3 back to
> 10.20? I thought that stopped at v2, but
> then it is so old that my DIMM memory
> could easily have bitrotted on that one.


Yes, we did port NFS V3, AutoFS and CacheFS back to 10.20. This was known as the infamous 10.20 PI release. I believe it was April 1997. I'll never forget it as it was one of the longest years of my life in HP support.

Dave

[Yogeeraj_1]

hi,

before you make any changes concerning the *lock parameters, i would suggest that you monitor these values using glance (SYSTEM TABLES REPORT)

e.g.
System Table Available Used Utilization High(%)
--------------------------------------------------------------------------------
Proc Table (nproc) 4096 807 20 20
File Table (nfile) 63498 8398 13 13
Shared Mem Table (shmmni) 512 13 3 3
Message Table (msgmni) 4096 2 0 0
Semaphore Table (semmni) 4096 27 1 1
File Locks (nflocks) 4096 95 2 2
Pseudo Terminals (npty) 60 0 0 0
Buffer Headers (nbuf) na 76704 na na

Did you try another flavour of linux? E.g. Red hat

We had no problems mount redhat as 4 file systems on our hp-ux 11.11

kind regards
yogeeraj

[clarkem]

I should mention that locks work fine from hpux-hpux, linux-linux and linux-hpux. Only hpux mounting linux seems to go wrong. Don't think there is a lock shortage, as both machines are lightly loaded, and 3/4 possible permutations work. Started debug logging, and tried the lock. Got the following attachment.

Added (rw,sync,insecure_locks,insecure,no_root_squash,no_auth_nlm,no_subtree_
check) to mount options - still no good.

[Yogeeraj_1]

hi,

have a look at this thread:
https://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1052898

do you have patch PHNE_34293 installed?

kind regards
yogeeraj

[Dave Olker]

Hi Clark,

Thanks for collecting the trace. It shows (at least to me) that the problem has to be on the Suse side. The HP-UX system is generating a proper lock request, sending it over the wire, and waiting for a response from the server. In other words, it's doing everything right.

The reply from the Suse box shows it sending back a status of 2 to the lock request, meaning the Suse box is rejecting the lock request with ENOLCKS. If the box is lightly loaded and there are no resource problems as you say, then I have to imagine this is some kind of configuration issue on the Suse side.

Just to clarify - you said you added a bunch of options to the mount options. Do you really mean "export" options? I assume you made these changes on the server in how it exports or shares the filesystem, not on the client when it tries to mount the filesystem.

I wish I knew more about each Linux vendor's NFS implementation but I don't know which are the best export options for a Suse server to allow locks from other clients. Perhaps a search of the Suse knowledgebase (assuming one exists) or even a call to the Suse support folks will help identify the proper configuration on the Suse system to allow locks to work from non-Linux clients.

Regards,

Dave

[clarkem]

Thanks for the patch suggestion. The problem is that any fix I have must work with HPUX versions 10.20 to 11.11 - and the patch suggested is for 11.11. I am aware that 10.20 is no longer supported by HP, but we need to keep it running on a box, so that we can compile software and make depots for a product that runs on 10.20.


I suspect I am going to have to find a work-round.

[clarkem]

Dave,

That was more or less the impression that I got - and confirmed by the log - that the NAS was rejecting the HP request. Hence the exports options on the NAS, as I think the NAS is objecting to some peculiarity of HP syntax. I'll reiterate that locks work in every direction and O/S combination but HPUX mounting on Linux, so I dont't think the NAS is running out of resources, just rejecting the call.

I think that if there is a aolution, it will be in making the NAS co-operate with HPUX, given that I need it to work for 10.20 - 11.11.

[Steven E. Protter]

Shalom again,

You have a tough job, with 10.20 a few years out of support and 11.00 reaching end of support this year.

HP has made life tougher because NFS v4 is only releasesd for 11i v2.

So long as the 11.11 box is not controlling access to the NAS device, patching it should not have an adverse effect on the older boxes.

SEP

[Dave Olker]

Steven,

Please, please stop posting that NFS V4 is available on HP-UX 11i v2. It's not. We'll be releasing NFS V4 in 11i v3 (11.31). I've posted this information in numerous threads and yet I keep seeing you post that V4 is available. Please stop posting this.

Thanks,

Dave

[Dave Olker]

Hi Clark,

I spent much of the weekend trying to duplicate your problem. I'm not sure I duplicated it exactly because I'm not getting an ENOLCKS error, I'm getting an EPERM error.

I had a SuSE 9.2 server in my datacenter and I configured it as an NFS server. I took one of my HP-UX file locking utilities and ported it to Linux so that I could try locking from both HP-UX and Linux clients.

What I found was that locking worked just fine from the Linux client but failed when originating from an HP-UX client. I tried various combinations of exports options and finally found the one that allowed my configuration to work - insecure.

Unfortunately your previous post indicates you already tried that option:

> (rw,sync,insecure_locks,insecure,
> no_root_squash,no_auth_nlm,
> no_subtree_check) to mount options -
> still no good.


Are you absolutely certain you not only updated the /etc/exports file with these options but that you re-exported the filesystem with these options?

On my server I used the following options in my /etc/exports file:

/test *(rw,insecure,insecure_locks,no_subtree_check)

and then exported the filesystem using the syntax:

# exportfs -rafv

This syntax wipes out the current kernel export table and replaces it with the contents of the /etc/exports file. I then verified the filesystem was exported properly by using the command:

# exportfs -v
/test (rw,wdelay,insecure,root_squash,no_subtree_check,insecure_locks,anonuid=65534,anongid=65534)

I can see all the export options in the list so I know they are in effect. Can you please verify that your filesystem is exported using the above options?

Regards,

Dave

[clarkem]

Dave, thanks for your efforts. I'm not at work until Wednesday, so I can't use your latest suggestions about export options. I have made manual changes and used exportfs -av to impose the options, and have also tried Yast, which is supposed to do this for you.


But I'll give it a go, anyway. I gather the insecure_locks option allows the use of unexpected port numbers (possibly among other things).

You say you had a permission problem - so, in debug mode, the return from your linux box was different?

[Dave Olker]

Hi Clark,

Actually, it was the "insecure" option that was blocking my environment from locking properly. It appears HP clients send lock requests over-the-wire from ports above 1024, which means you need to set the "insecure" export option to allow the Linux server to process those messages.

The debug trace file showed a reply status of 1 instead of 2 as in your case. Assuming you were doing everything right, it's likely that the problem I fixed in my environment is different from yours. Your NFS server is reporting "No Locks", which usually means the kernel lock table is full. However, I'm sure there are other cases in the rpc.lockd code on SuSE where it will return an ENOLCKS for different reasons other than a full lock table.

If the export options don't resolve the problem, and no one else in these forums are able to help, you might want to place a call with SuSE and have them research all the reasons why their rpc.lockd would return an ENOLCKS error.

The only other suggestions I can offer are:

1) Try posting this problem in the ITRC Linux forums as well as HP-UX

2) Try downloading, compiling and installing the latest version of the nfs-utils package from Sourceforge.

I did this on my server just to make sure I wasn't seeing a known and resolved problem. I had to compile the nfs-utils package with several configure flags to get it to work, but I eventually got the latest NFS package running on the server. You might want to give this a try as well.

Regards,

Dave

[clarkem]

The update to latest nfstools plus all "insecure" modifiers seems to make file locks work, but only if the file permissions (and the directory containing the file) are read/write for all. Swpackage still does not work, and tricking it with a specially prepared depot gives ACL errors later in the depot build.

I've decided to use the work-round, copy all the stuff to the HP box local disk, make the depot, then copy it back to the NAS disk. Depots available as flat-file only.