topic Re: NFS lockd issue 11iv3 & redhat linux AS in Operating System - HP-UX

NFS lockd issue 11iv3 & redhat linux AS

wvsa — Mon, 30 Aug 2010 19:42:12 GMT

Good afternoon all;

Having a strange problem with NFS. We have a rx6600 mounting two directories to a redhat linux server. Here are the dfstab mount options:

share -F nfs -o sec=sys,anon=0 /nsp_mart
share -F nfs -o sec=sys,anon=0 /sas_bi

On the redhat linux box we mount the directories as follows:

romans02:/nsp_mart /nsp_mart nfs bg,soft 0 0

romans02:/sas_bi /sas_bi nfs bg,soft 0 0

With these options I can log on as a user and read and write a file under the /nsp_mart directory.

However in running a application (SAS) the application hangs when it attempts to open or write a file to directories under /nsp_mart.

The SAS application will work if it is run in the following manner:

sas -filelocks none

Should mention that SAS is run from the redhat linux server and is attempting to read and write files to the /nsp_mart directory. At any rate when the -filelocks none option is used all is well SAS writes and reads from the /nsp_mart directory.

My question is this is there a problem with lockd on the hpux 11iv3 server?

I have cleared the locks using the clear_lock command Have kctune klm_log_level=9 to see if there are any errors in syslog, no messages appear in syslog.

Not sure how to proceed, see that lockd is on port 4045 and running lsof -i -P have never seen this port being used.

Not sure what is causing this problem would greatly appreciate any input.

Thank you!

Norm

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Tue, 31 Aug 2010 00:07:01 GMT

Hi Norm,

So are you saying that you enabled debug KLM logging and reproduced the hang but never saw any messages related to KLM? That tells me this may not be a locking problem. Have you tried collecting a network trace while reproducing the problem? I'd suggest collecting a nettl trace or Wireshark trace on the rx6600 while reproducing the problem. Once the hang occurs, let it hang for a few seconds then stop the trace. The trace will hopefully show what over-the-wire requests were happening when the hang happened.

Regards,

Dave

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Tue, 31 Aug 2010 01:45:35 GMT

Hello Dave;

Thank you for responding. Once I figure out how to get nettl trace and/or wireshark configured and running will let you know what I find.

Norm

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Tue, 31 Aug 2010 02:33:32 GMT

Here's what I'd suggest on the HP-UX NFS server:

1) Turn on nettl tracing:
# /usr/sbin/nettl -tn pduin pduout loopback -e ns_ls_ip -f

2) Reproduce the hanging application

3) Turn off nettl tracing:
# /usr/sbin/nettl -tf -e all

This will create one or two files using the name you supplied in step 1 and the suffix ".TRC000."

Once you have this file you can load it into Wireshark (yes, Wireshark can directly read HP-UX nettl format) and filter on the IP address of the client, or NFS packets, or KLM packets, etc.

Hope this helps,

Dave

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Tue, 31 Aug 2010 14:56:14 GMT

David;

Thank you for your responses. Will running trace this morning. Would you be willing to be another set of eyes and look at the trace. Will provide all the necessary background data.

Thanks again

Norm

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Tue, 31 Aug 2010 14:57:55 GMT

Sure Norm.

I'd want the *raw* trace files, not any formatted stuff. I'd also want the IP addresses of the NFS client and server. You can either post the trace files to this thread, or if you're not comfortable with that (I wouldn't blame you) you can send the trace files to me directly: dave.olker@hp.com.

Regards,

Dave

Re: NFS lockd issue 11iv3 & redhat linux AS

Duncan Edmonstone — Tue, 31 Aug 2010 15:39:36 GMT

Dave,

Once you have the emails from Norm, you might want to ask a forums admin (such as Melvyn) to edit your post and take out your email address before the dreaded spam-bots strike...

Duncan

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Tue, 31 Aug 2010 15:41:25 GMT

Hi Duncan,

I'm a moderator of the forums so I can edit any post. I appreciate your concern, but I have no problem listing my HP email here. I encourage HP customers to contact me, and many do. :)

Dave

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Wed, 01 Sep 2010 13:42:28 GMT

Hi Norm,

I looked at the first round of nettl traces you sent me and here's the concerning thing I see in the trace:

172.17.6.19 -> 172.17.6.29 Portmap V2 GETPORT Call NLM(100021) V:4 TCP
172.17.6.29 -> 172.17.6.19 Portmap V2 GETPORT Reply (Call In 27711) PROGRAM_NOT_AVAILABLE
172.17.6.19 -> 172.17.6.29 Portmap V2 GETPORT Call NLM(100021) V:4 TCP
172.17.6.29 -> 172.17.6.19 Portmap V2 GETPORT Reply (Call In 28515) PROGRAM_NOT_AVAILABLE
172.17.6.19 -> 172.17.6.29 Portmap V2 GETPORT Call NLM(100021) V:4 TCP
172.17.6.29 -> 172.17.6.19 Portmap V2 GETPORT Reply (Call In 28695) PROGRAM_NOT_AVAILABLE
172.17.6.19 -> 172.17.6.29 Portmap V2 GETPORT Call NLM(100021) V:4 TCP
172.17.6.29 -> 172.17.6.19 Portmap V2 GETPORT Reply (Call In 28720) PROGRAM_NOT_AVAILABLE
172.17.6.19 -> 172.17.6.29 Portmap V2 GETPORT Call NLM(100021) V:4 TCP
172.17.6.29 -> 172.17.6.19 Portmap V2 GETPORT Reply (Call In 28812) PROGRAM_NOT_AVAILABLE

The Linux client is attempting repeatedly to retrieve the port number of the NLM (Network Lock Manager) daemon running on the HP-UX system. The HP-UX box should reply with port number 4045. The fact that it doesn't reply with a port number tells me that rpc.lockd and rpc.statd may not be running on your 11i v3 system.

Can you verify that you have the LOCKMGR variable set to 1 in your /etc/rc.config.d/nfsconf file on the HP-UX system? Can you also confirm that the following command run on the HP-UX server returns the expected results:

# rpcinfo -t localhost 100021
program 100021 version 1 ready and waiting
program 100021 version 2 ready and waiting
program 100021 version 3 ready and waiting
program 100021 version 4 ready and waiting

You should get a "ready and waiting" reply for all 4 versions of program 100021, which is the Network Lock Manager. If you don't then we need to figure out why the lock manager is not getting started on your HP-UX system.

Regards,

Dave

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Wed, 01 Sep 2010 16:02:16 GMT

Hello Dave;

here is the info you requested:

#***************************************************************************
LOCKMGR=1
LOCKD_OPTIONS=""
STATD_OPTIONS=""

#***************************************************************************
# NFS client configuration variables
:q!
root@romans02:/etc/rc.config.d
# rpcinfo -t localhost 100021
rpcinfo: RPC: Program not registered
program 100021 is not available

The rpcinfo does not look good, so what is the next step?

Norm

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Wed, 01 Sep 2010 16:04:10 GMT

Try this on the 11.31 system:

# /sbin/init.d/lockmgr stop
# /sbin/init.d/lockmgr start

Then re-try the rpcinfo command and see if you get the appropriate output.

Dave

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Wed, 01 Sep 2010 16:10:31 GMT

Hello Dave;

Ok got the following:

# rpcinfo -t localhost 100021
program 100021 version 1 ready and waiting
program 100021 version 2 ready and waiting
program 100021 version 3 ready and waiting
program 100021 version 4 ready and waiting

Interesting, all of rx6600 servers have the same problem, looks like the administrator (me) is doing something consistantly wrong, how shocking.

So should I try the application now?

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Wed, 01 Sep 2010 16:11:53 GMT

Dave;

Any thoughts as to why lockd is not starting up? Guess the problem is in /etc/rc.config.d

Thank you for your help

Norm

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Wed, 01 Sep 2010 16:13:50 GMT

No, the problem isn't in the config file. The only parameter in the config file is LOCKMGR=1 and you already confirmed that.

Problem sounds like rpc.lockd stops responding to requests after running awhile. No idea why that's happening. That would require a separate action plan to figure out.

What version of ONCplus are you running on your 11.31 systems?

# swlist ONCplus

Dave

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Wed, 01 Sep 2010 16:19:18 GMT

Hi Dave;

# ONCplus B.11.31.09.01 ONC+ 2.3
ONCplus.NFS B.11.31.09.01 ONC/NFS; Network-File System,Information Services
,Utilities

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Wed, 01 Sep 2010 16:20:11 GMT

Ok, so you're on the latest stuff. I was gonna suggest updating to the latest stuff to see if you're hitting a known/fixed problem.

Does the application work now?

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Wed, 01 Sep 2010 16:21:47 GMT

Dave;

Is there anything I should do before testing the application again?

Re: NFS lockd issue 11iv3 & redhat linux AS

Dave Olker — Wed, 01 Sep 2010 16:21:58 GMT

No, please try the application

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Wed, 01 Sep 2010 16:50:22 GMT

Dave;

It works! yes indeed. It would be sure nice know why the lockmgr had stopped on all of our servers.

Thank you for your help!

Norm

Re: NFS lockd issue 11iv3 & redhat linux AS

wvsa — Wed, 01 Sep 2010 21:17:25 GMT

Hi Dave;

How should I go about finding out why lockmgr is not starting. It seems to be a problem on many of our hpux 11iv3 servers. Should we start a case or ?

Norm