NIS NFS Users window hangs when opened

[LV2DIVE]

Running NIS
Serving user's directory
login to CDE and open a window and it will hang before prompt.

Created a local user with local dir works fine.
Created a local user with a NFS shared directory same issue.

Thanks

[Peter Nikitka]

Hi,

obviously your NFS mounts are not without problems.
- Look for messages in the syslog
- try to access a NFS mounted directory from a local dtterm
- check if all NFS mounted devices are accessible and all NFS hosts are reachable

mfG Peter

[LV2DIVE]

I agree that there are issues, my brain is burnt out trying to resolve it. Needing fresh opinions.

NFS Server is a HP c8000 connected to a Complellent SAN Drive. Copies from the NFS directory to the local are very fast when logged in as root, but when logged in as a served user it is slow.

I am thinking there might be a kernal adjustment as a server for 50 Workstations and or additional nfsd may need to be running?

First time with a Compellent drive, use to NetApps.

[Dave Olker]

Hi Eric,

Let's back up a few steps.

What type system is the NFS client in this scenario? What OS is it running?

You said the NFS server is an HP c8000. What OS is it running?

Are you saying the problem occurs when you create a local user account on the NFS client whose home directory is NFS mounted, and that user login hangs prior to getting the shell prompt? If so, what shell is configured for this user?

I'm wondering if this isn't the age old issue of KSH hanging trying to lock a history file on the NFS server.

Regards,

Dave

[LV2DIVE]

>What type system is the NFS client in this scenario?

The system is a HP c8000.

>What OS is it running?

HP-UX-11.11

>You said the NFS server is an HP c8000. What OS is it running?

HP-UX 11.11

>Are you saying the problem occurs when you create a local user account on the NFS client whose home directory is NFS mounted, and that user login hangs prior to getting the shell prompt? If so, what shell is configured for this user?

Yes, and yes it is KSH.

>I'm wondering if this isn't the age old issue of KSH hanging trying to lock a history file on the NFS server.

Maybe, do tell more?

Thank you.

[Dave Olker · Accepted solution]

If this is the KSH history problem then what you're seeing is the inability of the NFS client and server to successfully lock files across the NFS mount. This is usually caused by a configuration problem on one of the two systems.

There are two ways to address this:

1) Cheat

By cheat, I mean don't resolve the underlying file locking issue but instead just redirect the KSH history file to a directory that is not NFS mounted. This is done by setting the HISTFILE environment variable in the user's .profile or .kshrc:

HISTFILE=/tmp/.sh_history.$$

This causes the shell to create and lock the history file locally instead of over NFS. Again, I consider this a hack/cheat because you're really not addressing the underlying NFS file locking problem, and if you use any other applications that lock files in the NFS mounted directories with this server they'll hang as well.

However, if the KSH history file is the *only* file locking performed between these two systems then you may not be interested or motivated to get to the bottom of the underlying problem and in that situation redirecting the history file is acceptable.

Even if you are interested in pursuing the underlying problem, setting the HISTFILE variable to a local directory is usually a fast and easy way of verifying that you're seeing an NFS file locking problem.


2) Troubleshoot the file locking problem

The easiest way to troubleshoot this problem is to collect debug rpc.lockd log files from both the client and server when reproducing the hang.

If you're interested in pursuing this let me know and I can walk you through it. I won't bother documenting the steps here until I know that you're actually having a file locking problem and that you're interested in pursuing it. If you're just going to use the HISTFILE variable to work around it then there's no point in collecting more data.

Regards,

Dave

[LV2DIVE]

Dave, thank you for the cheat. I added the variables to the user's .profile and it worked for su - user hang. The user still has an issue when logging into CDE. The window' prompts still hang.


I would like the instructions how to diagnose the rpc lock.

Seems like the big problem was cut in half though.

Thank you,
Eric

[Pete Randall]

Eric,

Add the same cheat to the user's .dtprofile.


Pete

[LV2DIVE]

Eureka! The user prompt lag has been eliminated!

Is there a better fix for HP-UX 11.11 then the cheat? I have about 40 machines with about 80 users.

The .profile is shared by all already.

[Dave Olker]

Hi Eric,

What this proves is your underlying problem is an NFS file locking issue. If you're still interested in resolving that issue I can give you a set of steps to get to the bottom of it. Again, if login is the only time you lock files over NFS then it might not be important to you.

Let me know if you still want to pursue identifying the underlying problem.

Regards,

Dave

[Peter Nikitka]

Hi,

you can setup this in /etc/profile, so you'll have to change only one file per host.

Create a file /etc/profile.local containing something like this:
LOGNAME=${LOGNAME:-`id -un`}
if [ "$LOGNAME" != root ]
then
HISTFILE=/tmp/.histfile_$LOGNAME
export HISTFILE
fi
You may add more site/host specific changes here.

Add an anchor to /etc/profile (plus other files where this change max be necessary):
[ -s /etc/profile.local ] && . /etc/profile.local

mfG Peter

[LV2DIVE]

Dave,

You are right, NFS was the issue due to the HISTFILE being generated on the NFS mount point.

The data directories and applications are also served.

Could this also affect these mounts speed too?

As always thank you,
Eric

[Dave Olker]

Eric,

It's extremely unlikely that the same problem that would stop NFS file locking to work would also affect throughput - unless you're talking about seeing an application hang. If an application is unable to lock a file over NFS it will likely hang, but that's usually different from application "performance".

If you're interested in figuring out the file lock hang here's what you can do:

1) Take an NFS client and remove the workaround previously identified (HISTFILE stuff) and make sure this client can reproduce the login hang.

2) On the NFS client and server systems issue this command:

# kill -17 $(ps -e | grep lockd | awk '{print $1}')

This will turn on debug rpc.lockd logging into the file /var/adm/rpc.lockd.log.

3) Reproduce the login hang

4) Repeat step #2 above where the rpc.lockd on the client and server are sent a kill -17 signal. This will turn off debug logging.

5) Post here or email the rpc.lockd.log files from the client and server systems to me and I'll analyze the logfiles to see why the locks are failing.


Usually I can determine the cause of the hang by looking at these log files. In some cases a network trace is also needed, but we can start with the debug logfiles.

Regards,

Dave

[LV2DIVE]

I will try this.

Thank you everyone for your input.

Thank you,
Eric Syversen

[Peter Nikitka]

Hi,

please pay attention to the uniq point system of this forum:
http://forums1.itrc.hp.com/service/forums/helptips.do?#33

mfG Peter

[Dennis Handly]

>Dave: Usually I can determine the cause of the by looking at these log files.

When we had the stops respondingit was because of not having the latest NFS patches. Of course that was from older OS releases, like 10.x too.

Typically until we got the right set of patches, we would have to kill off the rpc.lockd and rpc.statd (in a particular order) (and maybe do something on the server?). Then remove the right temp files in /var/statmon/.
Then restart them in a particular order.

[Dave Olker]

Hi Dennis,

What you're talking about is the case where locks hang because of a bug in rpc.lockd as opposed to a configuration problem. You're correct that *some* rpc.lockd defects did result in hangs, but that was about .001% of the cases I worked on.

The vast, vast majority of times I helped customers with a file locking problem it turned out to be a configuration issue, such as:

o DNS/NIS/hosts configuration bug
o Kernel Resource configured too low
o Network configuration problem
o Stale port information
o etc.

Again, a debug rpc.lockd log file will usually show very quickly why a particular NFS file lock failed. If the failure is caused by an rpc.lockd defect then patching the system is the obvious solution. However, I don't usually like asking customers to patch their system until I have some evidence that the patches will help them.

Regards,

Dave

[LV2DIVE]

Ok, I have applied the changes to the .dtprofile and the .profile and it cleared up the latency with the login windows. When I have time I want to diagnose the root cause.

I do not know if this is a related issue or enough for a new thread, but have a couple of users that cannot start an application from a NFS directory that everyone else can until I did the following:

1. Renamed the hostname of the machine
2. Gave the user a new name (but kept his UID the same) in /etc/password and changed it in /etc/group to reflect the passwd file.

[LV2DIVE]

The HISTFILE was the quick fix. When I have time, I will diagnose the reason why it did this.

Thank you for your support help.