Login Hangs - Need a real solution

[Dustin Black]

Hello all,
I have been doing some masive research on these forums about console and terminal login hangs when user home directories are nfs mounted. It has become apparent that this is a well-known, but not well documented problem. Everyone and their brother seems to have a "workaround" whereas you point the HISTFILE to a local directory and BOOM!.. problem fixed. Well, no.. problem worked-around. I have seen numerous references where people thought there were patches for the issue, but nobody seems to know or be able to find what those patches are. I have hunted and hunted through the patch database myself, and I can't seem to find it either.

If anyone has a clue as to an actual solution to this issue is, I'd love to hear it. My machines are running 11i, and the issue arises each time after I apply a series patches that are required for another bit of software.

Thanks!

[Dustin Black]

By the by... the HISTFILE thing won't even work in my case. The users in question don't even have reference to this in their profiles (sh shell).

[Victor BERRIDGE]

Hi Dustin,
Willing to help but dont quite understand your issue, cant you explain a bit more?
Is it get logged that take enormous time or screen (or terminal emulation) that seem to freeze or?


All the best
Victor

[Dustin Black]

After applying some patches (needed to run other software), users get a "hung" or "deadlocked" blank window with a blinking cursor when attempting to bring up a terminal window in CDE. Or, as root, when su - user, profile script runs but hangs at the end and gives only a blinking cursor. In either case, ^C will eventually give a command prompt.

[Steven E. Protter]

Applying patches should not trigger this problem.

Perhaps try a soft mount on the nfs /etc/fstab instead of a hard mount.

We have a bit of code in /etc/profile to direct HISTFILE where we want (which should not be an NFS mount).

HISTFILE=$HOME/.sh_history
export HISTFILE

That insures the HISTFILE is set properly. You can also limit its size the the HISTSIZE variable.

If patches are triggering this issue, check space on the /var filesystem.

Consider committing patches that are superseded:

cleanup -c 1

commits(removes files) patches superseded once or more.

SEP

[Dustin Black]

Steven:

I'm not sure what you mean by a soft mount vs. a hard mount.

There is no HISTFILE reference whatsoever in either /etc/profile or the user's .profile.

/var filesystem is at 25% usage.

Did the cleanup.

Problem persists...

Thanks!
-Dustin

[Victor BERRIDGE]

Are you in NFS?

my advice would be to start very simple. create a new user with a default .profile form /etc/skel.
And try its connection without fancy applications, can you type all commands without having to wait?
Do you have gpm installed?
In that case try your X emulation by launching gpm&

Is that still OK?
Then its must be something to do with that specific application that requires your patching, how does it work?


All the best
Victor

[Stephen Keane]

Dustin, can you post a typical .profile from a user that experiences the problem? Also you might want to run nfsstat to see if you have any NFS problems.

[Dustin Black]

Victor:

Created a new user and tested. Problem persists with new user.

GPM is on the NFS server, but not on the workstation in question.

[A. Clay Stephenson]

Login hangs - need real data.

We need to isolate your problem. Is it strictly an NFS issue or is it network? Automounter in use or explicit mounts? If automounting are the maps managed by NIS/NIS+?

To determine if the network itself might be the culprit, I would ftp about a 10MB file in both directions and see if the transfer speed is about as expected? Surprisingly, a mismatched duplex setting will almost work and might not even be obvious in a telnet session, for example.

The hard/soft mount is simply an nfs mount option; man mount_nfs for details.

[Dustin Black]

Stephen:

This is the complete .profile for one of the problem users:

# This is the default standard profile provided to a user.
# They are expected to edit it to meet their own needs.

MAIL=/usr/mail/${LOGNAME:?}

-----
And this is the complete .profile for the new test user created:

# @(#)B.11.11_LR

# Default user .profile file (/usr/bin/sh initialization).

# Set up the terminal:
if [ "$TERM" = "" ]
then
eval ` tset -s -Q -m ':?hp' `
else
eval ` tset -s -Q `
fi
stty erase "^H" kill "^U" intr "^C" eof "^D"
stty hupcl ixon ixoff
tabs

# Set up the search paths:
PATH=$PATH:.

# Set up the shell environment:
set -u
trap "echo 'logout'" 0

# Set up the shell variables:
EDITOR=vi
export EDITOR

-----

This is the only thing of concern from the nfsstat output:

Connectionless oriented:
calls badcalls retrans
5121 1171 35
badxids timeouts waits
1036 683 0
newcreds badverfs timers
0 0 0
toobig nomem cantsend
0 0 0
bufulocks
0

[Dustin Black]

Clay:

I can pretty much rule out the network issue. This same issue has happened on multiple client machines, always after applying the same set of patches.

Found something interesting. A previous system admin has apparently setup the mounts as BOTH nfs and automount. IE - I have an entry for the same mounts in both /etc/fstab and in /etc/auto.direct. Could this be causing my problem?

[Victor BERRIDGE]

Dustin,
With new user AND without launching anything?
I mean basic connection then just pure unix commands? (very important for understanding!)

What part is on NFS What is local?

Are both boxes on same network?
I there speed connection mismatch between the box and the switch its connected to?

If you are using NFS how is /etc/nsswitch.conf configured?

Good luck
Victor

[Dustin Black]

Victor:

Created a new user (.profile is posted above). Logged onto the problem machine via rlogin as root. Did su - testuser. The copyright information is displayed, but no prompt is given afterward unless ^C is used.

The user's home directory is mounted NFS. I'm not sure what other information you are seeking about "What is on NFS?"

Same network. Haven't tested the network connection issues (see note to Clay above).

/etc/nsswitch.conf file below:

passwd: files nis
group: files nis
hosts: files nis dns [NOTFOUND=return]
networks: files nis dns [NOTFOUND=return]
protocols: nis [NOTFOUND=return] files
rpc: nis [NOTFOUND=return] files
publickey: nis [NOTFOUND=return] files
netgroup: nis [NOTFOUND=return] files
automount: files nis
aliases: files nis
services: files nis

[Victor BERRIDGE]

Whats in your /etc/profile?
I believe it may come from there


All the best
Victor

[Victor Fridyev]

Hi,

My latest solution for this problem is to build a link:
ln -s /tmp/$LOGNAME.hst ~/.sh_history

HTH

[A. Clay Stephenson]

Now that I see NIS is in play, all of your problems can be explained as NIS related. Do a ypwhich -x and see if the response is rapid. Do this on a bad client. Man ypwhich for details. A ypcat auto_master would probably be usefulk as well. I also note that hostnames are resolved via NIS so a slow response there could be hurting you as well. Do a ypcat hosts. Man ypcat for details.

[A. Clay Stephenson]

Ooops, that should be a ypwich -m to list the automounter maps.

[Stephen Keane]

Do an 'nfsstat -z' to zero the statistics. Then try and log in to a 'bad' user, then re-run nfsstat to see more accurate figures (they are cumulative). At least you'll see whether you've got a problem with your NFS (and probably your NIS).

[Dustin Black]

Thanks for all of your help... but I finally got the information I needed from a call-in to a support rep.

There is an issue with the sh and ksh shells that can cause this login hang problem. The fix is as below:

On the NFS server -
kill the rpc.statd service
kill the rpc.lockd service

rm -r /var/statmon/sm
rm -r /var/statmon/sm.bak

(restart rpc.statd and rpc.lockd)
/usr/sbin/rpc.statd
/usr/sbin/rpc.lockd

On the client -
Try the login again. It may hang for up to a minute the first try after performing the fix on the server, but all subsequent logins should go fine.


Also, look to this patch to actually (hopefully) resolve the issue:
PHNE_30661

[Dustin Black]

See above post