All but one NIS user unable to login via console or ssh

[CMyers_1]

Original symptoms nis users were unable to ssh to the HP-UX system. Additional debugging of SSH seemed to indicate that SSH authentication is ok but that the user never gets logged in.
I have compared the ssh debug information from the one NIS user that succeeds to one that never gets the prompt and they are identical except for username and pid. Also SFTP works and drops the user into their /h/$user folder no problem

# ypwhich
LEXDC4.xxx.com

# ypcat passwd
user1:w6yyNRxH/8idI:10004:10004::/h/user1:/bin/sh
user2:hPbfdH5kRvVxU:10025:10003::/h/user2:/bin/sh
user3:cRwFl688EIrWE:10026:10003::/h/user3:/bin/sh
# ls -al /h
drwxrwxrwx 2 user1 ps 64 Dec 9 08:48 user1
drwxrwxrwx 2 user2 ps 64 Dec 9 08:48 user2
drwxrwxrwx 2 user3 ps 64 Dec 9 08:48 user3

#mount
/h on xxxxFS01.xxx.com:/unixhome soft,rsize=32768,wsize=32768,NFSv3,dev=2000005 on Wed Jan 19 17:21:06 2011

so user 1 is able to login via ssh remotely and also if I as root so an ssh user1@localhost
i can login as the user
but all the other users have no joy

# /etc/nsswitch.nis:
#
# An example file that could be copied over to /etc/nsswitch.conf; it
# uses NIS (YP) in conjunction with files.
#

passwd: files [NOTFOUND=continue] nis
group: files [NOTFOUND=continue] nis
hosts: files [NOTFOUND=continue] nis [NOTFOUND=continue UNAVAIL=continue] dns
networks: files [NOTFOUND=continue] nis
protocols: nis [NOTFOUND=continue UNAVAIL=continue] files
rpc: nis [NOTFOUND=continue UNAVAIL=continue] files
publickey: nis [NOTFOUND=continue UNAVAIL=continue] files
netgroup: nis [NOTFOUND=continue UNAVAIL=continue] files
automount: nis [NOTFOUND=continue UNAVAIL=continue] files
aliases: files nis
services: files nis

cat /etc/passwd # cut a bit here
root:plOkBWORxN5Ds:0:3::/:/sbin/sh
dazel::115:20::/home/dazel:/sbin/sh
+::-2:-2:::


# uname -a
HP-UX unknown B.11.23 U ia64 1194092371 unlimited-user license

here is what the user who can not login sees

Last login: Wed Jan 19 18:49:38 2011 from localhost
(c)Copyright 1983-2003 Hewlett-Packard Development Company, L.P.
(c)Copyright 1979, 1980, 1983, 1985-1993 The Regents of the Univ. of California
(c)Copyright 1980, 1984, 1986 Novell, Inc.
(c)Copyright 1986-2000 Sun Microsystems, Inc.
(c)Copyright 1985, 1986, 1988 Massachusetts Institute of Technology
(c)Copyright 1989-1993 The Open Software Foundation, Inc.
(c)Copyright 1990 Motorola, Inc.
(c)Copyright 1990, 1991, 1992 Cornell University
(c)Copyright 1989-1991 The University of Maryland
(c)Copyright 1988 Carnegie Mellon University
(c)Copyright 1991-2003 Mentat Inc.
(c)Copyright 1996 Morning Star Technologies, Inc.
(c)Copyright 1996 Progressive Systems, Inc.


RESTRICTED RIGHTS LEGEND
Use, duplication, or disclosure by the U.S. Government is subject to
restrictions as set forth in sub-paragraph (c)(1)(ii) of the Rights in
Technical Data and Computer Software clause in DFARS 252.227-7013.


Hewlett-Packard Company
3000 Hanover Street
Palo Alto, CA 94304 U.S.A.

Rights for non-DOD U.S. Government Departments and Agencies are as set
forth in FAR 52.227-19(c)(1,2).


And then nothing no # or $ or anything
~

[Steven Schweda]

As usual, potentially interesting:

uname -a
ssh -V

> [...] but that the user never gets logged
> in. [...]

But the user does get logged in.

Are the users' home directories local (or
at least accessible, if remote)?

> And then nothing no # or $ or anything

Apparently, the user's shell got so far as
displaying the copyright text, presumably as
a result of:
cat /etc/copyright
in "/etc/profile", but not all the way
through all the shell start-up command files.

As a quick experiment, you might move a
user's shell start-up stuff ("~/.profile",
...) out of the way. If that helps, then
adding a
set -x
and/or
set -v
near the beginning of that file might provide
some clue to where it gets busy/hung.

[CMyers_1]

the users home directory in case it was not clear with all the jabbering above is located at /h/$username
no .profile or anything is currently populated within the /h/username
/h is mounted via nfs from /etc/fstab
xxxFS01.lex.adapps.hp.com:/unixhome /h nfs rw,suid,soft,intr 0 0

bad form I know as all users home dirs can be seen but I have not gotten automounting working yet since I am focused on this issue first.

So as an experiement I changed the users home directory to /home/user2 and validated it showed up in ypcat
user2:w6yyNRxH/8idI:10004:10004::/home/user2:/bin/sh

At this point user2 was able to login via ssh

So I went back and deleted the /h/user2 chown'ed and 777'ed it and added a .profile with set -v and set -x in there but the symptom returned upon trying to login to /h/user2.

Deleted .profile and still was not able to login. So it appears to be tied to the nfs mount which is odd considering one user can still log int.

I deleted

[CMyers_1]

also
# uname -a
HP-UX unknown B.11.23 U ia64 1194092371 unlimited-user license
# ssh -V
OpenSSH_5.6p1+sftpfilecontrol-v1.3-hpn13v7, OpenSSL 0.9.8o 01 Jun 2010
HP-UX Secure Shell-A.05.60.002, HP-UX Secure Shell version

[Steven Schweda]

> no .profile or anything [...]

You could throw in an "echo something" at the
end of "/etc/profile", just to make sure that
you're getting all the way through that.


> /h is mounted via nfs from /etc/fstab

Can't see how it's shared from the remote
system. "mount" on the client system might
say something interesting. If the problem is
tied to NFS, then all the NFS details become
(potentially) important.

> So I went back and [...]

Ok. Actual output from actual commands might
be more informative than vague descriptions
("chown'ed and 777'ed it") and
interpretations. Output from "ls -l", for
example, run on both the server and the
client, ideally.

Any messages in anyone's syslog file?

[Dennis Handly]

If it hangs need the end of the motd it could be due to a hang on a NFS .sh_history file.

[CMyers_1]

Adding an echo to the end of /etc/profile indicates the login process completes to that point.
steps used to test if /h/user2 home directory is causing the problem
### check users current in nis
# ypcat passwd | grep 'user2'
user2:w6yyNRxH/8idI:10004:10004::/h/user2:/bin/sh
### change users homedir and ensure that it propagates to server
# ypcat passwd | grep 'user2'
user2:w6yyNRxH/8idI:10004:10004::/home/user2:/bin/sh
### make new home dire
# cd /home
# ls
support thun pkaz
# mkdir /home/chmy
# chown -R /home/chmy chmy:cs
chown: unknown user id /home/chmy
# chown -R chmy:cs /home/chmy
#ls -al
...
drwxr-xr-x 2 chmy cs 96 Jan 20 09:23 chmy
...
### check that no user processes are left over
# ps -ef | grep 'user2'
root 15022 2942 1 09:28:12 pts/0 0:00 grep user2
### rm /h/user2
# rm -rf /h/user2

At this point I was able to ssh into the system successfully and validate that I was in /home/user2

### time to change back home directory to nfs and check if ssh login is broken.
### validate that /h is mounted
# cat /etc/fstab
...
xxxFS01.lex.adapps.hp.com:/unixhome /h nfs rw,suid,soft,intr 0 0
....
# mount
/h on xxxFS01.lex.adapps.hp.com:/unixhome soft,rsize=32768,wsize=32768,NFSv3,dev=2000005 on Wed Jan 19 17:21:06 2011
### /h/user2 should not be there since we deleted it earlier
# ls /h/user2
/h/user2 not found
# mkdir /h/user2
# chown -R user2:cs /h/user2
### change home directory back and verify shows on server
# ypcat passwd | grep 'user2'
user2:w6yyNRxH/8idI:10004:10004::/h/user2:/bin/sh
### check that all processes are gone
# ps -ef | grep 'user2'
root 15161 2942 0 09:39:46 pts/0 0:00 grep user2

### Tried to SSH in and it failed to complete. Log message located at end of /etc/profile was displayed.

[CMyers_1]

A little more info I was able to see that /etc/profile was ran as indicated above
I then created a .profile in /h/user2 and was able to see that echo message as well when trying to ssh.
#touch /h/user2/.profile
# vi /h/user2/.profile
>>echo "got to user2 dot profile"
# chmod 777 /h/user2/.profile
# chown user2:cs /h/user2/.profile

# ssh user2@localhost
Rights for non-DOD U.S. Government Departments and Agencies are as set
forth in FAR 52.227-19(c)(1,2).
end of etc profile
got to user2 dot profile

[CMyers_1]

Bah,
I came across a thread while googleing this morning that indicated sh had the same issues with nfs locks and ksh. I had dismissed the nfs lock on .sh_history since i was not using ksh (the only place i had heard this occurring)
but this thread indicated that sh and ksh share the same code base to a certain extent
http://lists.fini.net/pipermail/ldap-interop/2005-September/000693.html

at which point i edited /etc/profile and added the following
### create a new .sh_history to test if nfs
### lock is causing login issues
touch /tmp/.sh_history.$LOGNAME
chown $LOGNAME /tmp/.sh_history.$LOGNAME
chmod 755 /tmp/.sh_history.$LOGNAME
HISTFILE=/tmp/.sh_history.$LOGNAME

after which i was able to login as the user! Yahtzee!

dont know if this was the best way to do it or if there is a better best practice so hopefully someone chimes in!
going back to award some points here shortly after some more testing.

[Dennis Handly]

I've only had ksh hang but that's all I use for non-root.

>don't know if this was the best way to do it or if there is a better best practice so hopefully someone chimes in!

That will make it local. It seems fine except it shouldn't be executable (755) and if the directory is sticky, nobody else can remove the history file. You want others to read it?