LDAP-UX: First ldap login takes about 5 minutes, others are fine

[dev44]

I have ldap-ux client running on my hpux 11.31 system. I am able to login to the various ldap accounts setup for testing on the AD. The only curious thing, is that the first login takes like 5 minutes. All consecutive logins to the same account are immediate. Any ideas why the first would take so long?

[Eckle]

Hi dev44,

I had such same troubles some days ago on my recent HP-UX 11.31 box...
For me it has to do with the ldapclientd daemon which was in DEBUG mode... (Disabling it solve me problem)

1. Perhaps check this on you server:
http://www.docs.hp.com/en/J4269-90041/ch04s14.html

2. Are you on the last Version of LDAP-UX : 4.20?
(swlist -l product |grep LDAP-UX)

3. You can try to disable the "passwd cache" when starting ldapclientd: /opt/ldapux/bin/ldapclientd -D passwd
By careful that this is one "one-shot" action... to modify it definitely You must change the "rc-file" /sbin/init.d/ldapclientd.rc...

The last thing I can suggest is to tune the ldapclientd process through tusc.

Good Luck.

[dev44]

I captured the attached when trying to login to a new account that took 5 minutes. Anyone want to take a crack at it, I would appreciate it.

[dev44]

Thanks for the reply Eckle....they are in short supply these days it seems ;)

Anyway, I was in INFO mode...

I think it might have to do with my pam.conf but I am not sure.

I will give you 10 points for your help once I can solve this.

Thanks again...

[dev44]

Somebody?
Anybody?????

[Don Mallory · Accepted solution]

Hi dev44,

What does the output look like once you have logged in once? That's neat that it sits for 4 minutes doing nothing form 14:35 to 14:39.

Can you send the output of an sshd -ddd on the server side and ssh -vvv on the client side as well?

You don't happen to have the pwgrd daemon off do you? (password & group caching daemon.) It's not technically required, but works in conjunction with the ldapclientd caching.

Just out of curiosity, your primary ldap server is available, right? You can have up to three ldap servers listed in the ldapuxprofile, but what they don't tell you is that EVERY query goes to the 1st, then the 2nd, then the 3rd, regardless of whether the 1st or 2nd were down on the last request. If you used DNS names, it will also attempt to resolve each one, with a 10 second timeout per DNS request. So, you can see that if your DNS is down, and/or your LDAP server is down, or worse, your primary LDAP server is your DNS server (been there), you could end up with VERY long login times.

Don

[dev44]

Hi Don, thanks for the reply....

So pwgrd is running, and we only have the one DC listed in the profile.

I will try and get that sshd -ddd and ssh -ww info for you.

[dev44]

I attached two files....login.slow, is the inital login which takes 5 minutes. The file login.quick represents all subsequent logins.

[dev44]

Here is the login.slow file

[Bob Neal-Joslin]

From the trace, it does appear that LDAP-UX is able to query the directory server quickly. The delay seems to occur after the first searches have completed and before the later ones have started.

I think the recommendation for debug output from sshd is a good idea, to see where it thinks things are hanging up. If it's in PAM, then you can turn on PAM debugging using:

1. Configure syslog:
save a backup copy of syslog.conf, then
add this line to /etc/syslog.conf
*.debug /var/adm/syslog/syslog.log
note: this requires tab characters like the other entries

2. Turn on debug in pam module(s):
save a backup copy of /etc/pam.conf, then
add "debug" to the end of each entry in /etc/pam.conf; e.g
sshd account required /usr/lib/security/libpam_unix.1 debug
(actually, you could do just the lines you care about)

3. turn on libpam debug: touch /etc/pam_debug

4. start syslog daemon: /sbin/init.d/syslogd start

5. do tests to generate pam syslog debug info in
/var/adm/syslog/syslog.log

6. cleanup
/sbin/init.d/syslogd stop
rm /etc/pam_debug
restore /etc/pam.conf and /etc/syslog.conf

[Don Mallory]

There appears to be nothing from the sshd -ddd, and both files look identical.

Were you able to do an ssh -vvv from the client perspective?

Do you have glance (OV Perf tools) installed, can you pull it up, find the process and select it (g to list all processes, s to select one in particular) It should be able to tell you what the process state is.

[Don Mallory]

Is there a reason that the home dir for the user is missing on the destination? If you have /etc/defaults/security set with "ABORT_LOGIN_ON_MISSING_HOMEDIR=1", the user would not be allowed to log in. This is a highly recommended security practice.

[dev44]

Hi Don,

It happens with or without a home directory. I tried it both ways.

[Don Mallory]

Hi dev44,

I just meant that it seemed odd not to have a home dir. From a security perspective, it is highly recommended to have a home dir for each user, and disallow logins to users that it does not exist.

This type of thing doesn't tend to slow logins, it tends to prevent them entirely.

I think the ssh -vvv from the client and, as recommended by Bob, pam debug logging on the ssh items is the next step.

Don

[dev44]

Oh, I see...sorry Don.
These are only test users, so we don't bother with the home directory because once it logs in the once, then it is fine. So another account has to be created for further testing. I will continue with the recommendations. Thanks

[Don Mallory]

Just so I'm sure I read that correctly, the first time >EVER< the account logs in, it's slow, anytime after, no matter length of time, it logs in fine.

Does a reboot, or restart of sshd, ldapclientd, pwgrd or anything else seem to have an impact?

[Don Mallory]

Can you enable event logging on the DC you are getting your LDAP from and review it from that perspective as well?

[Steven E. Protter]

Shalom,

I have now reached the conclusion that there is no problem with this system.

There probably is a patch for the server that will make this perform better, but a lot of things get checked for first time login and it might be best just to ignore the problem or look for an update for the LDAP server.

Do the basics such as make sure network traffic is flowing freely.

SEP

[dev44]

Sorry Don, I wasn't in yesterday.

Anyway, the AD folks watched it come in from their end and it came in for a second. So it seems to be on the HP end.

I know an initial login will take a little more time but 5 minutes is unacceptable. If there were network problems, they would show in subsequent logins. There are no network problems.

[Don Mallory]

Hi dev44,

No problem. So we need the PAM debug log (just add " debug" to the end of every pam line in your /etc/pam.conf, in /etc/syslog.conf, add a *.debug entry to an output file (don't forget that whitespace must be TAB), touch the file, then kill -HUP your syslogd.

And we also need the ssh -vvv from the client.

Is the response the same for other login methods other than ssh? (can you temporarily turn on telnet for example?), does restarting pwgrd, ldapclientd, sshd, or rebooting the host have an impact on previously working users?

Don

[dev44]

Ok, so here are the logs from the debug and the ssh -vvv

[Don Mallory]

Hi dev44,

The only thing that is standing out is that your hptest6 account password is expired:

Dec 16 10:19:07 server sshd[22288]: pam_sm_acct_mgmt: exiting, error 17
Dec 16 10:19:07 server sshd[22288]: pam_acct_mgmt: error User account has expired
Dec 16 10:20:29 server sshd[22295]: warn_user_passwd_will_expire: -1 -1 14594 -1

Then it sits for 3 minutes on a permission denied error:

Dec 16 10:20:32 server sshd[22295]: pam_setcred: error Permission denied

before moving on to try to fail again.

Get your account admins to leave the account without a password that needs to be changed on first login, or log in on a windows box, change the password and log in again.

You could also try getting them to set the acount with the password to never expire, and not change the password on next logon.

Don't put host or time limits on the account either.

Don

[dev44]

Thanks Don....I will test that out and let you know.

[dev44]

We had a 550MB wtmp file....

[Don Mallory]

Wow, that's cool. I had no idea there was a limit. Just out of curiosity, was /var full, and it wasn't able to write any more activity?

Don