topic Re: Problem with users authenticated by ldap in Operating System - HP-UX

Problem with users authenticated by ldap

Pioo — Thu, 24 Sep 2009 15:49:33 GMT

Hi,

Our ERP system relies on OS to authenticate users. Some users are defined locally on a hp-ux machine, and others come from an ldap server.
We're facing a strange issue with the ldap users, all is fine with the local Unix accounts.

The OS version is:
# uname -s -r -v -m -l
HP-UX B.11.31 U ia64 unlimited-user license

The LDAP Client version:
ux300:/etc # swlist | grep -i ldap
LDAPUX B.04.10 LDAP-UX Integration

The problem occurs when an ldap user submits a batch job. The job scheduler process, which is running with root privileges, forks a new process owned by the user.
Unfortunately the new process terminates immediately with the following error (from the application log):

error code: 114, pc=0, call=1, seg=0
114 Attempt to access item beyond bounds of memory (Signal 11)

I used tusc to trace a failing process. The process terminated at the very beginning, it hadn't even started processing its business logic. In the trace file I found the following lines:

mmap(NULL, 8192, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) ......................................................................... = 0x1ffffffffcb0e000
stat("/etc/opt/ldapux/ldapux_client.conf", 0x9fffffffffffc8b0) ............................................................................................. = 0
open("/etc/opt/ldapux/ldapux_client.conf", O_RDONLY|0x800, 0666) ........................................................................................... = 7
ioctl(7, TCGETA, 0x9fffffffffffc920) ....................................................................................................................... ERR#25 ENOTTY
read(7, "# L D A P - U X C l i e n t ".., 8192) ........................................................................................................ = 8192
read(7, "n e o f t h e s e r v i c ".., 8192) ........................................................................................................ = 1084
read(7, 0x6000000000117f98, 8192) .......................................................................................................................... = 0
close(7) ................................................................................................................................................... = 0
brk(0x6000000000140000) .................................................................................................................................... = 0
open("/var/spool/ldapcltd/status", O_RDONLY|0x800, 0) ...................................................................................................... = 7
fstat(7, 0x9fffffffffffcf40) ............................................................................................................................... = 0
mmap(NULL, 12, PROT_READ, MAP_SHARED|MAP_VARIABLE|MAP_FILE|MAP_ADDR32|0x800, 7, 0) ......................................................................... ERR#12 ENOMEM
close(7) ................................................................................................................................................... = 0
Received signal 11, SIGSEGV, in user mode, [caught], partial siginfo
Siginfo: si_code: SEGV_ACCERR, faulting address: 0, si_errno: 0
PC: 00000001000000a0.0 break.m 0x16000

I looks that the issue comes from the mmap() system call.

I noticed that the 'su' command works perfectly for both types of users. Even ldap users can establish a valid OS session.

I would be very grateful for any suggestions which might help me resolving the problem.

Regards,
Pit

Re: Problem with users authenticated by ldap

TwoProc — Thu, 24 Sep 2009 19:56:54 GMT

I'm guessing here - you might want to take a look at "man 1m setprivgrp", and set those users up in a priv group that can use maybe MLOCK or SHM_LOCK.

But... I'm confused as to why a person being an ldap user vs not being makes a difference - unless it's inferred that a user that's local gets some of these privs and a user that comes from the outside world doesn't because it's not defined in a group, much less a priv group.

Or, it's a bug...

Re: Problem with users authenticated by ldap

Pioo — Fri, 25 Sep 2009 12:07:46 GMT

Hi,

Thanks a lot for the reply, but unfortunately the setprivgrp command didn't resolve my issue ;(
All application users (local and ldap) belong to the same Unix group, and only to the one.

I investigated the issue a little bit more. When a batch job is started, the job scheduler (running with root privileges) forks a new process owned by the user who requested the job.
It always fails for the ldap users. As I wrote the ldap users can establish a valid Unix session. I connected to the system as an ldap user and attempted to manually start a batch.
When â pwgrâ was running, some jobs started and completed successfully and some not (about 60-70% succeeded). When â pwgrâ was not running, all jobs failed!

Moreover I traced (tusc) the â suâ command when switching to an ldap user. I checked which authentication module was used and I found:

# grep -iE "(pam|nss|ldap)" /tmp/su_01.trc | grep -v "ENOENT$"
open("/usr/lib/hpux32/libpam.so.1", O_RDONLY, 0) ........................................................................................................... = 4
stat("/etc/pam_debug", 0x7ffff300) ......................................................................................................................... = 0
stat("/etc/pam.conf", 0x7ffff2e0) .......................................................................................................................... = 0
open("/etc/pam.conf", O_RDONLY, 0666) ...................................................................................................................... = 5
open("/etc/nsswitch.conf", O_RDONLY, 0666) ................................................................................................................. = 5
open("/usr/lib/hpux32/libnss_files.so.1", O_RDONLY, 0) ..................................................................................................... = 5
open("/usr/lib/nls/msg/C/libpam.cat", O_RDONLY, 0101770) ................................................................................................... = 5
open("/usr/lib/hpux32/libnss_files.so.1", O_RDONLY, 0) ..................................................................................................... = 4

When the application starts a job and the operation fails, the following is in use (from a tusc trace file):

# grep -iE "(pam|nss|ldap)" /tmp/tstjob_01.trc | grep -v "ENOENT$"
open("/etc/nsswitch.conf", O_RDONLY|0x800, 0666) ........................................................................................................... = 7
open("/usr/lib/hpux64/libnss_files.so.1", O_RDONLY|0x800, 0) ............................................................................................... = 7
open("/usr/lib/hpux64/libnss_ldap.so.1", O_RDONLY|0x800, 0) ................................................................................................ = 7
open("/usr/lib/hpux64/libldap_send.so", O_RDONLY|0x800, 0) ................................................................................................. = 7
stat("/etc/opt/ldapux/ldapux_client.conf", 0x9fffffffffffc8b0) ............................................................................................. = 0
open("/etc/opt/ldapux/ldapux_client.conf", O_RDONLY|0x800, 0666) ........................................................................................... = 7
open("/var/spool/ldapcltd/status", O_RDONLY|0x800, 0) ...................................................................................................... = 7

Any ideas?

Regards,
Pit

Re: Problem with users authenticated by ldap

Pioo — Tue, 13 Oct 2009 21:11:40 GMT

Hello again,

I isolated the problem a little bit more.

1) When the ldap client demon (ldapclientd) starts, it creates and opens the status file /var/spool/ldapcltd/status. Than it requests memory mapping using the mmap system call:

close(232) ............................................... ERR#9 EBADF
open("/var/spool/ldapcltd/status", O_RDWR|O_CREAT, 0644) . = 6
close(244) ............................................... ERR#9 EBADF
close(233) ............................................... ERR#9 EBADF
close(245) ............................................... ERR#9 EBADF
close(234) ............................................... ERR#9 EBADF
fstat(6, 0x7ffff380) ..................................... = 0
close(246) ............................................... ERR#9 EBADF
close(235) ............................................... ERR#9 EBADF
getuid() ................................................. = 0 (0)
close(247) ............................................... ERR#9 EBADF
close(236) ............................................... ERR#9 EBADF
close(248) ............................................... ERR#9 EBADF
close(237) ............................................... ERR#9 EBADF
ftruncate(6, 12) ......................................... = 0
close(249) ............................................... ERR#9 EBADF
close(238) ............................................... ERR#9 EBADF
mmap(NULL, 12, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_VARIABLE|MAP_FILE|MAP_ADDR32|0x800, 6, 0) = 0xc0d01000
close(250) ............................................... ERR#9 EBADF
close(239) ............................................... ERR#9 EBADF

2) When the ldap client is running, all ldap users are able to connect and establish a valid UNIX session.

3) When an ldap user (or system on behalf of the user) invokes a command, an attempt to access the memory mapped status file is made:

a) when the ldap user starts an interactive program - it works:

brk(0x60000000006ef000) .................................. = 0
brk(0x60000000006f7000) .................................. = 0
open("/var/spool/ldapcltd/status", O_RDONLY|0x800, 0) .... = 6
fstat(6, 0x9fffffffffffcd20) ............................. = 0
mmap(NULL, 12, PROT_READ, MAP_SHARED|MAP_VARIABLE|MAP_FILE|MAP_ADDR32|0x800, 6, 0) = 0xc0d01000
close(6) ................................................. = 0

b) when the ldap user starts a batch program it fails with the error:

brk(0x6000000000140000) .................................. = 0
open("/var/spool/ldapcltd/status", O_RDONLY|0x800, 0) .... = 8
fstat(8, 0x9fffffffffffcf40) ............................. = 0
mmap(NULL, 12, PROT_READ, MAP_SHARED|MAP_VARIABLE|MAP_FILE|MAP_ADDR32|0x800, 8, 0) ERR#12 ENOMEM
close(8) ................................................. = 0
Received signal 11, SIGSEGV, in user mode, [caught], partial siginfo
Siginfo: si_code: SEGV_ACCERR, faulting address: 0, si_errno: 0
PC: 00000001000000a0.0 break.m 0x16000

What differs the two types of sessions (interactive and batch)? What causes that a session has not enough privileges to access the memory structure.

Re: Problem with users authenticated by ldap

Dennis Handly — Wed, 14 Oct 2009 06:41:06 GMT

mmap(NULL, 12, PROT_READ, MAP_SHARED| MAP_VARIABLE| MAP_FILE| MAP_ADDR32| 0x800, 8, 0) ERR#12 ENOMEM

You are out of 32 bit system shared memory. It is kind of hard to believe since you only want 12 bytes, a whole page.

Are you making multiple mmap(2) calls to that file?

>What differs the two types of sessions (interactive and batch)? What causes that a session has not enough privileges to access the memory structure.

It has nothing to do with privileges, just shared memory fragmentation. It would just be random.
You may want to remove 0x800 (MAP_GLOBAL).

What does "ipcs -ma" show?

It seems you should be contacting the Response Center, they have tools to look at shared memory.

Re: Problem with users authenticated by ldap

Pioo — Thu, 15 Oct 2009 13:13:02 GMT

Hi,

Thanks a lot for the reply.

> Are you making multiple mmap(2) calls to that file?
In the trace files I haven't seen more than one call to the file.

The error is not random. After I stopped the pwgr daemon it occurs always for batch jobs and never for interactive user programs.

> You may want to remove 0x800 (MAP_GLOBAL).
Unfortunately I am not a developer and the application is a commercial product, I don't have any possibility to make changes.

I don't know if I understand the problem correctly, but I suppose the file has already been mapped by another process - the ldap client daemon process. It's a shared mapping and each next session tries to read the same memory area. According to my observations, the memory can be read by some sessions (interactive programs) but cannot be accessed by batch jobs. The mmap() function, for the programs which execute correctly, returns the same address as the ldap daemon got at startup.

Regards,
Pit

Re: Problem with users authenticated by ldap

Dennis Handly — Fri, 16 Oct 2009 08:03:17 GMT

>I suppose the file has already been mapped by another process - the ldap client daemon process.

If mapped by a 64 bit process without MAP_ADDR32, a 32 bit process would have problems.

>the memory can be read by some sessions (interactive programs) but cannot be accessed by batch jobs.

That shouldn't be the case. It could fail because there is an environment variable that isn't set the same. Looking closely at your two tusc outputs, it seems everything is the same except the FDs.

>returns the same address as the ldap daemon got at startup.

Yes it should. Unless the demon isn't running.

Re: Problem with users authenticated by ldap

Pioo — Sun, 18 Oct 2009 11:34:32 GMT

Thanks again for the reply.

> If mapped by a 64 bit process without MAP_ADDR32, a 32 bit process would have problems.

That is not the case, the ldapclientd demon maps the file using the command:
mmap(NULL, 12, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_VARIABLE|MAP_FILE|MAP_ADDR32|0x800, 6, 0) = 0xc0d01000

> Looking closely at your two tusc outputs, it seems everything is the same except the FDs.

I think the FDs have nothing to do with the issue. A file descriptor (in the tusc trace output) is a next available sequential number which is not used by another file. Different file descriptors for the same file in different programs suggest that the programs behave on a different way and opened (and closed) a different number of files before opening the status file.

> Yes it should. Unless the demon isn't running.

But if the mmap() system call returns the same address, I suppose all programs access the same area of memory.

Regards,
Pit

Re: Problem with users authenticated by ldap

Dennis Handly — Sun, 18 Oct 2009 17:47:09 GMT

>That is not the case, the ldapclientd demon

Any other 64 bit processes involved?

>I think the FDs have nothing to do with the issue.

Most likely but I was just comparing everything.

>if the mmap() system call returns the same address,

Only if some process still has it mapped.

Have you made sure you have the latest patches and if that didn't fix it, contact the Response Center?

Re: Problem with users authenticated by ldap

Pioo — Mon, 19 Oct 2009 11:13:32 GMT

Thanks again for your answers!

> Any other 64 bit processes involved?

The application is a 64 Bit process, but the ldap client demon is 32 bit. The ldap client demon maps the file first.

# file /opt/ldapux/bin/ldapclientd
/opt/ldapux/bin/ldapclientd: ELF-32 executable object file - IA64

> Only if some process still has it mapped.

According to my observations, the ldap client demon opens and maps the file at the very beginning and keeps it until termination.

> Have you made sure you have the latest patches and if that didn't fix it, contact the Response Center?

Unfortunately I am not the admin of the server, I am even not a UNIX admin. I think our administrators have requested HP for help, but I am trying to help also ...and learn something interesting ;)

Pit

Re: Problem with users authenticated by ldap

Pioo — Mon, 02 Nov 2009 14:55:30 GMT

Hi,

Funny thing is that when the pwgr daemon is running, and an ldap user has a valid session, the user can execute some programs from within the session.

When the daemon is stopped, the same programs (executed in the same session) fail.

The pwgr daemon casches some user data. I suppose it put something vital into the cache when the user established the session.

Any ideas?

Regards,
Pit