topic Re: Inetd Hanging in Operating System - HP-UX

Inetd Hanging

Paul Murray_4 — Tue, 26 Apr 2005 02:59:05 GMT

I'm having a little trouble with inetd on one of my HP-UX 11.0 servers.

Even though the configuration files (/etc/inittab, /etc/inetd.conf and /etc/rc.config.d/netdaemons) and versions of the binaries match the other 11.0 servers on our network, this one particular box is showing rather strange behaviour.

When restarting inetd, inetd hangs until you issue the "inetd -b" command - it then works normally.

Restarting the process (and even rebooting the server) seems to make no difference - inetd still hangs until it gets the -b switch !!

HELP !?!?!?!?!

Re: Inetd Hanging

Alex Lavrov. — Tue, 26 Apr 2005 03:10:53 GMT

Do you have any entries in syslog?

Re: Inetd Hanging

Paul Murray_4 — Tue, 26 Apr 2005 03:16:04 GMT

Hi Alex - the only syslog messages written, hit the log after the "inetd -b" command is issued (before this, no messages are logged at all).

However, the messages after issuing "inetd -b" are :-

Apr 26 08:53:08 lndvs101 syslog: inetd sendsig: entered, sig=4
Apr 26 08:53:08 lndvs101 inetd[7227]: Extended connection logging enabled
Apr 26 08:53:08 lndvs101 syslog: inetd sendsig: exiting
Apr 26 08:53:08 lndvs101 inetd[7227]: protocol = tcp
Apr 26 08:53:08 lndvs101 inetd[7227]: telnet/tcp: Added service, server /usr/lbin/telnetd
Apr 26 08:53:08 lndvs101 inetd[7227]: ftp/tcp: Added service, server /usr/lbin/ftpd
Apr 26 08:53:08 lndvs101 inetd[7227]: login/tcp: Added service, server /usr/lbin/rlogind
Apr 26 08:53:08 lndvs101 inetd[7227]: shell/tcp: Added service, server /usr/lbin/remshd
Apr 26 08:53:08 lndvs101 inetd[7227]: exec/tcp: Added service, server /usr/lbin/rexecd
Apr 26 08:53:08 lndvs101 inetd[7227]: swat/tcp: Added service, server /opt/samba/bin/swat
Apr 26 08:53:08 lndvs101 inetd[7227]: cvspserver/tcp: Added service, server /usr/local/bin/cvs-server
Apr 26 08:53:08 lndvs101 inetd[7227]: rsync/tcp: Added service, server /usr/local/bin/rsync
Apr 26 08:53:08 lndvs101 inetd[7227]: protocol = tcp
Apr 26 08:53:08 lndvs101 inetd[7227]: Configuration complete
Apr 26 08:53:08 lndvs101 inetd[7227]: About to select()...
Apr 26 08:53:10 lndvs101 syslog: inetd sendsig: entered, sig=4

Thanks,
Paul.

Re: Inetd Hanging

Alex Lavrov. — Tue, 26 Apr 2005 03:20:45 GMT

If you have tusc, you can try and look what system calls make inetd hang with:

tusc -f

Do you have any additional lines that you added to inetd.conf? (even if they exist on other systems, it's still can cause problems here)

Re: Inetd Hanging

Joseph Loo — Tue, 26 Apr 2005 03:22:18 GMT

hi,

the -b option is a toggle which turns debug on and off. what is option running on inetd"

# ps -ef|grep -i inetd

also, r there the patches for the other 11.0 similar to this server.

regards.

Re: Inetd Hanging

Henk Geurts — Tue, 26 Apr 2005 03:26:30 GMT

is the latest inetd patch (PHNE_28312)installed ?

regards.
H

Re: Inetd Hanging

Alex Lavrov. — Tue, 26 Apr 2005 03:27:13 GMT

And another question:
How do you restart inetd?

Re: Inetd Hanging

Paul Murray_4 — Tue, 26 Apr 2005 04:08:17 GMT

The current version of inetd is :-

root@lndvs101..# what /usr/sbin/inetd
/usr/sbin/inetd:
Copyright (c) 1983 Regents of the University of California.
Revision: 1.12.214.3 Tue Aug 1 10:30:45 GMT 2000
Patch id: PHNE_21835

When restarting, I've tried using the stopping and starting using the /sbin/init.d/inetd script, and also by running "inetd -k" and "/usr/sbin/inetd -l" commands manually. Then, just to be paranoid, I've even tried rebooting the server.

The odd thing is, this server is running the same patch levels as all of our 11.0 servers, but started showing this behaviour at the end of last week even though nothing has changed !!!

Ok - slight update - I've just tried bouncing inetd again, but I couldn't get it to restart at all and had to reboot the server again.

I've got the output from "tusc -f", but I'm not overly familiar with what it's telling me - should I be looking for anything specific ??

(Thanks again for all the help !!!).

Re: Inetd Hanging

Alex Lavrov. — Tue, 26 Apr 2005 04:12:07 GMT

If inetd hangs, you can attach here the last lines from tusc output. Those should be the last system calls that inetd executes before it hangs.

Re: Inetd Hanging

Paul Murray_4 — Tue, 26 Apr 2005 04:27:02 GMT

Hi Alex - the last part of the tusc output is :-

setaudid(0) ...................................................... = 0
sigsetreturn(0x7f7bcd56, 0x6211988, 1392) ........................ = 0
sigvec(SIGHUP, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGINT, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGQUIT, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGILL, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGTRAP, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGABRT, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGEMT, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGFPE, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGKILL, 0x7f7e0820, 0x7f7e082c) .......................... ERR#22 EINVAL
sigvec(SIGBUS, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGSEGV, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGSYS, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGPIPE, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGALRM, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGTERM, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGUSR1, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGUSR2, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGCLD, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGPWR, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGVTALRM, 0x7f7e0820, 0x7f7e082c) ........................ = 0
sigvec(SIGPROF, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGIO, 0x7f7e0820, 0x7f7e082c) ............................ = 0
sigvec(SIGWINCH, 0x7f7e0820, 0x7f7e082c) ......................... = 0
sigvec(SIGSTOP, 0x7f7e0820, 0x7f7e082c) .......................... ERR#22 EINVAL
sigvec(SIGTSTP, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGCONT, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGTTIN, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGTTOU, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGURG, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGLOST, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGRESERVE, 0x7f7e0820, 0x7f7e082c) ....................... ERR#22 EINVAL
sigvec(SIGDIL, 0x7f7e0820, 0x7f7e082c) ........................... ERR#22 EINVAL
sigvec(SIGXCPU, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGXFSZ, 0x7f7e0820, 0x7f7e082c) .......................... = 0
sigvec(SIGCANCEL, 0x7f7e0820, 0x7f7e082c) ........................ ERR#22 EINVAL
sigvec(SIGGFAULT, 0x7f7e0820, 0x7f7e082c) ........................ ERR#22 EINVAL
sigvec(SIGRT1, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGRT2, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGRT3, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGRT4, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGRT5, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGRT6, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGRT7, 0x7f7e0820, 0x7f7e082c) ........................... = 0
sigvec(SIGRT8, 0x7f7e0820, 0x7f7e082c) ........................... = 0
semget(0x1090522, 1, IPC_CREAT|IPC_EXCL|0644) .................... ERR#17 EEXIST
semget(0x1090522, 1, 0) .......................................... = 325
semctl(325, 0, GETPID, 0) ........................................ = 0
semget(0x1090522, 1, 0) .......................................... = 325
semctl(325, 0, GETPID, 0) ........................................ = 0
( Terminating process 7742 ("/usr/sbin/inetd -l") )

Re: Inetd Hanging

Alex Lavrov. — Tue, 26 Apr 2005 05:37:54 GMT

Can you attach the file with all the output?

I see that you have some problems with getting the semaphores ....

Re: Inetd Hanging

Paul Murray_4 — Tue, 26 Apr 2005 05:40:33 GMT

Hi Alex - attached is the full tusc output as requested.

Thanks,
Paul.

Re: Inetd Hanging

rick jones — Wed, 27 Apr 2005 10:04:34 GMT

Unless it is poor wording on the part of tusc, I would not have expected to see a message about the process terminating - at least not if you were following forks in the tusc with -f. From startup inetd will fork at least once to disconnect itself from the controlling terminal and all that.

Might want to add the "display pid" option for tusc, and perhaps add timestamps to see where it may be spending lots of time.

Re: Inetd Hanging

Stephen Keane — Wed, 27 Apr 2005 10:37:18 GMT

What does ipcs -sa give you?

Re: Inetd Hanging

Alex Lavrov. — Wed, 27 Apr 2005 10:40:20 GMT

Hmm, this part looks weird:

mmap(NULL, 86016, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x5000) = 0xc0222000
mmap(NULL, 20480, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_SHLIB, 3, 0x1a000) = 0x7f797000
close(3) ......................................................... = 0
stat("/usr/lib/libxti.2", 0x7f7e2c70) ............................ = 0
open("/usr/lib/libxti.2", O_RDONLY, 0) ........................... = 3
fstat(3, 0x7f7e2d38) ............................................. = 0
read(3, "0210010e0512@ \0\0\0\0\0\0\0\0\0".., 128) ............... = 128
lseek(3, 128, SEEK_SET) .......................................... = 128
read(3, "10\0\004\0\0\0( \001D bc\0\010\0".., 48) ................ = 48
read(3, "80\0\0\v\0\0\004\0\0\0\0", 12) .......................... = 12
lseek(3, 20480, SEEK_SET) ........................................ = 20480
read(3, "058cy 10\0\0\0a8\0\0\0p \0\0\001".., 112) ............... = 112
mmap(NULL, 86016, PROT_READ|PROT_EXEC, MAP_SHARED|MAP_SHLIB, 3, 0x5000) ERR#12 ENOMEM
close(3) ......................................................... = 0

Why the first mmap call goes OK, but then, mmap with exactly the same parameters fails with ENOMEM?

Re: Inetd Hanging

Kent Ostby — Wed, 27 Apr 2005 12:35:45 GMT

Paul -- I don't see a specific tie in to your problem, but you might want to check to see if you have the latest inetd patch which is PHNE_28312.

Best regards,

Kent M. Ostby

Re: Inetd Hanging

Ermin Borovac — Thu, 28 Apr 2005 07:23:09 GMT

From your tusc output it seems that, on startup, inetd tries to create semaphore with key 0x01090522. In your case semaphore was already created possibly by previous inetd invocation and never cleaned up (maybe due to kill -9 on inetd process).

You first need to clean up semaphore to allow inetd to start.

# ipcrm -S 0x01090522

Then you can try tusc'ing inetd again with the following flags (especially -k).

# tusc -o /tmp/tusc.log -ccc -f -k -l -n -p -v -T%T -b 80 -r all -w all /usr/sbin/inetd -l

Trace output will go to /tmp/tusc.log.