I have some unkillable processes

[J.A.R. Karremans]

I have a few processes on HPUX 11i which seem unkillable;

oracle 23940 1 0 10:53:55 ? 0:00 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 28934 1 0 Apr 22 ? 0:02 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 28906 1 0 Apr 22 ? 0:02 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 23938 1 0 10:53:43 ? 0:00 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 23882 1 0 10:51:53 ? 0:00 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 23935 1 0 10:53:40 ? 0:00 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 29823 1 0 Apr 22 ? 0:02 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 23860 1 0 10:50:35 ? 0:00 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 6648 1 0 Apr 17 ? 0:07 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 10761 1 0 Apr 15 ? 0:07 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 25899 1 0 Apr 18 ? 0:04 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A
oracle 27550 1 0 Apr 22 ? 0:02 /oracle/8.1.7/Apache/Apache/bin/httpd -d /oracle/8.1.7/Apache/A

They won't respond to a kill or even a kill -9 but I need them gone and I don't wanna (jack, that's windows) reboot the machine.

Who has the answer?

[James R. Ferguson]

Hi:

A process that cannot be killed with 'kill -9' represents a process that is waiting on event (like an I/O). To be killable, a process must first be runnable.

Regards!

...JRF...

[J.A.R. Karremans]

Hi there James,

Right! Okay! I get that.

Then it could be that a directory is being held by something (directory being /Backbase) but I have nog clue how to figure out what would be responsible for keeping this directory in clamp...
Sure, a reboot would free up things, but I cannot afford that...

[Rita C Workman]

The processes are "oracle". So I might ask, is the oracle database down? If it is down then oracle (your DBA) did not clean up it's processes completely - as they should.

[Comment on kill -9. As Jim says this is the silver bullet of kill commands. But even when it kills a command - that doesn't mean things are left hanging around. Doing this can result in zombie/defunct processes that require a reboot to clear]

So you can run a few things to check and see what you can do, and maybe avoid a reboot.

I tend to go to basics:

who -u who's on this box - get out if it's not root.

fuser -cu / who's using the problem filesystem. This will give you user and PID. See results and kill what you can.

ipcs -a this will give you all the queue,semaphore,memory that is allocated. You can kill those oracle processes using:
ipcrm -

Hope this gives you something to help avoid a reboot - but that depends...

Rgrds,
Rita

[RobinKing]

try using fuser of lsof to see processes using that directory

[Rita C Workman]

Correction to my above stmt...

Should say.........."that doesn't mean things aren't left hanging around."

Sorry for the typo,
Rita

[J.A.R. Karremans]

All info tells that there would be something wrong with the /Backbase directory...

fuser /Backbase comes in the hang
All Oracle databases are up, no worries, but it's Apache which is bothered with this dir. And those processes were the first to hang.
I cannot go around killing off Oracle processes, they could be database processes...

The processes that are also blocked are a find-process and perhaps some other stuff...

[Yogeeraj_1]

hi,

goto /sbin/init.d and determine which script launched these processes.

e.g.

SRV2:sbin/init.d> grep -i oracle *

or

SRV2:sbin/init.d> grep -i 8.1.7 *



and then try to run:

./<script name> stop


this would be cleaner that just doing a 'kill -9'


good luck!
kind regards
yogeeraj

[Bill Hassell · Accepted solution]

Since most modern databases such as Oracle are network based, this is a classic network hang caused by waiting (without a timeout) on a network communication to some other process. As mentioned, the program cannot run until the I/O is completed and it never will. You'll have to shutdown Oracle and related processes and if that doesn't get rid of the hang, reboot. kill -9 is an absolute last resort and if it was used instead of a proper termination of some processes, that is likely the reason for the hang. kill -9 is a guaranteed way to corrupt data and leave orphaned processes around that cannot be killed. Always use kill -15 first, then kill -1. Quality software will see the signal and perform the appropriate actions for a termination.

[J.A.R. Karremans]

Hi there Bill,
Good to meet you on-line.

The /Backbase directory is used by Apache, on it's turn using modplsql to access several Oracle databases (across the network!). It could be that one of those are the reason...
Stopping the local Apache dispatcher (stopping Apache) doesn't kill the Apache childs (as could be expected) but they aren't killable either. I guess I am not over-using kill -9, but even this doesn't help...

I am still not completely sure what's buggin' me and where it's comin' from... More and more I am getting convinced though that a reboot will be my only way out of this death-trap... And I don't like that one bit.

Regards,
Jan

[Yogeeraj_1]

hi Jan,

you may also wish to look into your access (access_log.xxx) and error (error_log.xxx) logs which are found in /oracle/8.1.7/Apache/Apache/logs


anything abnormal at the httpd process level should be visible there...

just some thoughts

kind regards
yogeeraj

[J.A.R. Karremans]

Thank!
I did take a peek... But there was nothing abnormal to be found.
Anyways, now Apache will not start anymore because of running childs the Apache starter process cannot bind to port 3339.

[Dennis Handly]

>Rita: Doing this can result in zombie/defunct processes that require a reboot to clear

kill -9 can't create zombies, only orphans. It can't kill them directly. But killing the zombie master will cause init to reap the zombies.