topic Re: Who is killing my processes? in Operating System - HP-UX

Who is killing my processes?

Mikhail_7 — Sat, 07 Jun 2008 09:22:54 GMT

In one of the our customer production HP-UX server from time to time at midnight sometimes some processes receive -TERM signal.

I know that customer have a lot of their customized own written scripts in a cron that check some alive status of the critical processes and I'm suspecting that it could be the reason.

Is it possible to turn audit and determine who is killing my processes - which script either user does it?

Thank you in advance.
BR, Michael.

Re: Who is killing my processes?

TTr — Sat, 07 Jun 2008 12:40:46 GMT

How do you know the processes are killed with a TERM signal? Is there a core file?

Re: Who is killing my processes?

Volker Borowski — Sat, 07 Jun 2008 19:55:50 GMT

Hi,

rename "kill" to "kill.program".

Write a shell-script "kill",
that dumps some usefull Information, like
- echo $(date) $(time) >> /var/log/mykill.log
- whoami >> /var/log/mykill.log
- id >> /var/log/mykill.log
- env >> /var/log/mykill.log
and after this calls
kill.program $*

Volker

Re: Who is killing my processes?

Dennis Handly — Sat, 07 Jun 2008 20:02:05 GMT

>Is it possible to turn audit and determine who is killing my processes - which script either user does it?

I'm not sure. I've had it happen to me but never was able to track it down. It has to do with heavy loads on the system. Possibly over subscribing swap space?
I claim if the kernel is going to stab you in the back like that, it should log it in syslog.

Re: Who is killing my processes?

TTr — Sun, 08 Jun 2008 13:00:13 GMT

If you are going to use a wrapper as Volker suggested, you should add the "ps -a" and the "echo $*" to the log so that you know which process is getting killed (PID) and by whom (PPID)

Re: Who is killing my processes?

Steven Schweda — Mon, 09 Jun 2008 02:34:22 GMT

"man 5 audit"?

[...]
process Log all operations on processes, including
exit(2), fork(2), kill(2), mlock(2), mlockall(2),
munlock(2), munlockall(2), setcontext(2),
setrlimit64(2), sigqueue(2), ulimit64(2), and
vfork(2).
[...]

I know nothing, but it looks promising.

Re: Who is killing my processes?

Laurent Menase — Mon, 09 Jun 2008 07:11:20 GMT

There is also a support kernel instrumentation available only through hp support

Re: Who is killing my processes?

Mikhail_7 — Mon, 09 Jun 2008 11:51:30 GMT

Thank you for the advices.

I've written new kill script and have replaced it with the original one at /usr/bin/kill, here it is:

#!/bin/sh
export UNIX95=1
date >> /var/tmp/killBill.log
ps -efxH >> /var/tmp/killBill.log
echo $* >> /var/tmp/killBill.log
echo "\n" >> /var/tmp/killBill.log
/usr/bin/kill_orig $*

Unfortunately it does not write anything to log. I have an assumption that kill signal is sending via the kernel and does not uses /usr/bin/kill at all.
Isn't it?

Could you tell a bit more regarding
kernel instrumentation feature?

Thank you.

Re: Who is killing my processes?

Dennis Handly — Mon, 09 Jun 2008 12:08:30 GMT

>Unfortunately it does not write anything to log.

Some application could be calling kill(2) directly.

>I have an assumption that kill signal is sending via the kernel and does not uses /usr/bin/kill at all.

That was my assumption for my reply. Are you using chatr(1) or mmap(2) options to oversubscribe swapspace?

>regarding kernel instrumentation feature?
Why not try Steven's audit(5) suggestion first?

Re: Who is killing my processes?

Mikhail_7 — Mon, 09 Jun 2008 14:56:57 GMT

>>Unfortunately it does not write anything to log.

>Some application could be calling kill(2) directly.

I've tried to use: kill -TERM xxx from console and even this is not using /usr/bin/kill
Take a look below:

msklabss:/ (211) root% mx | grep mecsla
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME COMD
1 S 401 1699 3203 0 154 20 5336f380 243 4c79d6e ? 0:00 mecslamx
1 S 401 1729 3203 0 154 20 76c3f1c0 231 65678ee ? 0:00 mecslamx
1 S 401 1770 3203 0 154 20 5c3dae40 231 7428dae ? 0:00 mecslamx

msklabss:/ (212) root% kill -TERM 1699
msklabss:/ (213) root% kill -9 1729

msklabss:/ (214) root% ll /var/tmp/killBill.log
-rw-rw-rw- 1 root sys 0 Jun 9 19:38 /var/tmp/killBill.log

msklabss:/ (215) root% !mx
mx | grep mecsla
F S UID PID PPID C PRI NI ADDR SZ WCHAN TTY TIME COMD
1 S 401 1770 3203 0 154 20 5c3dae40 231 7428dae ? 0:00 mecslamx

msklabss:/ (216) root% ll /var/tmp/killBill.log
-rw-rw-rw- 1 root sys 0 Jun 9 19:38 /var/tmp/killBill.log

msklabss:/ (217) root% /usr/bin/kill -TERM 1770
msklabss:/ (218) root% ll /var/tmp/killBill.log
-rw-rw-rw- 1 root sys 76848 Jun 9 19:40 /var/tmp/killBill.log

msklabss:/ (219) root% which kill
/usr/bin/kill

msklabss:/ (220) root% cat /usr/bin/kill
#!/bin/sh
export UNIX95=1
date >> /var/tmp/killBill.log
ps -efxH >> /var/tmp/killBill.log
echo $* >> /var/tmp/killBill.log
echo "\n" >> /var/tmp/killBill.log
/usr/bin/kill_orig $*
msklabss:/ (221) root%

>>I have an assumption that kill signal is sending via the kernel and does not uses /usr/bin/kill at all.

>That was my assumption for my reply. Are you using chatr(1) or mmap(2) options to oversubscribe swapspace?

I do not use chatr or mmap options to oversubscrive swapspace,
as I know that we do not have problem with system overload either heavy swap space usage as well.
In our case I'm suspecting that some user-written cron script sometime kills right good processes by mistake either some user is doing that.

>>regarding kernel instrumentation feature?
>Why not try Steven's audit(5) suggestion first?

And here we have a problem as well, because to switch on audit feature hp-ux has request to convert to a Trusted System before proceeding and we can't do it as well.

Re: Who is killing my processes?

Steven Schweda — Mon, 09 Jun 2008 16:01:15 GMT

> I've tried to use: kill -TERM xxx from
> console and even this is not using
> /usr/bin/kill

"kill" is a shell built-in command for many
shells.

"man sh".
"man sh-posix" (or whatever).

And any progran can call kill().

"man 2 kill".

> And here we have a problem as well, because
> to switch on audit feature hp-ux has
> request to convert to a Trusted System
> before proceeding and we can't do it as
> well.

Somtimes you need to use a better operating
system to get your work done.

Re: Who is killing my processes?

Mikhail_7 — Fri, 27 Jun 2008 19:48:00 GMT

I still did not find any suitable solution in my case.