audisp - making use of what it is telling me

[Leon Allen]

I'm trying to track down a problem whereby a group of processes (Oracle) suddenly crash with no warning apart from a trace file which says:
Received unhandled signal: 15, code=800003ffbfff3338
Terminating.

In response to a prior thread last month, I turned on auditing of kill, and have waited patiently since.

Well, the problem has just reoccurred, after several weeks, our entire production oracle environment crashed. Nothing in any alert log or system log anywhere. But, I've captured the event in the audit tracing I turned on (via SAM), see audisp output below....

All ttys are selected.
Selecting successful & failed events.
TIME PID E EVENT PPID AID RUID RGID EUID EGID TTY

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 09:07:32 14778 F 37 14775 0 0 3 0 3 ?????
[ Event=kill; User=root; Real Grp=sys; Eff.Grp=sys; ]

ERRNO = 3; RETURN_VALUE 1 = -1;
PARAM #1 (int) = -14779
PARAM #2 (int) = 1
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 14:11:52 23549 S 37 23548 0 102 106 102 106 ?????
[ Event=kill; User=root; Real Grp=chris; Eff.Grp=chris; ]

RETURN_VALUE 1 = 0;
PARAM #1 (int) = -1
PARAM #2 (int) = 15
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 14:11:52 23549 S 37 23548 0 102 106 102 106 ?????
[ Event=kill; User=root; Real Grp=chris; Eff.Grp=chris; ]

RETURN_VALUE 1 = 0;
PARAM #1 (int) = 158
PARAM #2 (int) = 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 14:11:53 23549 S 37 23548 0 102 106 102 106 ?????
[ Event=kill; User=root; Real Grp=chris; Eff.Grp=chris; ]

RETURN_VALUE 1 = 0;
PARAM #1 (int) = 20806
PARAM #2 (int) = 15
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 14:11:53 23549 S 37 23548 0 102 106 102 106 ?????
[ Event=kill; User=root; Real Grp=chris; Eff.Grp=chris; ]

RETURN_VALUE 1 = 0;
PARAM #1 (int) = 8687
PARAM #2 (int) = 15
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 14:17:11 23548 S 37 1 0 0 3 0 3 ?????
[ Event=kill; User=root; Real Grp=sys; Eff.Grp=sys; ]

RETURN_VALUE 1 = 0;
PARAM #1 (int) = 21010
PARAM #2 (int) = 14
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 14:17:11 23548 S 37 1 0 0 3 0
3 ?????
[ Event=kill; User=root; Real Grp=sys; Eff.Grp=sys; ]

RETURN_VALUE 1 = 0;
PARAM #1 (int) = 21011
PARAM #2 (int) = 14
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
060207 14:21:05 22036 S 37 21686 0 0 3 0 3 pts/tb
[ Event=kill; User=root; Real Grp=sys; Eff.Grp=sys; ]

RETURN_VALUE 1 = 0;
PARAM #1 (int) = 23548
PARAM #2 (int) = 0
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can see at 14:11, there was a kill -1 15 issued. This is my prime suspect for the cause of the crash.

I can see some useful uid and group id info, but it's a little bit generic (our HR / Payroll system - Oracle back end).

I can also see some pid and ppid information.

Q. HOW CAN I GET MORE INFORMATION ON THESE PID PPID'S, eg when did they start, the command that might have started them. I just need a bit more info to really 'point the finger'

Will wtmp tell me anything?

Is there more in audisp to say when / how / where these pid's started?

Our HR system (CHRIS) is accessed through both a 'std' application, and through the web (apache fast cgi on HP11i rp5430 server.)

How might I disallow this uid gid from using the kill command?

Extract of passwd:
root:*:0:3::/home/root:/sbin/sh
oracle:*:102:102:Oracle UNIX User,33 King St,116,0438 400 891:/home/oracle:/usr/bin/sh
chriscs:*:108:106:Chris Payroll,,,:/home/chriscs:/usr/bin/ksh

group file :
chris::106:oracle,root,techone
dba::102:techone,chriscs

Your thought again would be greatly appreciated.

[paolo barila]

Hi,

you can't deny root from kill-ing processes.

Pablo

[Leon Allen]

I have discover what process is issuing the kill 15 -1 :- it is httpd!

So, it seems Apache is periodically killing all oracle processes! How can I prevent this from happening??

[paolo barila]

Hi,
httpd (apache) must run with a different user from oracle (usually www) and the two users shouldn't be in the same group.

Pablo

[Leon Allen]

Thankyou so much pablo for your valued input into my problem.

http starts under root, then switches to oracle/oinstall in the httpd config file.

I will change this to apache, as per installation guide and post feedback on how it goes.