topic Re: help for runaway script. in Operating System - HP-UX

help for runaway script.

ben_43 — Fri, 04 Oct 2002 12:32:03 GMT

Team:

We want to create a script for killing application runaway processes. (the user logs from PC thru xemulator and we constantly having runaway). There were several useful suggestions in the forums yesterday for my question on the same lines. This would be a temporary solution but we need to implement this ASAP. What we are trying to do is automate the manual stuff which the SA is doing. These are the steps the SA does..
1. run top (to identify the runaway)
2. monitor the process which are more than 90% for 25 minutes (only aplication processes)
3. If after 25 minutes if it stays over 90% then it would be killed with a -6 flag(to produce a core) and then -15 and if not -9.
These are the steps followed. Please give your valuable ideas.

Thanks
Ben

Re: help for runaway script.

Pete Randall — Fri, 04 Oct 2002 12:52:24 GMT

Check this nearly identical question:
http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x2573402f24d5d61190050090279cd0f9,00.html

Pete

Re: help for runaway script.

Geoff Wild — Fri, 04 Oct 2002 14:35:51 GMT

This script checks for a runaway BAAAN process.

Just modify the process you want, as well as add a kill command if you like...

Just cron it for say every 5 minutes...

Rgds...Geoff

#!/bin/sh
#
# Script to monitor runaway processess
# The ba6.1 process should never accumulate more then a few seconds
# of cpu time. This helps to determine runaway processess
# Geoff Wild Feb 16, 1998
#
#
# log file: /tmp/runaways.log
logfile=/tmp/runaways.log
if [ -f $logfile ];
then
rm $logfile
touch $logfile
else
touch $logfile
fi

echo "/scripts/runaways\n ">$logfile 2>&1

#This next line checks the cpu time (field 7 in a ps -ef) and if greater
#then 1 minute, warns admin

for i in `ps -ef|grep ba6.1|awk '$7 > 1 {print $1}'`
do
echo `ps -ef|grep ba6.1|awk '$7 > 1 {print $0}'`>>$logfile 2>&1
mailx -s "Check Runaway(s)" `cat /scripts/mailadmin.list` <$logfile
done

Re: help for runaway script.

James George_1 — Fri, 04 Oct 2002 15:25:10 GMT

Hi

You can grep for the defunct process ( run away process) and kill them ..
#ps -ef | grep dfunct

# kill (PID) of the dfunct

James

Re: help for runaway script.

Robin Wakefield — Fri, 04 Oct 2002 15:40:36 GMT

Hi Ben,

This could be the makings of what you want:

#!/opt/perl5/bin/perl

$TOP="/usr/bin/top";
$file="/tmp/top.out";

$interval=25;
$limit=90;

while ( true ) {
unlink $file if ( -e $file ) ;
system("$TOP -d 1 -f $file");
$head=0;
open FH,"$file";
undef %pids;
while () {
chomp;
$head=1,next if (/COMMAND/);
next unless $head;
@a=split;
$pid=$a[1];
$cpu=$a[10];
$totcpu{$pid}+=$cpu;
$entries{$pid}++;
$pids{$pid}=1;
}
foreach $pid (keys %totcpu) {
if ( ! defined $pids{$pid} ) {
delete $totcpu{$pid};
delete $entries{$pid};
next;
}
print "PID=$pid, #entries=$entries{$pid}, #tot cpu=$totcpu{$pid}\n";
if ($entries{$pid} == $interval){
$ave=$totcpu{$pid}/$interval;
if ( $ave > $limit ) {
print "ERROR: PID $pid has an average CPU over past $interval minutes of > $limit%\n";
}
delete $totcpu{$pid};
delete $entries{$pid};
}
}
print "Sleeping for 1 minute\n";
sleep 60;
}

Basically, this runs a top every minute, and stores the top %cpu and pids in a couple of arrays. Once a process has racked up 25 entries, i.e. 25 minutes worth of data, the average CPU is calculated, and an error printed if > 90%. Obviously you'd substitute code to perform the kills. If a process is not there at the next poll interval, it's removed from the array. Anyway, try this, it's not destructive, and it's got a bit of debug code in there too.

Rgds, Robin