process monitoring

[lawrenzo]

Hello,

I have certain issues here with jobs that are run from a cron schedule .... I have over 250 jobs with various levels of importance and impact and I want to look at putting monitoring around these jobs that run.

I have created a script that checks for the most critical jobs and alerts around them wether they have completed in time or have started - this only alerts on about 5 jobs and cron starts the check. I dont want to use cron to check all 250 jobs.

My question is .....

Is there any free software that will run these checks for all 250 schedules that can be programmed for average run time and alerts if the job doesn't start or hasn't finished on time?

or

If I was to code it what is the best way to run a script that checks all these process's and there start and finish times? Has anyone got some examples?

any help will be greatly appreciated.

Thanks

[Steven E. Protter · Accepted solution]

Shalom,

Not free: Glance does a pretty good job of process tracking.

Free: tusc
http://hpux.connect.org.uk/hppd/hpux/Sysadmin/tusc-7.8/

Be careful, the Q&A on the above link is suspect. HP maintains binary versions on it ftp servers though you may need to open a response center call to gain access.

SEP

[Peter Nikitka]

Hi,

for jobs started by cron there are valuable entries in the cron-logfile:
It contains information about the command submitted, including the username, the start- and endtime as well as its non-zero exit-status.

I think this logfile is worth reading/parsing.

mfG Peter

[spex]

Hi Lawrenzo,

Consider using 'timex'. Call a job with 'timex ' and save the the output line containing 'real '. Then pass the command name and real time to a script that stores acceptable runtimes for various commands. If a command takes too long to execute, alert the sysadmin.

Alternatively, you could store the output of the 'date' command at the beginning of the job and compare that value to the value after the job's completion to get the duration. Remember to account for the day changing.

Of course, these options are really only feasible if your jobs are or can made to be called by shell scripts.

PCS

[Bill Hassell]

As mentioned, the cron log file is a good start. It shows the command and PID with the start time. Then scan for the same PID to find the stop time. However, a better solution is to add a timestamp to each job using logger. Then use a script to summarize the completion time for each job and track the variations.

Note that this is the simplest method. But this assumes that none of the jobs hang or never complete. If the job is important, the front end of each job should first make sure that the previous run finished properly.

[Emil Velez]

DO you have OVPA (measureware) on the system.. I can be configured to track all "new" processes on a system and every minute you get a record for the process. So you can analyze the logs to see when a process started and died and see the resources it tool. YOu would have to take the nonew keyword out of the parm file to do that.

[Arturo Galbiati]

Hi,
use can use the log file for crontab (as guusgest) and see if teh start time contained into this matching teh star time in crontab. To see the log file for crontab type man crontab and look under the FILES section. In my case (HP-UX11i) the file is
/var/adm/cron/log:

FILES
/var/adm/cron Main cron directory
/var/adm/cron/cron.allow List of allowed users
/var/adm/cron/cron.deny List of denied users
/var/adm/cron/log Accounting information
/var/spool/cron/crontabs Directory containing the crontab files

To check about the 'strange' duration the only sure way is to touch a file when the script finished. I suggest you to develop a simple script as like this:

can x
#/usr/bin/ksh
script_to_run=$1
touch $script_to_run.BEG
$script_to_run 1>$script_to_run.out 2> $script_to_run.err
touch $script_to_run.END

and for all teh script in crontab use 'x' to run them.

This is only and example, you can improve the script as you prefere.

HTH,
Art