Cron Question

[Ali Imran Abbas]

How can we see if the commands or jobs scheduled by cron are successful or failed?

I have ITO configured for performance monitoring and its giving me status signals about CRON jobs on various machines that they are failing. Following is an example about the messages I am getting on ITO about CRON jobs on various systems:


Dup. Date/Time Auto St. Oper St. Sev. Message Group Node Name
----- ----------------- -------- -------- ---- ---------------- -------------------

01/11/08 08:40:11 undef defined norm Performance usunx31.reebok.com
Message Text : End of Memory Bottleneck Alert ( START: 01/11/2008
07:10:00 ; END: 01/11/2008 08:40:00 )

01/11/08 08:41:30 undef undef warn Job usunx08.reebok.com
Message Text : Cron command of prdadm with pid 14871 failed (rc = 1)

01/11/08 08:45:03 undef undef warn Job usunx29.reebok.com
Message Text : Cron command of prdadm with pid 29212 failed (rc = 1)

01/11/08 08:45:06 undef undef warn Job usunx33.reebok.com
Message Text : Cron command of prdadm with pid 13236 failed (rc = 1)

01/11/08 08:45:06 undef undef warn Job usunx01.reebok.com
Message Text : Cron command of conadm with pid 13412 failed (rc = 1)

01/11/08 08:45:15 undef undef warn Job usunx31.reebok.com
Message Text : Cron command of bwdadm with pid 19276 failed (rc = 1)

01/11/08 08:45:16 undef undef warn Job usunx70.reebok.com
Message Text : Cron command of bwtadm with pid 17027 failed (rc = 1)

01/11/08 08:45:18 undef undef warn Job usunx15.reebok.com
Message Text : Cron command of prdadm with pid 7244 failed (rc = 1)





Selected active Page: 2

Dup. Date/Time Auto St. Oper St. Sev. Message Group Node Name
----- ----------------- -------- -------- ---- ---------------- -------------------

01/11/08 08:45:38 undef undef warn OS RBKUSAPP10.reebok.com
Message Text : The Node RBKUSAPP10 is up but there is 20% packet loss
events

01/11/08 08:50:09 undef defined norm Performance usunx08.reebok.com
Message Text : End of CPU Bottleneck Alert ( START: 01/11/2008
08:40:00 ; END: 01/11/2008 08:50:00 )

01/11/08 08:55:09 undef defined norm Performance usunx47.reebok.com
Message Text : End of CPU Bottleneck Alert ( START: 07:35:00
01/11/08 ; END: 08:50:00 01/11/08 )

01/11/08 08:55:13 undef defined crit Performance usunx47.reebok.com
Message Text : Network Bottleneck probability= 95.00% ( START:
16:30:00 01/10/08 ; REPEAT: 08:50:00 01/11/08 )

01/11/08 09:00:02 undef undef warn Job usunx29.reebok.com
Message Text : Cron command of prdadm with pid 3538 failed (rc = 1)

01/11/08 09:00:03 undef undef warn Job usunx08.reebok.com
Message Text : Cron command of prdadm with pid 20786 failed (rc = 1)

01/11/08 09:00:04 undef undef warn Job usunx33.reebok.com
Message Text : Cron command of prdadm with pid 16828 failed (rc = 1)

01/11/08 09:00:08 undef undef warn Job usunx01.reebok.com
Message Text : Cron command of conadm with pid 15044 failed (rc = 1)

01/11/08 09:00:11 undef defined mino Performance usunx33.reebok.com
Message Text : Network Bottleneck probability= 70.00% ( START:
01/11/2008 09:00:00 )

01/11/08 09:00:15 undef undef warn Job usunx70.reebok.com
Message Text : Cron command of bwtadm with pid 19366 failed (rc = 1)

01/11/08 09:00:17 undef defined crit Performance usunx31.reebok.com
Message Text : Network Bottleneck probability= 95.00% ( START:
01/11/2008 07:10:00 ; REPEAT: 01/11/2008 09:00:00 )

01/11/08 09:00:18 undef defined norm Performance usunx45.reebok.com
Message Text : End of Network Bottleneck Alert ( START: 01/10/2008
20:10:00 ; END: 01/11/2008 09:00:00 )

01/11/08 09:00:19 undef undef warn Job usunx31.reebok.com
Message Text : Cron command of bwdadm with pid 21377 failed (rc = 1)





Selected active Page: 3

Dup. Date/Time Auto St. Oper St. Sev. Message Group Node Name
----- ----------------- -------- -------- ---- ---------------- -------------------

01/11/08 09:00:20 undef undef warn Job usunx15.reebok.com
Message Text : Cron command of prdadm with pid 10958 failed (rc = 1)

Can any one give me idea where to investigate these problems from?

[Patrick Wallek]

You could start by checking /var/adm/cron/log though that probably won't tell you much more than you already know.

By default cron will e-mail any output from a cron job to the user that the job is running as. So in your case check the conadm, prdadm and bwdadm users on the system running the jobs and see if they have mail. If they do, that may provide some clues.

Ideally any cron job should either write its output to a log file so it is easier to investigate issues.

[Ali Imran Abbas]

Thanks for your reply, How can I check the logs for cron jobs for individual users.
Follwoing is a job by one of the user:

# dayly cleanup of CCMS and Tivoli-logs
00 05 * * * $HOME/sapcoc/ccms_del_old_data.sh >/dev/null 2>&1
#
# Check - if monitoring is activ
00,15,30,45 * * * * $HOME/sapcoc/auto_checkmon DVEBMGS00 usunx70 >/dev/null 2>&1

How can I troubleshoot using this as guideline? What this job is doing?

[Patrick Wallek]

************
Thanks for your reply, How can I check the logs for cron jobs for individual users.
Follwoing is a job by one of the user:

# dayly cleanup of CCMS and Tivoli-logs
00 05 * * * $HOME/sapcoc/ccms_del_old_data.sh >/dev/null 2>&1
#
# Check - if monitoring is activ
00,15,30,45 * * * * $HOME/sapcoc/auto_checkmon DVEBMGS00 usunx70 >/dev/null 2>&1

How can I troubleshoot using this as guideline? What this job is doing?
************

It's going to be real hard to check these. All output from the cron job is being redirected to /dev/null.

As to what this job is doing -- I have no clue, you'll have to examine the scripts that the cron jobs themselves to see.

[Ali Imran Abbas]

Its a long script, How can I redirect its output to some logfile, can u pls give me an example?

[Patrick Wallek]

First, it sounds as if you need to study up on redirection, especially redirecting standard out and standard error.

For this script:

$HOME/sapcoc/auto_checkmon DVEBMGS00 usunx70 >/dev/null 2>&1

Take the '>/dev/null 2>&1' and change it to something like: '> /var/tmp/checkmon.log 2>&1'

So your line looks like:

$HOME/sapcoc/auto_checkmon DVEBMGS00 usunx70 > /var/tmp/checkmon.log 2>&1

Basically you are just replacing the '/dev/null' in the line with a real file name. Do this for any other scripts you want to see output from.

[Ali Imran Abbas]

Do I need to restart the cron after making modifications to that user cron job?

[Patrick Wallek]

No.