topic Re: Enema for at jobs? in Operating System - HP-UX

Enema for at jobs?

Pete Randall — Thu, 20 Jun 2002 12:34:14 GMT

My at jobs seem to be stuck:

yukon(271)root# at -l
user = ops 1024552800.a Thu Jun 20 02:00:00 2002
user = ops 1024545660.a Thu Jun 20 00:01:00 2002
yukon(272)root# date
Thu Jun 20 09:37:40 EDT 2002

Why would they not have executed and how can I get them unstuck? Any clues?

Pete

Re: Enema for at jobs?

Ian Dennison_1 — Thu, 20 Jun 2002 12:39:38 GMT

The 'jobs' command should show the actual status of these according to 'at'

ie. Are they waiting in 'at' to be started, or are they running and waiting for another event? A keypress or something?

to move a job to foreground, and resume it, use the 'fg' command, with the job number as an option,..

eg fg %2 # will resume job 2

share and enjoy! Ian

Re: Enema for at jobs?

Pete Randall — Thu, 20 Jun 2002 12:44:14 GMT

Ian,

Thanks, but jobs doesn't show anything:

yukon(273)root# jobs
yukon(274)root#

And I'm not sure how fg even relates.

Pete

Re: Enema for at jobs?

Ian Dennison_1 — Thu, 20 Jun 2002 13:04:38 GMT

How about 'at -d 1024552800.a'? what does that show?

Personally, I would trash them and start again if at all possible.

Share and Enjoy! Ian

Re: Enema for at jobs?

Pete Randall — Thu, 20 Jun 2002 13:08:00 GMT

Ian,

At -d shows the contents of the job (as expected). These jobs normally run every night by re-at'ing themselves - except last night.

Pete

Re: Enema for at jobs?

harry d brown jr — Thu, 20 Jun 2002 13:09:09 GMT

Pete,

is cron running?

live free or die
harry

Re: Enema for at jobs?

Frank Slootweg — Thu, 20 Jun 2002 13:15:27 GMT

What does /var/adm/cron/log say?

It should say something like:

> CMD: 1024582129.a
> franks 3167 a Thu Jun 20 16:08:49 METDST 2002
...
< franks 3167 a Thu Jun 20 16:10:52 METDST 2002

The first two lines are the job-id, PID (3167) and the start date/time. The third line is the end date/time.

If the first two lines are present, but not the last one, then the job was started, but still busy (and hence still in the "at -l" output), and something should be visible in the ps(1) output.

Re: Enema for at jobs?

Pete Randall — Thu, 20 Jun 2002 13:16:11 GMT

Harry,

Good question but yes, cron is running:
yukon(275)root# psg cron
root 3162 1 0 Jun 16 ? 4:26 /usr/sbin/cron

Pete

Re: Enema for at jobs?

Pete Randall — Thu, 20 Jun 2002 13:21:13 GMT

Frank,

Excellent (I'm always amazed at how I can ignore the obvious - like checking the log). The log has this entry repeatedly:
! a queue max run limit reached Thu Jun 20 00:01:00 EDT 2002
! rescheduling at job Thu Jun 20 00:01:00 EDT 2002

What is a max run limit and how do I increase it?

Pete

Re: Enema for at jobs?

Pete Randall — Thu, 20 Jun 2002 13:30:14 GMT

/var/adm/cron/queuedefs contains the following:
a.4j1n
b.2j2n90w
which seems to indicate that 4 jobs can execute simultaneously. I don't see how that could be the problem?

Pete

Re: Enema for at jobs?

Pete Randall — Thu, 20 Jun 2002 13:42:06 GMT

Looking further at the log, I can see where the jobs were started but never finished (for some unknown reason):
> ops 27094 a Mon Jun 17 02:00:00 EDT 2002
< ops 27094 a Mon Jun 17 02:00:04 EDT 2002
> ops 26677 a Tue Jun 18 00:01:00 EDT 2002
< ops 26677 a Tue Jun 18 00:42:31 EDT 2002
> ops 13671 a Tue Jun 18 02:00:02 EDT 2002 - start
> ops 5172 a Wed Jun 19 00:01:00 EDT 2002
< ops 5172 a Wed Jun 19 00:41:31 EDT 2002
> ops 21535 a Wed Jun 19 02:00:02 EDT 2002 - start
> ops 1438 a Thu Jun 20 10:33:07 EDT 2002 - start

I've purged the unfinished ones and see what happens.

Pete

Re: Enema for at jobs?

Frank Slootweg — Fri, 21 Jun 2002 08:47:41 GMT

Re: your June 20, 2002 14:42 PM GMT response:

Next time, before "purging the unfinished ones", check the *status* of the jobs. I.e., like I mentioned, the number in the log entries is the PID (process ID) of the shell that runs the job. With ps(1) you should be able to see that process and any children of that process an probably get clues as to why it/they are not finished.

[Sorry for my 'late' response. For some reason e-mail notification of new responses was not working.]

Re: Enema for at jobs?

Pete Randall — Fri, 21 Jun 2002 09:42:12 GMT

Frank,

Thanks again, that's what I'm doing this morning - it got stuck again last night. I don't see anything in the log this morning though. I wonder if my manual trimming of the log messed up cron? If nothing else, we've got some scheduled downtime coming up this noon and I expect the re-boot will straighten things out. I'll probably have to wait til Monday to find out for sure, though.

Pete

Re: Enema for at jobs?

Pete Randall — Mon, 24 Jun 2002 15:15:41 GMT

OK, here's the situation:

yukon(166)root# echo "/usr/bin/bdf -l" | at now +1 minute
warning: commands will be executed using /usr/bin/sh
job 1024934883.a at Mon Jun 24 12:08:03 2002
yukon(167)root# at -l
user = ops 1024984800.a Tue Jun 25 02:00:00 2002
user = root 1024934883.a Mon Jun 24 12:08:03 2002

In one minute I see the job kick off:
yukon(169)root# tail /var/adm/cron/log
> CMD: 1024934883.a
> root 26293 a Mon Jun 24 12:08:03 EDT 2002

However, I don't see the end of the job. I do a ps on the PID and all I see is the shell hanging around, doing nothing?

We used to use this technique daily and now it's suddenly stopped working.

Hmmm, I wonder if a patch affected it?

Re: Enema for at jobs?

Pete Randall — Mon, 24 Jun 2002 15:17:39 GMT

Sorry, I meant to put the ps output in:

yukon(170)root# psg 26293
root 26293 1647 0 12:08:03 ? 0:00 sh
root 26294 26293 0 12:08:03 ? 0:00 /usr/bin/sh

Re: Enema for at jobs?

Pete Randall — Mon, 24 Jun 2002 15:42:54 GMT

Got it !!!
PHCO_24702 introduced behavior whereby cron(1M) may no longer
execute at(1) jobs. When one of the environment variables
LC_ALL, LC_CTYPE or LANG is set when an at(1) job is
scheduled, at(1) jobs may not complete and may leave the
executing shell in a hung state.
- To work around this behavior, these three environment
variables may be unset prior to scheduling an at(1) job.
- If this work around is not acceptable, to avoid this behavior
HP recommends that PHCO_24702 be removed from all those
systems where the LC_ALL, LC_CTYPE or LANG environment
variables are set when at(1) jobs are issued. PHCO_24702
should also be removed from software depots used to installed
patches on these systems.
- The previous patch, PHCO_22767, does not exhibit this same
behavior. To ensure as many known issues as possible are
addressed, HP recommends that PHCO_22767 be installed after
PHCO_24702 is removed. If PHCO_22767 was installed prior to
PHCO_24702, it will automatically be restored when PHCO_24702
is removed and it will not need to be re-installed.

I'll try removing the patch as suggested. If that doesn't work, I'll keep on posting my musings here. If you don't see anything further from me, you can assume it worked.

Pete

Re: Enema for at jobs?

Pete Randall — Thu, 07 Nov 2002 20:11:26 GMT

By the way, this was fixed by PHCO_27141.

Pete