Script NOT in rcN.d executed after "shutdown -r"

[Ken Polton]

I don't know if this can be replicated; personally I hope never to see it again. Some of you who do more shutdowns than I do may have some idea how it could happen...

An ordinary shell script usually runs all day until midnight, submits an atjob to restart itself at 05:00 and shuts down. If halted during the day, it does not submit the atjob and has to be started from a root telnet session. A simple "running flag" technique ensures only one instance of the script runs at a time.

Before an innocuous server shutdown to increase the size of /var (nothing to do with the script in question), I halted the shell script tidily and watched it finish.

Then, I "shutdown -y now" into single-user mode; adjusted /var; "shutdown -r -y now" to restart.

I was watching the web console as everything came up and, a couple of seconds after the login prompt appeared, I was being bombarded by mail telling me that some functions in this shell script were failing to execute. "ps -ef" showed four instances of the script had been invoked, essentially simultaneously (they'd all got in before the "running_flag" got set).

The shell script is most certainly not linked in /sbin/rcN.d directories, which have only had two amendments since the server was installed.

There are no hard-wired terminals and, anyway, I can't imagine any buffers surviving a server shutdown. Only a couple of us have the root password and I was the only one around at the time.

A bit of a thought experiment, then, but your hypotheses will be welcome.

Ken

[Ravi_8]

hi,
checkout in /etc/inittab file.

[Rainer von Bongartz]

Other place where things get started is /etc/inittab

[Ken Polton]

I hadn't thought to check /etc/inittab, so thanks for the reminder; but it has been unchanged since February and has nothing in it that I can associate with this event.

[John Palmer · Accepted solution]

Have you checked if they were started by cron?
/var/adm/cron/log will list the start of at jobs with a line like:-
> CMD: 994257000.a

where 994257000 is the start date in seconds since the epoch for the job.

[Ken Polton]

I'm grateful, John.
The following is the top of /var/adm/cron/log:-

! *** cron started *** pid = 986 Tue Jul 3 09:54:43 GMT 2001
> CMD: 993790800.a.security
> root 998 a Tue Jul 3 09:54:43 GMT 2001
> CMD: 993790800.a.security.2
> root 999 a Tue Jul 3 09:54:43 GMT 2001
> CMD: 993877200.a.security
> root 1000 a Tue Jul 3 09:54:43 GMT 2001
> CMD: 993877200.a.security.2
> root 1001 a Tue Jul 3 09:54:43 GMT 2001
! a queue max run limit reached Tue Jul 3 09:54:43 GMT 2001
! rescheduling at job Tue Jul 3 09:54:43 GMT 2001
! a queue max run limit reached Tue Jul 3 09:54:43 GMT 2001
! rescheduling at job Tue Jul 3 09:54:43 GMT 2001
! a queue max run limit reached Tue Jul 3 09:54:43 GMT 2001
! rescheduling at job Tue Jul 3 09:54:43 GMT 2001
! a queue max run limit reached Tue Jul 3 09:54:43 GMT 2001
! rescheduling at job Tue Jul 3 09:54:43 GMT 2001
! a queue max run limit reached Tue Jul 3 09:54:43 GMT 2001
! rescheduling at job Tue Jul 3 09:54:43 GMT 2001
! a queue max run limit reached Tue Jul 3 09:54:43 GMT 2001
! rescheduling at job Tue Jul 3 09:54:43 GMT 2001



...and I can relate the files "security" and "security.2" to the event.

Any ideas on what the queue max run limit is, and why if there is any reason it should be reached during a shutdown?

[Thierry Poels_1]

Hi,
the "queuedefs" file defines how many at-, batch- & cronjobs can be executed simultaneously.
The a-queue is for atjobs, default simultaneous jobs is 4 (you can see in the log that 4 jobs at started immediated, the others will wait for a specific time and then try again)
b-queue is for batchjobs, c-queue for cronjobs.
Other queue can be defined if needed, and you can select the queue of your choice when you submit the atjob.

regards,
Thierry.