Booting from backup - old batch jobs

[Duane Sadowski]

Hi. I'm writing with a question regarding booting from a restored copy of one's system disk. When our system boots, disks get mounted automatically, and the batch and print queues start automatically. (The system is a standalone system, not part of a cluster.) This makes it tricky to boot from a restored backup of the system disk, which I like to do occasionally to check the integrity of the backups. Batch jobs submitted with the /AFTER qualifier that were in the queue when the backup was created will start executing if their scheduled start time has passed.

Could someone please comment on ways people avoid this problem? (For example, do people have the startup command procedure issue a prompt? Do people manually start the queues after the system is back up?)

(I have a step in the site-specific startup command procedure to skip the mounting of the disks if a particular dummy file is present. If I know before I do a particular backup that I will later want to test its integrity, I can create the dummy file before shutdown, but this doesn't seem good as a general solution.)

- Duane

[Robert Gezelter]

Duane,

I would consider using one of the user specifiable parameters (USERD1, USERD2, USER3, USER4) to the STARTUP process together with a Conversational Boot to resolve this problem.

I would then add the DCL to NOT start the queues if the parameter is set to the value that you are using to indicate a backup verification.

I hope that the above is helpful.

- Bob Gezelter, http://www.rlgsc.com

[John Gillings · Accepted solution]

Duane,

I've had exactly the opposite question from people who do BACKUP/IGNORE=INTERLOCK. Their queue manager files are never saved properly because they're open. Their complaint is that they WANT old batch jobs to start, so how can they make it happen?

Seems crazy to me but customer's are always right... I guess you can't please everyone all the time.

Robert's idea to use the USERDn SYSGEN parameters is a good one. It requires your startup procedures to check parameters and have tested code paths for each of the startup scenarios you want to cover. Human recovery procedures are also written down with all the details of EXACTLY how to restart in each circumstance, and your operations staff need the discipline to follow them precisely even when in the panic state of recovering from some kind of problem or disaster. They also need to be tested!

One school of thought, especially in disaster tolerant systems is to never let a system boot by itself from power on (default boot flags set to conversational boot), and never let disks (especially shadow sets) mount automatically. You define a number of states your systems can be in. Manually let them progress through the states only when you've verified they're ready for the next state.

It may be more prudent to invert the USERDn model and have the system boot by default in FAILSAFE mode. Have a setting to override and specify a "fast boot" if you're sure everything is ready.

[Phil.Howell]

this is in our startup, if there is someone there to answer the question as N, then it does a minimal startup, otherwise it does a "world" startup
Phil
$set_startup:
$ say " "
$ open/read oper_console opa0:
$ read -
/time_out = 10 -
/prompt = "Is this startup a world startup [Y] ?" -
oper_console startup_answer
$ close oper_console
$ if startup_answer .eqs. "" then startup_answer = "Y"
$ if startup_answer then goto not_stand_alone

[Ian Miller.]

Consider not using BACKUP/IGNORE=INTERLOCK and using DCL to save the state of the queues and jobs and restore it if necessary. There is DCL around from Keith Parris which does this.

Having jobs start when you restore is often a bad thing.

The USER system parameters are a good way to modify startup behaviour. Having systems do a conversational boot every time can work if proper procedures are in place and the operations staff follow them.

[Nic Clews]

A technique I have employed in the past is to delay the start of batch jobs after the end of start up.

The reason for this is two fold, firstly, if the system was in a crash-reboot situation, it stopped the batch jobs from slowing down the system when first logging in, and the second reason is that the batch job in question could potentially be the cause of the problems, i.e. system slowdowns or the crash! (Some batches were run at higher than normal base priority).

The technique is quite simple, autostart has to be off of course, and we're talking a stand alone system, not clustered with a remote queue manager, but one of the last lines in the system start up is a RUN/DET LOGINOUT/INPUT= and the command file has a WAIT in it. We used 5 minutes. Plenty of time to log in, stop that process.

But in your case of restoring backups, another possibility is to set the system date and time earlier than that of the backup, but you may also need to perform a MIN boot to reset passwords for access prior to the full boot to access the data you need, so this may be the point to set the clock.

[Duane Sadowski]

Thanks to all of those who provided suggestions. You provided me with a variety of alternatives and presented different philosophies.

I liked the way that two of you emphasized the human element (the potential for panic, the need for staff to follow established procedures).

The approach that seems most appealing is the one with the prompt. It's simple, although it lacks the potential power and flexibility of the scheme the "USERDn" approach.

Reflecting on what one of you wrote about the potential for panic, I'm wondering about the potential for distraction or forgetfulness at the point one would need to kill a detached process that automatically starts the batch jobs.

- Duane

[Ian Miller.]

Personally I prefer an explict start. That is someone (operator, system manager) has to do something to start the application. It should be a simple thing so it is easier to do and does not unduely delay the start.

[Duane Sadowski]

Ian, you advocate an explicit start. Can you please tell me whether you are referring only to a conversational boot with the use of USER parameters, which you discussed earlier in the thread, or could that also include a command procedure that someone needs to run manually after the system has rebooted?

- Duane

[Robert Gezelter]

Duane,

I for one prefer a process that is optionally based upon something along the lines of the USERx parameters.

My reasoning is that the normal path can be completely automated, allowing for automatic, unattended restarts. The unusual case can be managed by performing an OPTIONAL conversational boot.

This prevents the problem of being unable to restart the system (or delay the startup of the system) in the normal cases, while providing an escape hatch.

- Bob Gezelter, http://www.rlgsc.com