Queue manager not starting on boot (VAX, VMS 7.1)

[Willem Grooters]

We encountered a problem in testing a fail-over scenario, where the queue manager failed to start during system startup - see attachement.

In order to re-enable the queueus we tried it once the system was up and running.

$ START/QUEUE/MANAGER didn't work either, the message "%JBC-E-QMANNOTSTARTED, queue manager could not be started" is reported.

$ START/QUEUE/MANAGER/NEW did re-create a new environment, but I would have to re-define all forms and re-create all queues.
But since the previous files were still there, it was attempted to stop the queue manager, remove the newly created files QMAN$MASTER.DAT,SYS$QUEUE_MANAGER.QMAN$QUEUES and SYS$QUEUE_MANAGER.QMAN$JOURNAL and restart the queue manager.

STOP/QUEUE/MANAGER was issued, the files removed, but SYS$QUEUE_MANAGER.QMAN$JOURNAL was "locked by another user" - as I found out, job_control and QUEUEMANAGER processes were still running. I killed these two and removed the file, and I issued START/QUEUE/MANAGER; but this failed because of a DEVOFFLINE message: Device off line or not configured.

(I checked formentioned files on the existance of some fixed, hard device specification but found none. So what device is mentioned, remains unclear)

$ SHOW QUEUE now hangs and must be aborted using ^C. Even SHUTDOWN fails, when trying to stop the queues, QUEMAN is activated and hangs, so SHUTDOWN does not continue.

We found that file SYS$QUEUE_MANAGER.QMAN$JOURNAL was missing in the image backup from which the bootdevice was restored. We did find this file in another image backup of the same sourcedisk, taken a month earlier; since the file seems to be re-created on several occasions, it might be a valid environment.

So we still have a few questions on this test that need to be answered.
* A missing SYS$QUEUE_MANAGER.QMAN$JOURNAL file should not keep the queuemanager to start; this file is re-created (as has been observed)
* After START/QUEUE/MANAGER/NEW, a new queue manager database is created; but STOP/QUEUE/MANAGER does not seem to work, or doesn't work as expected (of course, I could expect the wrong behaviour :))
* What could have caused the DEVOFFLINE error later on? What device?

One more thing:
There was a minor problem in DECNet connectivity from another system to the system under test. We found that the original production system was stopped, but not switched off, nor has it been disconnected from the network. Since DECNet changes the MAC address of the NIC, the same MAC address may have been used on both systems. Could that have caused a problem?

[Willem Grooters]

Attachement....

[Volker Halle]

Willem,

to completely stop the QUEUE_MANAGER, you need to specify STOP/QUE/MANA/CLUSTER - even on a standalone node.

Volker.

[Hakan Zanderau ( Anders]

Willem,

Are you running a cluster ?
Is the logical QMAN$MASTER used ? ( database relocated )

The Journalfile contains all jobs on all queues. You don't restore the journalfile from a backup. It its missing, an empty file will be created when the queuemanager starts.

Hakan

[Volker Halle]

Willem,

the %QMAN-E-OPENERR -RMS-E-FEX -SYSTEM-W-DUPFILENAME will most likely be caused, if there are 2 versions of SYS$QUEUE_MANAGER.QMAN$JOURNAL file !

The QUEUE_MANAGER, when it re-creates the .QMAN$JOURNAL file, ALWAYS re-creates the file as ;1. If you had an existing ;1 and you had copied another file as ;2, this error can be explained.

If you want an up-to-date copy of your queue, forms, jobs etc., you need to move both files together. Note that if you move them separately to another disk, you will most likely loose your pending jobs, as the information stored is the device-name and file-id !

It may be better to provide some procedures to automatically re-create missing batch-jobs, if you move to your failover system.

Volker.

[Hoff]

The FIXQUE.COM procedure on the Freeware is useful here. That procedure can usually extract and reconstitute a queue database, and potentially from a damaged queue database.

Don't use and try to use more than one queue database in a cluster; that causes all sorts of bad.

[Jess Goodman]

FIXQUE.COM has several deficiencies - to use a polite term. Jon Pinkley has documented these.

I have attached as a .ZIP file with the latest versions of my complete package of robust queue management tools. This package includes:

QUEUE_COMMANDS.COM - see thread http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1303169
DISPLAY_JOBS.COM - see thread http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1149896
DISPLAY_CHARACTERISTICS.COM - generates DEFINE/CHAR commands
DISPLAY_FORMS.COM - generates DEFINE/FORM commands
DISPLAY_QUEUES.COM - generates all commands necessary to recreate queues.
QUEUE_MANAGER_SAVE.COM - see below:

$ SET NOON !QUEUE_MANAGER_SAVE.COM
$! This is run from a daily batch job (preferably on the nodes the queue
$! manager runs on) to save, as DCL commands, all information we need to
$! recreate the queue manager database and resubmit scheduled batch jobs.
$! Dec. 2008 Jess Goodman - First pass
$!
$ MAIL_TO = P1 !Optional: mail address(es) to notify of changes
$!
$!Save SUBMIT commands for current batch jobs that were submitted /AFTER=.
$ @COM:[SYSTEM]DISPLAY_JOBS /OUTPUT=SYS$MANAGER:SUBMIT_JOBS.COM -
* * * VERSION,USER,DELAYED
$ PURGE SYS$MANAGER:SUBMIT_JOBS.COM /KEEP=5
$!
$!Save DEFINE/CHAR commands for current characteristics.
$ OUTFILE := SYS$MANAGER:DEFINE_CHARACTERISTICS.COM
$ @COM:[SYSTEM]DISPLAY_CHARACTERISTICS /OUTPUT='OUTFILE' * /SAFE
$ GOSUB CHECK_FOR_CHANGES
$!
$!Save DEFINE/FORM commands for current forms.
$ OUTFILE := SYS$MANAGER:DEFINE_FORMS.COM
$ @COM:[SYSTEM]DISPLAY_FORMS /OUTPUT='OUTFILE' * /SAFE
$ GOSUB CHECK_FOR_CHANGES
$!
$!Save INIT/QUEUE/BATCH commands for current batch queues.
$ OUTFILE := SYS$MANAGER:INIT_BATCH_QUEUES.COM
$ @COM:[SYSTEM]DISPLAY_QUEUES /OUTPUT='OUTFILE' * /BATCH/NOSTART/NOID
$ GOSUB CHECK_FOR_CHANGES
$!
$!Save INIT/QUEUE/DEVICE= commands for current symbiont queues.
$ OUTFILE := SYS$MANAGER:INIT_DEVICE_QUEUES.COM
$ @COM:[SYSTEM]DISPLAY_QUEUES /OUTPUT='OUTFILE' * /DEVICE/NOASSIGN/NOSTART/NOID
$ GOSUB CHECK_FOR_CHANGES
$!
$!Save ASSIGN/QUEUE commands for current logical queues.
$ OUTFILE := SYS$MANAGER:ASSIGN_QUEUES.COM
$ @COM:[SYSTEM]DISPLAY_QUEUES /OUTPUT='OUTFILE' */DEVICE/GENERIC/ASSIGN_ONLY
$ GOSUB CHECK_FOR_CHANGES
$!
$ EXIT
$!
$CHECK_FOR_CHANGES:
$ IF (F$SEARCH("''OUTFILE';-1") .NES. "")
$ THEN
$ IF (MAIL_TO .NES. "")
$ THEN
$ DIFFERENCE /OUTPUT=NL: 'OUTFILE' !Any changes today?
$ IF ($STATUS .NE. %X006C8009) THEN -
$ MAIL /SUBJ="''OUTFILE' changed" NL: 'MAIL_TO'
$ ENDIF
$ IF (F$SEARCH("''OUTFILE';-2") .NES. "")
$ THEN !Are previous 2 versions identical?
$ DIFFERENCE /OUTPUT=NL: 'OUTFILE';-1
$ IF ($STATUS .EQ. %X006C8009)
$ THEN !yes, save oldest
$ DELETE 'OUTFILE';-1
$ RENAME 'OUTFILE' ;0
$ ENDIF
$ ENDIF
$ ENDIF
$ RETURN

Using these tools you:
* Can easily reSUBMIT a scheduled batch job that is no longer on a queue for some reason.
* Have a complete backup of all queue manager data in case of corruption.
* Can easily create a duplicate of your queues, forms, characteristics, and scheduled batch jobs on another cluster.
* Be notified of any changes made to your queues, forms, or characteristics.
* Have a historical record of these changes.

QUEUE_COMMANDS, DISPLAY_JOBS, and DISPLAY_QUEUES (this is its first public release) all have built-in help which is displayed with:
$ @command_procedure ?

[Jess Goodman]

Sorry, here is the promised .ZIP file attachment.

[Hoff]

Are these tools and procedures considered Freeware? eg: would BSD- or MIT- or GPL-style licensing and redistribution be permitted, and are there objections to posting these for downloads at the HoffmanLabs site, or the Process Software Freeware servers, or other Freeware servers?

[Jess Goodman]

Yes, they are freeware. The can be copied as long as my name and email are maintained.

[Willem Grooters]

Hakan: Standalone.
Volker: The disk from which we booted was created from an image backup of the original system's systemdisk. As stated, we later found that SYS$QUEUE_MANAGER.QMAN$JOURNAL was missing in that saveset. I do not recall the file existed after boot - I doubt it, or it should have been re-created.
Jess/Hoff: My guess is that this software requires the queue manager to run. It's of no use if it fails; and that's what happened...

One thing that I learned later: to do an ANALYZE/DISK/REPAIR after having restored the system disk from a backup. It seems to be documented ;)

[Jess Goodman]

Willem,

In your initial post you said:
"$ START/QUEUE/MANAGER/NEW did re-create a new environment, but I would have to re-define all forms and re-create all queues."

My post was in response to this. You can use QUEUE_MANAGER_SAVE.COM to save off queues, forms, characteristics, and batch job so that you can repopulate them after START/QUEUE/MANAGER/NEW.

[Ian Miller.]

Jess,
don't forget to post the latest versions to dcl.openvms.org

[Willem Grooters]

Having a record of the whole queue infrastructure, including scheduled jobs, is always advisable.
One remark though: Have this information stored off the node. If your fail-over system wouldn't have these files, you're having the same trouble.