Reset Entry numbers

[owilliams]

A runaway process caused our entry numbers to jump from low numbers to very high numbers. Is there a way to reset the entry numbers to a lower number? Thanks for any assistance.

[Dale A. Marcy]

Here is a similar thread that might answer your question:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1097922

[Jan van den Ende]

owilliams,

3 possible answers.

1. Just wait, they will roll over and then start a 1 again.

2. If you _REALLY_ want to get rid of the large numbers, make an inventory of all queues (and, if really needed, the jobs).
Stop the queue manager, delete the queue files (well, better rename them away for backup!) and create a new set of que manager files.
Recreate your queus, and resubmit your jobs.

3. Use the queue reset commands. I do not have them at hand, but will look them up if nobody beats me to it. I do remember that there were some 'interesting' details though.

Finally: both with 2. and with 3. : be prepared for a repeating issue.


We have also had it happen, we have done the reset once, and next month it was there again.

And, below the line:
What problem are you trying to solve?
A little nuisance with big numbers in occasional cases?
Better just learn to live with it.

Proost.

Have one on me.

jpe

[owilliams]

Is there a way to manually reset the numbers without playing with the queues?

[Hoff]

The classic HP response is that the job controller entry number is an opaque longword, and no further assumptions about the value should be made.

On more recent releases, there was a change made to the queue manager to try to compress the range of numbers downward where feasible. Off the top, I don't remember the ECO and release details that were involved there.

On older releases, the classic approaches were to force-roll the entries back to the low range (basically wrapping the counters), or to brute-force re-create the entries and the whole of the queue database with a start /queue /manager /new.

Darren Boyle posted the following example of wrapping the entries a number of years back, to seek to force the wrapping of the value back to its low range:

$ loop:
$ set noon
$ submit/hold login.com
$ job_no = $entry
$ delete/entry=job_no1
$ submit/hold login.com
$ job_no1 = $entry
$ dele/entry= job_no
$ goto loop

[owilliams]

Is there any downside to forcing the wrapping of the value back to a lower range of numbers?

Is there any evidence of a high number being a detriment to the system in any way?

[Robert Brooks_1]

Is there any evidence of a high number being a detriment to the system in any way?

--

Only if there is poorly-written software that expects an entry number to be less than a certain value.

The VMS Operating system itself does not have that flaw; your local applications may have that problem.


-- Rob

[owilliams]

I didn't think so but we have some nervous folks at work that thought it might be a problem.

[Hoff]

The appropriate response to >>> I didn't think so but we have some nervous folks at work that thought it might be a problem<<< would seem to involve testing; to run series of big entry numbers into play, and to simply try it. This testing while you're awake, aware, prepared and ready and not otherwise wondering what just happened when a key application tipped over.

[Jiri_5]

You can stop old queue manager and start new with param (/new). After that you must create all queues which you need. on this node. It reset you batch number to number 0

[The Brit]

I know of at least one major application which had a problem with this. Their queue administration module assumed that the entry number would be 4 digits or less. To make things worse, when the entry#'s jumped from 4 to 7 digits, the module displayed the first 4, making it virtually unusable. Although the App Vendor is now peddling a new product to replace the old app, there are probably still many clients using the old app.

[SDIH1]

I ran in to this a year ago, producing entry numbers that in some apps came up as negative numbers :-(.
On the freeware cd's there was a utility called fixque ( I think by Keith Parris) that dumped the queue database, and parsed out the output of show queue/full etc. Problem was that it couldn't handle the large entrynumbers either :-). So I wrote a dumpqdb.com based on getqui
that produced an output file with all the commands necessary to create a new queue database.

Couple of things missing : characteristics, we don't use it. ACL's on queue, ditto. /new switch to the start/que/manager command. Probably more, there are a lot of things that we don't use, but are possible.


To use it, create the output file by @dumqdb/out=w.w, then carefully compare w.w to your queues and entries, edit in omissions or additions, add the /new switch to the start/que/manager command.
Then stop the queumanager, backup, and then rename or delete all queue database files, as they probably have grown beyond any reasonable size and run w.w. You should (almost, right in three tries is my sad motto) be back to normal.

This was all written as an emergency procedure, and it did work 99% in our situation, but this might be different in yours.

[Ian Miller.]

Jose,
can you post that to dcl.openvms.org as well as it looks like it could be useful for somebody.

[Dean McGorrill]

I suppose one could locate the longword cell
where the entry number is kept and stick a
low value in it. assuming all the jobs are
high numbers they wouldn't collide. (I've
never tried it though)

[David Jones_21]

I like having 3-digt entry numbers just for asthetic reasons. Since we already have command procedures to re-create all queues from scratch, it isn't that big a deal to reset the queue database.

Samba seems to assume job numbers below 10,000, but I think Unix job numbers are per-queue rather than global as in VMS.

[John Gillings]

A brute force approach to resetting entry numbers. No special privileges required.

The attached procedure submits itself /HOLD, then immediately deletes the entry. It then compares the entry number against a parameter, and if greater, repeats.

If you're in a high range, execute the procedure with a parameter of (say) 10000. It will repeat until the entry number wraps. Provided the number you specify is more than the number of concurrent entry numbers you have, the loop will eventually complete. With a high enough threshold, you could safely run the procedure regularly to make sure your entry numbers don't exceed your limit.

Once the number has wrapped, just wait for the high numbered entries to complete (or resubmit them), and the queue manager should reset itself into the lower range.



$ IF p1.EQS."" THEN EXIT
$ self=F$PARSE(";",F$ENVIRONMENT("PROCEDURE"))
$ loop:
$ SUBMIT/HOLD 'self'
$ DELETEX/ENTRY='$ENTRY'
$ IF F$INTEGER($ENTRY).GT.F$INTEGER(p1) THEN GOTO loop
$ EXIT

[Jon Pinkley]

Here are my experiences with FIXQUE and dumpqdb.

We have a relatively complex queue database, (1088 total queues. 667 batch queues, of which 153 are generic. Queues with text descriptions, acls, characteristics, many forms and multiple stock types)

The point is, just because something doesn't work for us, does not mean it will not work for you.

----------------------------------

The version of FIXQUE I tested is a modified version that was posted to comp.os.vms to allow it to work better with larger queue entry numbers.

FIXQUE evidently originated in the Colorado Support Center, I've seen versions with Keith Parris and Mark Jilson's comments, so I don't know how it evolved, but it is quite old and probably has the work of many people in it.

FIXQUE works by creating a listing using show queue commands, and parsing the output. This reason alone is enough to make it impossible to recreate an arbitrary set of queue entries. The thing that caused it to break at our site is an entry that has long quoted parameter values, and these get truncated in the output from show queue/full. Since FIXQUE gets its information from the listing file, it cannot reproduce the original entry, although it doesn't detect that the value has been truncated.

FIXQUE does do an admirable job given the limitations of the implementation, i.e. recreating the "state" from only a listing, and doing it using only software on a standard VMS system, i.e. DCL and the standard utilities.

FIXQUE creates a command file that will recreate forms, characteristics, queues (both generic and execution), acls on queues, and the entries that were in the queues at the time it was run. For jobs that are currently executing, it generates a SUBMIT /HOLD, since these jobs should be manually verified before allowing them to start over.

It also gives warnings about jobs that reference files that do not exist in any directory.

------------------------------------

dumpqdb (see sdi-1 (Jose Baars, mostly) entry in this thread dated Jul 24, 2007 18:18:03 GMT) command file is attached to entry.

dumpqdb (stated in the note) was written for a specific situation, and as such does not attempt to be as general purpose as FIXQUE.

However, it was written later than FIXQUE, and uses f$getqui to get the information to recreate the queues, which is much less likely to break in the future that attempting to parse the output of a listing.

I was happy to see that someone had used the more direct approach to gathering the queue information with f$getqui instead of using the more limited output of show queue.

I am glad it was provided, it definitely has potential, and is a good example of what can be done with f$getqui. It is much smaller than fixque.

The first time I ran the procedure, I got

$ @ROOT$USERS:[JON.SCRATCH]DUMPQDB.COM;1/out=scr:w.w
%DCL-W-IVVERB, unrecognized command verb - check validity and spelling
\SAY\
%DCL-W-IVVERB, unrecognized command verb - check validity and spelling
\SAY\
%NONAME-F-NOMSG, Message number 00000004
$

I entered

$ say :== write sys$output

and then it got further, but died when it got to a print queue that was defined like this:

$ show queue /full b00acc1
Printer queue B00ACC1, idle, on SIGMA::"acc-4000tn-1:9100", mounted form DEFAULT

/AUTOSTART_ON=(SIGMA::"acc-4000tn-1:9100",OMEGA::"acc-4000tn-1:9100") /BASE_PRIORITY=4 /DEFAULT=(FORM=DEFAULT) /LIBRARY=LEXLZ
Lowercase /OWNER=[SYSTEM] /PROCESSOR=UCX$TELNETSYM /PROTECTION=(S:M,O:D,G:R,W:S) /SCHEDULE=(NOSIZE)
(IDENTIFIER=B00MEM,ACCESS=MANAGE)
$


$ @ROOT$USERS:[JON.SCRATCH]DUMPQDB.COM;1/out=scr:w.w
looking at queue B00ACC1
%DCL-W-IVCHAR, invalid numeric value - check for invalid digits
\4000TN\
%NONAME-F-NOMSG, Message number 00000004
$

The cause of the confusion is the quoted values, and the way DCL parses quoted strings.

Specifically, the line

$ autonode = f$getqui("display_queue","autostart_on",qname,"freeze_context")

works, but what is returned in this case has embedded quoted strings.

$ show symbol autonode
AUTONODE = "SIGMA::"acc-4000tn-1:9100",OMEGA::"acc-4000tn-1:9100""

Then autonode is used in the following:

$ write sys$error "queue ''qname' has autostart set to ''autostart', autonode = ''autonode', host=''host', node = ''node'"

And when DCL substitution occurs, the following happens:

$ write sys$error "queue B00ACC1 has autostart set to TRUE, autonode = SIGMA::"acc-4000tn-1:9100",OMEGA::"acc-4000tn-1:9100", host="
acc-4000tn-1:9100", node = SIGMA"
%DCL-W-IVCHAR, invalid numeric value - check for invalid digits
\4000TN\
%NONAME-F-NOMSG, Message number 00000004
$

I wish f$edit had a built in edit rule to process a string so it could be used in a quoted DCL string, e.g. processing the single quotes and double quotes so they will work identically when enclosed in a double quoted string. I don't know of an easy way to do this in DCL.

For example: (non-implemented)

autonode_for_quoting = f$edit(autonode,"QUOTE_FIXUP") ! this does not exist

would result in

$ show symbol autonode_for_quote
AUTONODE_FOR_QUOTE = "SIGMA::""acc-4000tn-1:9100"",OMEGA::""acc-4000tn-1:9100"""

This is where I stopped the testing of dumpqdb, as to get it to work with all the features we are using would be harder than to fix fixque to use f$getqui for the entry parameters.

------------------------------------

The following is probably obvious, but needs saying:

Running any of the tools to recreate the queue database on an active system is somewhat equivalent to backup/ignore=interlock, i.e. the state of the queue can be changing underneath the software that is attempting to reproduce it.

If there are jobs that are going to synchronize on or release other entries using the entry numbers, any software that recreates the database is going to break those.

My point is that if there are a lot of queue entries in your system at the time you recreate it, you are much more likely to have problems than if there are none, and you are just recreating the queues, forms, characteristics etc.

[Art Wiens]

There must have been some sort of lunar cycle ... we also had a runaway train this week! Instead of submitting itself once the next day and carry on with it's task, a coding error looped it back to the submit. It managed to do this over 148,000 times before someone remarked "the system's a bit slow isn't it?". Just to note how the entry numbers seemed to go - it looks like it counts up to 9999, increments the million's field and goes again ie. 1000000 - 1009999. I'm not sure if it finally gave out or when I killed the queue_manager and job_cocntrol pids, but it made it up to/over 14,000,000. Yes it was drastic to kill but the queue manager was solid compute and had crashed on it's own 21 times. No queue related commands were completing, no jobs were executing. I had to modify SYS$COMMON:[SYS$STARTUP]VMS$CONFIG-050_JOBCTL.COM to include a larger value for /page_file=3000000 because it was dying with :

%QMAN-F-ALLOCMEM, error allocating virtual memory
-LIB-F-INSVIRMEM, insufficient virtual memory
%JBC-E-QMANDEL, unexpected queue manager process termination

Once the queue manager was running again I saw what had happened to the entry numbers - what a mess! I was able to produce a listing of that queue and it's 148,000+ entries (all waiting to "go" at 22:10!!), learned a keyboard macro to delete the entry, EVE/TPU was able to replay it over 148,000 times (!) and I executed it. Took about half an hour to delete them all. Phew!

Not a very good friday on sysadmin day!!

http://www.sysadminday.com/

Cheers,
Art

ps. the "coding error" was NOT mine, I just had to clean it up ;-)

[SDIH1]

Hi,

I tried to overcome the fixup quote thing, attached an maybe improved procedure.
It handles the specific example of the autostart_on nodes, as far as I quickly tested.

In our case the qmanager journal file had eaten up all disk space, so a backup of the queuemanager files was not desirable or even possible. In our case we had to use a tool like this.

[owilliams]

Thanks all, you have been extremely helpful.

[Jess Goodman]

I've attached my DISPLAY_JOBS.COM which is an extremely robust procedure that uses F$GETQUI to recreate SUBMIT commands for batch jobs. DISPLAY_JOBS was originaly based on a very old DCL example from a DEC web site, but I completely rewrote it.

It handles quoted strings in /Parameters, the job /NAME, and /NOTE (see the GOSUB routine DOUBLE_QUOTE). It will not write lines longer than 255 characters on versions of VMS before Extended DCL.

By default:
* It outputs SUBMIT commands to the terminal.
This can be redirected with DCL's @ /OUTPUT feature.

* It processes jobs on all batch queues. Or P1 can be a wildcarded queue name. Or if P1 is "" then P2 can either be an entry number or a wildcarded job name for the current user.

* It skips jobs that are currently executing or retained. This can be changed by setting P3 to "*" or to any list of F$GETQUI job flag keywords.

* It does not use /USER=me for the current user's jobs, on the assumption that the same user will execute the SUBMIT commands. Use P4 = "/USER" to force /USER=me anyway.

* P4-P8, if present, are added to the end of the SUBMIT command.

It handles all documented SUBMIT qualifiers with the exception of /RAD which I could not test since I don't have an Alpha GS system. (Apparently not only does a job have to execute on a GS system to use /RAD, but any such job must be submitted from a GS system.)

I was thinking of adding an option to delete each job and then resubmit it, and an option to use the most current version of the batch command file for the submit instead of the version previosly submitted, but I haven't settled on a user interface to select those options.

When I have time I want to write a DISPLAY_QUEUES.COM that is just as robust. But until then you can modify FIXQUE or DUMPQDB to not process jobs and use one of them along with this.

[Jess Goodman]

I forgot to mention that if P1 is not null (so it's either * or any other [wildcarded] queue name) then P2 can be keyword "GENERIC" or "NOGENERIC".

[Jan van den Ende]

owilliams,

from your profile
>>>
I have assigned points to 53 of 99 responses to my questions.
<<<

Maybe some assigning is in orfer?

PS. Zero [oints for this. please

Proost.

Have one on me.

jpe

[Jon Pinkley]

Jess Goodman,

Please add a note to my thread about migrating queue entries from one device to another with a pointer to your notes about DISPLAY_QUEUES.COM, and I will give you 10 points. Looks like a very nice utility to deal with batch jobs. In fact, I will probably use a combination of it and FIXQUE.COM to deal with our situation.

I can't give you points here, but I can in threads I start.

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1149896

[Jess Goodman]

Ok, I have attached a newer version of DISPLAY_JOBS.COM to that thread.

It is now better documented and will output help with the command:
$ @DISPLAY_JOBS ?

I fixed a couple of minor typos and a bug with batch jobs for multiple files.

I guess I might as well attach the new version to this post also...