how to optimize queue parameters

[J Ancheta]

Jan,

Thanks again for your reply. I posted on my first reply on this thread - unless your asking me to run a different command.

If I understand correctly, your suggesting to split some of the workload for example: 100 jobs process in DKC400 and the other 100 jobs process in DKC200, right?

[J Ancheta]

Hein,

Thanks for your reply again. A lot for me to digest and absorb.

Off the bat you mentioned increase UAF record which will make more memory available. Can you please give me a hint on how to do that?

[Jan van den Ende]

J.

>>>
Thanks again for your reply. I posted on my first reply on this thread - unless your asking me to run a different command.
<<<

Yeah, my bad. I overlooked your firsrt attachment.

And somehow Hein's and my reply intertwined.

As is to be expected, his answer is much more detailed, especially on RMS related stuff.

I ment more or less the same, but his wording in much better and better detailed.

DO try bto digest it, and ask detailed questions on anything unclear!

hth

Proost.

Have one on me.

jpe

[John Gillings]

J,
Shave off 10% is a pretty good result!

>can this queue be optimized?

The word "optimized" should never appear in any question without being immediately followed bny "for ". There is no such thing as generic optimization, you have to optimize for something, for example, reduced elapsed time, reduced CPU time, fastest I/O, minimum memory, etc... By definition, improving one metric will cost some other metric. All you can do is move the bottle neck around.

I'll assume the metric you're interested in is ELAPSED TIME.

Queues don't really have much impact on performance, so they're not really a good place to perform tuning, with the SOLE exception of the JOB_LIMIT. This will control how many jobs can run in parallel.

CONVERT and COPY tend to be I/O bound, so it's possible you can run several in parallel (which you can control with JOB_LIMIT), however, if you have multiple I/O streams from or to the same disks, they may get in each other's way and REDUCE performance over single stream. You can also end up fragmenting files unnecessarily.

Without knowing the hardware configuration, or the sources and destinations of your files, it's not possible to give specific advice, but if you think about it as if it's a road network, you may be able to determine a good sequence of jobs. For example, suppose you have 4 disks, A, B, C, D and you have 4 jobs, two of which are A->B and two C->D. If you run them two at a time you could run the two A->B in parallel, then the C->Ds in parallel. This would maximise contention and probably perform poorly. If you go the other way and run A->B against C->D they don't get in each other's way, so perform better.

(This is simplistic, as it doesn't consider controllers, busses and other I/O infrastructure, but hopefully you get the idea)

The key to performance for this type of job is the exact sequence of operations, to prevent contention. You can't really do that if you just generate a bunch of jobs and dumping on a single queue.

What you may want to consider is to analyze your environment to determine which operations can be done in parallel. Create a separate queue for each "contention group" and give it /JOB_LIMIT=1.

Now divide up your jobs and submit each to the appropriate queue. Now you can run as many jobs as possible in parallel (one in each queue), while minimising contention, and thereby minimising total elapsed time.

[J Ancheta]

Hein, Jan, and John,

Thank you all for posting and guidance. I now understand of how our maintenance program works and see the loop holes you've indicated and have been actively researching various terminology.

I created new scripts using the old scripts as a basis and applied some of your suggestions and letting her rip as of the moment. I know this is quite a bold attempt but nevertheless I was extremely careful in not modifying various core commands.

I appologize as I realize that I mislabelled this post but I had no clue what was going on, what to look for, and where to start (newbie).

I will return to post how much time was saved.

Cheers.

[Hein van den Heuvel]

Sounds like you are doing great!
We are proud of you!

:-)

Seriously, it sounds like you are picking up terminology and understanding quite fast.

Your next step, if needed, is to review the FDL files used.
Specifically you may want to do monthly/yearly ANALYZE/RMS/FDL and EDIT/FDL/NOINTERACTIVE cycle for those files which are not 'hand tuned' already.

Hint: do the ANAL/RMS outside the maintenance window, possibly after all converts are done, on the OLD files.

Cheers,
Hein

[J Ancheta]

The new maintenance procedure that I ran took approx 2 1/2 hours which used to be 5 hours!

[abrsvc]

Cutting the elapsed time in half is a wonderful start. With the new procedures, take a look at paging and disk activity. As John said earlier, making sure that the disks are not fighting each other is important. Check the IO stats for activity as well.

Careful use of the DCL SYNC can help to coordinate jobs as well.

One side note: I'm not sure if contiguous files are as important these days as in the past. You may want to test whether or not that requirement is necesasry. Depending upon how the individual "bits" of the file are accessed, neing contiguous may not be buying you much in performanc.

Dan