- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - OpenVMS
- >
- Re: how to optimize queue parameters
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-08-2010 10:22 PM
тАО09-08-2010 10:22 PM
We run a maintenance program that submits a lot of jobs to one queue. (each jobs contains a script with CONV and COPY commands for one file) We expected that the maint program would run much faster however it appears that we only shave off 10 percent compared to before.
Queue:
$ INIT/QUE/BATCH/START/JOB_LIMIT=4
/BASE_PRIORITY=4
/WSDEFAULT=1024
/WSQUOTA=8192
/WSEXTENT=40000
/CPUMAX=INFINITE MQUE
My question is, can this queue be optimized? I noticed that these parameters are the same as our old machine but perhaps it can be tweaked as we have a better equipped machine than before. I've tried to research on my own however I'm not familiar with various OpenVMS terminology.
I've included an attachment that provides info on our system (sh work, sh cpu, sh mem)
TIA.
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-08-2010 11:20 PM
тАО09-08-2010 11:20 PM
SolutionFirstly, does this system also do interactive work? In that case, I would lower the queue base priority (to 3 or possibly 2) to achieve better EXPIRIENCED performance.
Second, I do not see any system utilisation, but if "a lot" of jobs means as many as to get jobs waiting in the queue, then JOB_LIMIT might be enlarged.
But, most importantly, WHAT are the (SYSUAF) params for the account that runs the jobs?
If you drive a Ferrari, but the driver only uses 1st gear, speed records will be rare!
Please post the output of
$ MCR AUTHORIZE SHOW
Proost.
Have one on me.
jpe
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 04:48 AM
тАО09-09-2010 04:48 AM
Re: how to optimize queue parameters
That is wide open for interpretation, but let's assume that was for the 4 jobs during the maintenance window.
Well, then 1 cpu can handle all the work, but serializes some, so only a modest improvement can be expected.
So one potential way to speed this up is to set the JOB_LIMIT to 8 or 10, allowing more jobs to run.
Check $ HELP SET QUE /JOB
I suspect though that you will soon run into a CPU
How many jobs are there to run through ? 'a lot' does not mean too much.
TO get a better overview of where time is spend you want to see resource consumption from either SHOW ACCOUNTING, or maybe start with a simple through all batch jobs for a day
$ SEAR/SINCE *.LOG terminated,elapsed
Attach a text in a reply if ned be.
It should show you whihc jobs took long and give a first indication why (cpu or not)
Maybe one long job 'hides' the speedup in the rest?
Now about the jobs.... can those not be tweaked to become faster? With new disks, maybe the IO load can be spread wider (CONVWORK, SORTWORK, 'temp'...)?
You mention files are converted and also copied? Why ?
Can the convert not put the file in the right place? ( logical names for directories/devices? )
If you have many similar jobs, then can you show the core commands for one as example?
During the batch jobs, check out MONI MODE and MONI DISK.
Beter still, try to get T4 going (google: openvms t4)
Good luck!
Hein van den Heuvel
HvdH Performance Consulting
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 05:21 AM
тАО09-09-2010 05:21 AM
Re: how to optimize queue parameters
Somehow I posted>>> I suspect though that you will soon run into a CPU
Meant to write:
"I suspect though that you will soon run into an IO bottleneck instead."
Hein
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 06:20 AM
тАО09-09-2010 06:20 AM
Re: how to optimize queue parameters
Firstly... Yes. I terminate user sessions and jobs running in our main queue (sys$batch) and part of the maint program shuts downs other queues
Second... the maint program throws in about 200 jobs. I've noticed that some jobs complete within a couple of minutes, while one particular job took 3 hours
For example:
Prior to Maintenance
--- CILINE.DAT;1 3427270/3427270 29-APR-2010 03:52:52.53
After to Maintenance
--- CILINE.DAT;1 2950045/3097570 27-AUG-2010 07:37:52.23
... which comes to mind, what if I seclude these jobs that process large files into a separate queue with similar setup as MQUE?
I've attached the sysuaf parameters. I run it under user JOE but I can run also under another user REOTRC but I don't know the difference between the two.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 06:38 AM
тАО09-09-2010 06:38 AM
Re: how to optimize queue parameters
I was looking at the graph on $ MONITOR process/topcpu. I've also attached graphs. The BATCH_xxxx are the jobs that are running the maint.
...How many jobs? - about 200 or more.
I typed in $ show accounting
Accounting is not currently enabled.
... IO load spread wider - I don't know what that means. Create another queue to run the other half of jobs?
... core commands - I don't known what you mean by core commands.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 06:51 AM
тАО09-09-2010 06:51 AM
Re: how to optimize queue parameters
At a minimum:
1) Hardware configuration (CPU/disk/etc)
2) Specifics of file size, location, type.
3) An idea of what these batch jobs do.
4) Are the "other" disks usable as temporary file areas?
Dan
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 07:07 AM
тАО09-09-2010 07:07 AM
Re: how to optimize queue parameters
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 08:08 AM
тАО09-09-2010 08:08 AM
Re: how to optimize queue parameters
>> Accounting is not currently enabled.
So folks are not really interested in performance, they just want it to become better miraculously. (throw HW money at the problem)
Get that going! No excuse not to. Minimal overhead, lots of data.
>> IO load spread wider - I don't know what that means
For example, the provided example shows:
"$ ASSIGN DKC400:[WORK] SYS$SCRATCH"
Is that the same for all?
Are there other disks?
Can each job 'rotate' through a list, perhaps based on target file, or even the lowest digit of the job number or something else?
Start using TEMP1 !
Start using the system disk more!
Start using those DR drives more, define an extra (RAID-0) one for more scratch?
Also check out CONVWORK to a drive different from SORTWORK/SYS$SCRATCH
>> core commands - I don't known what you mean by core commands.
Those that do the real / time-consuming work.
Hint: check out: $ HELP SET PREFIX EXAMPLE
Looks like the core command were:
$CONV/NOSORT/FAST/FDL=FDL:'P2'.FDL/EXCE='P1''P2'.EXC -
/SECOND='YY' 'P1''P2'.DAT 'TDEV''P2'.NEW
$ COPY/CONT/ALLO='ALLOC' 'TDEV''P2'.NEW 'P1''P2'.DAT
That COPY seems costly, avoidable and scary.
Scary because you have half a file for a while. I prefer RENAME to bring a file in production.
Costly because there are 3 copies of the data: OLD, TMP, DAT
Avoidable by having properly maintained FDLs getting the 'right' allocations/attributes.
Note 1... where is the /STAT[=FULL] on those converts. Use that free data!
Note 2... Blindly using /SEC=NOK can cause performance issues due to excessive CONVWORK sizes.
>> .. which comes to mind, what if I seclude these jobs that process large files into a separate queue with similar setup as MQUE?
You want to order the jobs by descending anticipate time, which is likely related to the file size. Get the largest one going first, the little ones will 'fill the holes' and self-balance towards the end.
Regards,
Hein van den Heuvel ( at gmail )
HvdH Performance Consulting
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО09-09-2010 08:21 AM
тАО09-09-2010 08:21 AM
Re: how to optimize queue parameters
The first thing I would try is check for [work] on dkc200, and create ir if not existent.
Next in REORG_ONE.TXT change
$ ASSIGN DKC400:[WORK] SYS$SCRATCH
into
$ ASSIGN DKC200:[WORK] SYS$SCRATCH
(note: please do this ONLY if DKC200 has enough free space! )
Now check how much of the IO to DKC400 gets split over the two devices.
And I already asked: Please publish
MC AUTHORIZE SHOW JOE /FULL
I have the impression that you have the equivalent of an entire football field at your jobs' disposal, but restrict the user to never move more than 1 meter from his start position.
Proost.
Have one on me.
jpe