how to improve batch job buffer io performance

[Desmond Or]

I have a server ALPHA DS25 with 8GB memory. This week we start a new job but it takes 3 hours to finish. I find that the jobs Buffer IO count is over 1700000 and Direct IO count is over 300000. I modified the user account parameters BIO to 32767 and DIO to 32767 but the job running time is still 3 hours. Does anyone who knows what parameters would influence the io count and improve the job's running time?

[abrsvc · Accepted solution]

There is some confusion here. The IO counts in the log file indicate the number of IOs compelted. Without rpogram changes, this count is not likely to change. The biolm and diolm parameters indicate the number of outstanding IOs at one time, not the limit of IOs or their size.

If the job is IO bound, then an examination of the application needs to occur in order to perhaps find ways to make the IOs more efficient or less frequent.

Can you provide any information about the application here?

Thanks,
Dan

[Desmond Or]

It is because I am not the developer of the application. I need to talk to the programmer to find out the application details. My job is to tune the system to get a good performance for the job run.

Thanks.

[Jan van den Ende]

Desmond,

What is that job doing? Does it REALLY need that many IOs or is it somehow repeating the same IO many times?

The best IO always is the one that is avoided...

_IF_ the job is doing RMS IO, _THEN_ it may well pay to have a good look at the various RMS_* parameters. There are many, but YOU know the job, so only YOU can decide which are relevant to the job.
... and they can be SET at the beginning of the job, making the adaptation apply only to the job, or on a system-wide, more permanent basis.

Happy hunting.

Proost.

Have one on me.

jpe

[Jan van den Ende]

Desmond,

the previous answers crossed each other.

>>>
My job is to tune the system to get a good performance for the job run.
<<<

Remember, almost always the best tuning rule is: "Redesign your application".

Proost.

Have one on me.

jpe

[Hoff]

The application clearly implements a substantial chunk of I/O, but whether that's reasonable (or not) depends on the size of the data and what the code is doing. You've previously asked questions which point to substantial data; to the existence of data files containing 1.3 million records.

If this is a traditional application, point the programmer at the DECset PCA product, the Performance and Coverage Analyzer tool, and figure out where the application is spending its wall-clock time.

If this is Java, all bets are off. Have a look at reducing the number of stat calls and logical name translations, and other such magic. (Both C and Java can benefit from caching this stuff via the DECC* feature logical names, particularly if the code is tossing gazillions of these calls.)

There can also be triggers elsewhere, such as extreme fragmentation within the file system, or within RMS files.

This could easily be a case where your I/O queues are stuffed solid for three hours, and if that's the case you need less I/O, I/O that's distributed across more spindles, or faster hardware. That you need faster hardware or an application redesign or a system redesign; that you've hit the proverbial wall with this box and with whatever disks are on this box.

But that's all determined via instrumenting the code, directly or via PCA, or via other application and system profiling tools. Figuring out what the wallclock is, and potentially figuring out what's generating that I/O rate.

And probably also point the programmer at the OpenVMS Programming Concepts manual in the documentation set, if they've not taken the time to read through that document.

[Andy Bustamante]

>>>My job is to tune the system

You really need to know what the application is doing to the system. We can't do that without many questions and even then might miss somehting that an experienced system manager would spot while logged on.

If this job is IO bound, look at the source and target devices. Are you reading and writing to the same disk? Does the disk have an IO backlog ($ monitor disk/item=que). Does the job have quota restrictions?

Use the Availability Manager, http://h71000.www7.hp.com/openvms/products/availman/index.html?jumpid=/products/openvms/availabilitymanager, and T4 http://h71000.www7.hp.com/openvms/products/t4/ to monitor and record the system during the job run.

Availability Manager won't preserve a record so you'll need to monitor the display and messages, but it can pick out some performance issues. Note that some metrics are too small out of the box and you can reset the trigger point for warnings. Which ones to reset? It depends on your system hardware and application.

[Steven Schweda]

> Does anyone who knows what parameters would
> influence the io count and improve the
> job's running time?

Does any one here (other than a psychic) know
what "batch job" is doing? Do you?

I've seen some programs benefit from
adjusting RMS default parameters:

HELP SHOW RMS_DEFAULT
HELP SET RMS_DEFAULT

Knowing practically nothing about "batch
job", I have no idea if that might help it.


> I need to talk to the programmer to find
> out the application details.

Details?!?! Even a few _generalities_ would
be better than what we have now, which is,
essentially, nothing.

> My job is to tune the system to get a good
> performance for the job run.

That might be a little easier if you knew
_anything_ about what "the job" was doing.

[John Gillings]

Desmond,

Unless your system is very, very badly configured, you're unlikely get even a 10% improvement by "tuning". Best case you might get it down to 2 hours 50 minutes, which I'd guess wouldn't be worth the effort. As others have suggested, you'll need to look at the application if you want to make significant improvements.

Roughly speaking, DIOs are disk I/Os and BIOs are terminal, network or mailbox I/Os. BIOLM and DIOLM refer to the number of concurrent I/Os, so unless your program is specifically written to do aynchronous I/Os, increasing these limits from the defaults won't affect anything.

Batch jobs typically don't talk to terminals, so your BIOs are likely to be network or mailbox. If it's network I/O you're probably limited by your network. Check for obvious things like duplex mismatches or network congestion.

If the job doesn't use the network, the next thing I'd check for is use of the PIPE command. It's very handy and useful to build procedures as sequences of pipes, and you'd expect that pushing data through memory, rather than temporary disk files would improve performance, but sadly this is not the case. The I/O performance of PIPE is atrocious (something to do with using RWMBX state for synchronization). The symptom I'd expect to see is slow performance with an excessive number of BIOs.