Linear time expansion for multiple sequential reads

[Mike_Hellier]

I'm something of a beginner so please bear with me...

I'm running OpenVMS 7.2 and we've been having some ill-defined performance issues so we wrote a test program to try to emulate some of the things that the application does.

The test program opens a file, shared and read only and sequentially reads 10,000 records (fortran program where I don't believe anything special has been done with FAB/RAB settings)

Running one instance gives me a 15sec time, running two gives me 30 sec, three 45 sec etc - so my guess is that we're running into some kind of contention/waiting scenario.

Reading an awful lot of documentation suggests that I can open the file with various settings (no locking, read regardless and no query record locking) - and I understand that in 7.2 upwards these latter options actually ignore the lock manager completely.

I've tuned the FDL to death (with my limited skillsets) using the /optimize technique... enabled global buffers etc.

My question really is to what extent these read processes really do speed up parallel reads to the file - or what other options I need to explore.

In this case all we need to do is read a bunch of records into an internal array where we'll process them.

If you can give me some things to look for or suggestions it would be much appreciated

[Robert Gezelter]

Mike,

With the limited information at hand, your problem does sound like contention. You did not mention:

- what physical devices you are using
- whether the device is actively doing other things
- what buffer settings are the default on your system (SHOW RMS will show the system and process defaults).
- and to what extent you have cacheing enabled on your system.
- and I have probably missed some things in the above list, offhand.

One experiment that often in such situations is to do use an approach described as "Exagerate System Parameters (ESP)" (my recollection is that the first time I saw this term was the aerospace author Stine, but I digress). That said, you can start run an experiment by increasing the working set (ensuring the the page file quotas are consistent), and then increasing the default RMS parameters, say to /BLOCKS=20/BUFFERS=20 and see what performance you get.

In contention situations, dramatic increases in buffer size and number can produce significant performance improvements (your mileage will definitely vary, it depends on many factors including disk performance, CPU performance, and contention between applications). That said, I have an application similar to yours that I can vary the performance up and down by an order of magnitude in this way.

- Bob Gezelter, http://www.rlgsc.com

[David Jones_21]

Are you sure the limiting resource isn't the CPU (you don't say what system you are using). Formatted I/O in FORTRAN can be awfully expensive processor-wise, especially if the statements have loops in them.

[Antoniov.]

Mike,
your question appear to too generic.

However, I guess you could read more record as possible in your application so you should optimize to sequential read and conseguently you should set an high value of bucket size.
In this scenario your application should lock many records (I don't know fortran and I think you can set manually record lock) but your software stops parallel read from other processes.

Antonio
htpp:/it.openvms.org

[Hein van den Heuvel]

Mike,

Welcome to the OpenVMS ITRC Forum

I suspect you are running into bucket lock contention, not an IO problem.

Your best tool is probably simply MONITOR.
Notably the MODE and LOCK classes.
But of course also MONITOR RMS/ITE=(LOCK,CACH,OPER)
That would be during the run after:
$SET FILE/STAT testfile
Be sure to also check out my RMS_STATS tool on the VMS Freeware V6 (5?)
or... $ANAL/SYS ... SET PROC ... SHOW PROC/RMS=FSB

The solution for the test is probably to OPEN READ ALLOWING READERS (or whatever the FORTRAN syntax is for that).

For the real application this may not be feasible.
For the real application you may find that using global buffers solves the problem, because a side effect of global buffers is concurrent read lock caching for bucket locks.

>>> no query record locking) - and I understand that in 7.2 upwards these latter options actually ignore the lock manager completely.

No true. They avoid the locak manager for RECORD locks. But every time you tell RMS to find a record in a bucket in a shared file it will take out a bucket lock to make sure the buffer is still valid.

Are these short records (single floats or ints?). In that case you will have many records per bucket/buffer and if all programs start roughly at the same time and place in the file they will get severe contention.

If you continue to experience performance problems, consider to contact me offline. (see profile).

Regards,
Hein van den Heuvel.

[Robert Gezelter]

Mike,

Re-reading your post, I see a potential ambiguity.

Is the file a sequential file, or is the file an indexed file, from which you are doing 10,000 sequential reads (at least sequential in some key).

There is a significant difference between the two versions of the question. On first glance, I answered the first question. Hein and others have answered the second possibility.

Please clarify which of the possibilities is the one you intend?

- Bob Gezelter, http://www.rlgsc.com

[Hein van den Heuvel]

I has assumed a fixed length sequential file, but missed the lines:

>>> I've tuned the FDL to death (with my limited skillsets) using the /optimize technique... enabled global buffers etc.

This suggests an indexed file.

And Mike mentions global buffers already, which I suggest might be a solution, but apparently were not.

Two possiblities:
1) Maybe Mike ran into the 'special case' where RMS does NOT use global buffers for readonly access allowing no writers:

See:
http://h71000.www7.hp.com/doc/731FINAL/4523/4523pro_006.html#1178_fabw_gbcfield

"To specify a read-only global buffer cache, the initial accessor must set the FAB$B_SHR field FAB$V_SHRGET and FAB$V_MSE bits on. Selecting the FAB$V_MSE option turns on locking to coordinate access to the global buffer cache. "

So for global buffers you want ACCES READ ALLOW UPDATE.

2) The wrong VMS version.
The NQL option, as well as CR locks on global buggers (:-) were introduced in 7.3 and backported to V721-1H1, V7.2-2 but plain 7.2 (as indicated) does not have this.

Anyway...

MONI RMS / ITEM=CACH will show what really is happening as well as

ANAL/SYS ... SHOW PROC xxx/RMS=GBH


[Bob,... you are up early :-) ]

Greetings from Amsterdam,
Hein.

[Robert Gezelter]

Hein,

Yes, I did get up early today. In light of your posting, I re-read Mike's original posting. I came away with a degree of ambiquity (if the FDL was posted, it would be unambiguous).

Also, the question occured to me as to what key the file is being read sequentially with. If it is an alternate key, that could be painful (or if the file has a lot of internal fragmentation).

- Bob Gezelter, http://www.rlgsc.com

[Robert Gezelter]

Mike,

Just a thought.

What does the result of MONITOR PROCESS/TOPCPU show? Are all three processes running at about the same level of resource consumption (some variance is expected and normal).

- Bob Gezelter, http://www.rlgsc.com

[David Jones_21]

Since they've written their own test program, it would be easier to instrument it with LIB$INIT_TIMER and LIB$SHOW_TIMER calls to take some of the guesswork out.

[Mike_Hellier]

Thanks for all your comments and suggestions - I thought that I'd update you on progress.

My issue seems to be that I do not have a sufficiently qualified sysadmin for the box. The application programmers are pretty good but we're lacking that low-level of o/s knowledge.

I'll start by defragmenting the disks (they are quite fragmented) and implementing new FDLs and I'm going to have to suggest that we acquire a sysadmin from somewhere - even part-time in a consultancy role.

I might also raise a thread re RAXCO's performance suite as I need to automate much of the system management asap and getting the budget for a sysadmin is unlikely.

Thanks again for your help, I have been reading up on your suggestions so they have been helpful.

[Hein van den Heuvel]

>> Thanks for all your comments and suggestions - I thought that I'd update you on progress.

Appreciated. If you ever get to actually explain the problem described, then please share that with us here.

>> My issue seems to be that I do not have a sufficiently qualified sysadmin for the box. The application programmers are pretty good but we're lacking that low-level of o/s knowledge.

Sys-admin (configure, backup, schedule) is one thing.
Coding up an applicaton an other thing.
That are inbetween the two is more tricky and harder to fill. Judging by your comments you are well on your way to fill the gap... if you care to go there. A little more training (OpenVMS bootcamp? Consultancy) and you'll be there.

>> I'll start by defragmenting the disks (they are quite fragmented)

Don't bother. That's a 2% improvement problem.
You may well have 'low hanging fruit' offering 20+% performance mprovement out there.
Of course it is nice to 'sweep the floor' and remove a point of concern, but don't expect it to mage a measurable difference while the big bad stuff is still active...

>> and implementing new FDLs

Yes.

>> and I'm going to have to suggest that we acquire a sysadmin from somewhere - even part-time in a consultancy role.

Don't label it sysadmin, it's more than that.
I may be able to provide some names if you like.

>> I might also raise a thread re RAXCO's performance suite as I need to automate much of the system management asap and getting the budget for a sysadmin is unlikely.

Sounds good.

>> Thanks again for your help, I have been reading up on your suggestions so they have been helpful.

Be sure to re-read the replies in a week as new understanding grows.

Good luck,

Greetings from Nashua, NH.
Hein.

[Robert Gezelter]

Mike,

I second Hein's comments. Both he and I have worked with many client situations, with a wide range of complexity.

The sysadmin title is a *NIXism that is often misleading. This is more into the realm of traditional systems programming, than administration. IMHO, automated tools, while nice to have, are not likely to address or correct this type of problem.

On an ongoing basis, a relationship with a well-qualified systems specialist is quite economical. In my practice, I find that the overall bills to any individual client are often far less than expected. In many sites, that level of expertise is required a handful of days per month, or less, on average (I am not trying to account for random emergencies). The ongoing monitoring of the site also ideally allows emerging problems to be identified before they become crises, which is far less disruptive.

I hope that the above missive is helpful!

- Bob Gezelter, http://www.rlgsc.com

[David Jones_21]

To be frank, I'm totally lost as to what is going on with this guy. His rant doesn't convince me he's identified the problem, but the thread has been closed. Oh well.

[Mike_Hellier]

Thanks David for your helpful and instructive comments.

To be honest I thought that I asked a sensible question. It was obvious that I don't have the level of expertise to accurately pose the question correctly - I can "feel" that we have an issue but lack the skills to correctly interpret and present the data.

I take Hein's comments onboard and we'll try to find someone through our channels who can work with the application programmers on the rms/vms interaction aspects.

We still need a sysadmin though as the normal operator functions are ill-defined and need ownership within the company - which I'll get sorted out.

In the meantime we'll continue our learning process ... !