Fortran Unformatted Real

[Joseph Huber_1]

Sorry I overlooked
>>I am not using RECORDTYPE, and I'm not even sure what that is.

But how then can the result be in the 'shapefile' format as in the paper cited ?

[Robert Gezelter]

JD,

Ok, the specification that was cited in this thread is 34 pages long, and certainly does mix Big-endian and Little-endian numbers (for unspecified reasons). Admittedly, that is unusual. Most specifications pick one or the other and stick to that choice. However, since it is a specification for an outside product, it is what it is.

A casual reading of the specification would seem to indicate that its use of the term "record" has little to do with the meaning of the term "record" as used in FORTRAN. At first glance, it appears to be a binary byte stream.

Writing to this specification is not the most complex programming problem, but it is certainly not a beginner project. One post mentioned that your experience in both C and FORTRAN is limited. What languages are you familiar with? What is your general programming experience?

Since generating this data is inherently a serial process, increasing the number of processor cores is not likely to appreciably increase the speed of the code.

That said, if the format is what it appears to be, several orders of magnitude of performance (100 to 100,000 times) beyond 4K items per second would seem to be achievable.

Your organization may want to consider finding a supplemental resource with more experience in C/FORTRAN, binary files, and related areas [Disclosure: our firm provides services in this area, as do other active ITRC contributors]. ITRC participants contribute their time and insight for no compensation.

- Bob Gezelter, http://www.rlgsc.com

[JDoe_1]

My programming experience would be likened to a "Jack of all trades, master of none" sort of situation. Fortran is just the preferred method for the lab I'm working in currently. I've written in Java, PHP, Visual Basic, but mostly MatLab's specialized code.

Also, I should have just kept my mouth shut, but the part I'll be making parallel doesn't have to do with the file generation.

In any event, I will give the initialize and extend options a go and see what happens.

Thanks.

[Robert Gezelter]

JDoe,

Be careful about the use of the term record. As I noted, the term "record" in the shapefile description appears to be emphatically NOT the same as the meaning of the word "record" in the RMS and FORTRAN context.

- Bob Gezelter, http://www.rlgsc.com

[Steven Schweda]

> [...] I am not using RECORDTYPE, [...]

What _are_ you using?

> I assume OPEN specifies [...]

If a responder needs to assume anything, then
it's likely that the questioner is not doing
his job properly.

> Also, I should note that the program does
> work currently, [...]

Which suggests that something is being done
right, even if sub-optimally.

> [...] I will give the initialize and extend
> options a go [...]

Plausible, but it might be wise to see where
the time is being spent now, rather than poke
at the problem in a hopeful but disorganized
way. There are actual documents and software
which can help with performance analysis on
VMS systems. You may have some truly
horrible code, which can be greatly improved,
but optimizing horrible code which takes only
1% of the time won't buy you much.

[Joseph Huber_1]

JD,
>>
I am taking the 8 byte real and writing it to a file on the hard drive, and reading back the binary as two 4 byte integers and just continuing on my merry way.
>>

Do I understand it right: just for the purpose of converting a real number in memory, You write it to a (temporary) file, and read it back ?
If yes, no wonder it is slowing down a lot.
For floting point in-memory conversions, VMS run-time library offers conversion routines.
Read
HELP RTl CVT$
to see if it offers the correct thing.

[Robert Gezelter]

JD,

Joe has a valid point. Creating a file on disk for each number is a very expensive number. FORTRAN does have in-memory IO, but the best solution is to use the conversion library (mentioned in Joe's recent post).

- Bob Gezelter, http://www.rlgsc.com

[Hoff]

What sort of VAX, Alpha or Integrity Itanium box are we talking about here?

What are the OpenVMS and Fortran versions?

What sorts of disk(s) and I/O hardware here?

On no source code and no evidence, I'd tend to guess this program was I/O bound.

Better yet, run DECset PCA to see where you're really spending your run-time and your wall-clock time here.

Why not build and buffer and write a hundred or a thousand or a million (of what are here confusingly called) records at a time, at memory speeds? Writing small wads of data into big buffers stored in virtual memory and then knocking out whole buffers en-mass is going to be massively more efficient and expedient than legions of smaller I/Os. Double-buffer and overlap your writes for better speed.

If working within the constraints of OpenVMS and of RMS here, I'd (still) be tempted to build a big array of entries and write it to disk with one big I/O. Record writes are slow, and (small) file extensions are to be avoided.

If you know how much data you're starting with, then you could conceivably map and write to and use a process section with a backing storage file and largely remove RMS (and the RMS concept of records) from the design. That's going to be about as fast as you can reasonably go here within the constraints of your hardware, after all...

Stephen Hoffman
HoffmanLabs LLC

[Steven Schweda]

> Do I understand it right: [...]

And did you notice that we resolved this
problem back around "May 27, 2009 05:08:10
GMT"?

[Hein van den Heuvel]

Steven, Steven... don't confuse us with the facts. We are just about to switch over to trolling for work. :-).



Speaking of facts

" Currently I'm doing about 4500 records/second on a 2 ghz machine on 1 core."

Jdoe,

What Hardware Platform and Operating System, is this exercise running on?

There are no 2 Ghz OpenVMS systems, other than emulated Alpha's on 2 Ghz X86's

Even with the relatively vague specification in place, I think it is safe to say the 4500 records/second is LOUSY performance no matter what. Maybe consider using an OpenVMS solution? Maybe some performance consulting?

( Is the 2Ghz the desktop you use to connect to an OpenVMS system? :-)


hth,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting.