getting around alpha vms P0 memory limit

[David Ben-Nathan]

One of the main batch programs of the system I work on loads a lot of data into arrays (its an Alpha VMS Cobol program). These arrays keep having to be extended and we are getting close to the P0 limit and failures have occurred recently because this limit has been exceeded. I am looking at ways of getting around this limit and trying to find out about various options like using 1. global sections or 2. if it's possible to use shareable images. I'm not a whizz at this very technical stuff and frankly don't understand what I'm reading on various sites. Global sections is a possibility and I have examples written in house that could be cloned, but for the amounts of data needed to be loaded in this case is it a good idea to load 100 million bytes into a global section - can the system handle it ? Also, if the program loaded more than 1000 million bytes, would it then need special calls that use 64 bit mapping. I take it that this sort of problem is not that uncommon and would be really grateful for suggestions of what can be done or where to read up on these topics [sites that are not too hard to understand].

[Volker Halle]

David,

starting with OpenVMS Alpha V7.0, you can use P2 space (up to 4 TB on OpenVMS Alpha) for process-private data storage.

Please have a look at Chapter 10 and 11 of the HP OpenVMS Programming Concepts Manual:

http://h71000.www7.hp.com/doc/82final/5841/5841pro.html#first_page

Volker.

[Hoff · Accepted solution]

You have ten kilograms of data and a five kilogram address space. That's never fun.

--quote the COBOL reference manual--
"The OpenVMS Alpha operating system (as of Version 7.0) and OpenVMS I64 can dynamically allocate data in 64-bit address space. HP COBOL does not support data in 64-bit address space, but limited use of it may be made with the USAGE IS POINTER-64 clause on OpenVMS Alpha and I64. You might need to describe a data item as USAGE IS POINTER-64 if your program interfaces with an application in another language that requires a 64-bit pointer. Then you would use the SET statement or a VALUE clause to assign the address of a static COBOL variable to the pointer variable. The pointer variable can be passed to a routine whose interface definition requires a 64-bit pointer.

You can also use an appropriate system service or the Run-Time Library routine LIB$GET_VM_64 to allocate data and store the address in the pointer variable. Because COBOL does not support dynamic allocation, there is no way to dereference the pointer and access the allocated data. However, you can pass the pointer to other languages that require a 64-bit pointer in 64-bit address space."
--

AFAIK, COBOL lacks 64-bit addressing support.

Technically, you can access P0 space and part of P1 (stack) space here; somewhere between 30 (P0 space) and 31-bit (P0 and P1 space, less the not insubstantial quantity of "other stuff" residing in P1) addressing.

Yes, this quantity of data can be directly addressed in a 64-bit language. With less address space available than necessary, you get to learn what you can have in memory, and manage it directly. (I've been through this on Apple II, RSX-11M & M+, Mac OS X prior to 64-bit, with VAX, and with Alpha prior to V7.0.)

OpenVMS Alpha V7.0 and later can address 64-bit data, but (without access to 64-bit addressing) you can't get at P2 space without (also) mapping that data into P0 space.

Which means you're either mixing your COBOL with a language that can access this quantity of data, or you're segmenting your data and your routines, and (if you're segmenting) you're probably also looking at implementing what amount to overlays; your own application-specific memory management and address-space re-use. With mapping, you're "winking" data sections into and out of 32-bit space as you need to access the data within. Or you can use I/O to load and unload the data into buffers, but (depending on the data access pattern) that's going to consume I/O bandwidth.

Depending on the data access patterns here, you can use routine-level segmentation (stack-local storage) or global storage with file-based or section-based backing (and "winking", for instance), or splitting up into multiple cooperating processes, or other such.

You might well be familiar with this stuff, but (if not) here are some introductory materials related to shareable images and to virtual memory management:

http://labs.hoffmanlabs.com/node/163
http://labs.hoffmanlabs.com/node/228

Stephen Hoffman
HoffmanLabs LLC

[David Ben-Nathan]

Thanks very much for all of that - that's very helpful and appreciated. I'll check it out next week in the office.

[Ian Miller.]

I remember overlays from RSX11M - not good.

Best do this in a language that will deal with 64 bit virtual addresses and load the data into P2 or S2.

[Robert Brooks_1]

I remember overlays from RSX11M - not good.

--

TKB for ever and ever and ever . . .

[Hein van den Heuvel]

>> its an Alpha VMS Cobol program)

What version of OpenVMS ?

>> failures have occurred recently because this limit has been exceeded.

I trust you already acertained that you are hitting the 1GB boundary, but I'd like to verify because so far you did not utter two magic words: VASFULL and FREP0VA
Do you know what those refer to?

>> These arrays keep having to be extended

Is there an end in sight?
Are the arrays 'tight'.
For example, some integers or dates stored as
text by now more appropriatly be store as binaries, not just for space required, but also for the time needed to access them.

> getting around this limit and trying to find out about various options like using 1. global sections or 2. if it's possible to use shareable images.

Probably global section, but those are tedious in Cobol. You may want to use a service routine written in a different language (C, Macro,...) to request access to an array element.

>> I'm not a whizz at this very technical stuff and frankly don't understand what I'm reading on various sites.

Consider engaging a consultant to get you over the hump?

>> dea to load 100 million bytes into a global section - can the system handle it ?

Yes, easily.

>> Also, if the program loaded more than 1000 million bytes, would it then need special calls that use 64 bit mapping.

Yes, correct, and to make live easier for Cobol, you may want to wrap that as suggested earlier.

Or.. you could have multiple sections in memory and map them as needed in the same address windows. Overlays! Yuck!

>> I take it that this sort of problem is not that uncommon

Actually, it is. Folks that are heading that way tend to figure this out early on and opt for alternatives (files!?).

The only times I have run into VASFULL problems it was with RMS global buffers involved, 'laddering' Va address usage... but that is pretty much fixed in VMS V8.2
If RMS is a used (often is for cobol) the you can possibly save a Megabyte or 50 by helping it to allocate all of it's buffers in P1 space, but if the growth it uncontrolled, then saving a megabyte here or ther eis now good enough.

Hope this helps some,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting.

[David Ben-Nathan]

Again, thanks very much for that response - fortunately as you mentioned the arrays are not tight and a lot can be done to save space - it is a matter of planning ahead though and knowing what options there are.

[John Gillings]

David,
If you're doing work on the data structure, I'd strongly recommend abstracting it into a procedural interface. Hide the physical details behind routine calls to store or fetch data.

That will allow you to change the data structures as required without having to change the application logic.

[John McL]

David, you've not indicated why you need this much data in memory simultaneously, nor why it should continue to grow. I suspect a report generating program but it's not clear.

Are other jobs expected to run on the machine at the same time? The first level trimming by the memory management subsystem could have your working set size bouncing around all over the place and potentially put a big demand on pagefiles and therefore on the I/O subsystem.

You might find that redesigning the architecture of the processing is a big gain without much pain. I'm thinking particularly of some kind of preprocessing of your data so that the task becomes more sequential. I did this years ago and took an 8-hour job down to 6 minutes.