getting around alpha vms P0 memory limit

[David Ben-Nathan]

One of the main batch programs of the system I work on loads a lot of data into arrays (its an Alpha VMS Cobol program). These arrays keep having to be extended and we are getting close to the P0 limit and failures have occurred recently because this limit has been exceeded. I am looking at ways of getting around this limit and trying to find out about various options like using 1. global sections or 2. if it's possible to use shareable images. I'm not a whizz at this very technical stuff and frankly don't understand what I'm reading on various sites. Global sections is a possibility and I have examples written in house that could be cloned, but for the amounts of data needed to be loaded in this case is it a good idea to load 100 million bytes into a global section - can the system handle it ? Also, if the program loaded more than 1000 million bytes, would it then need special calls that use 64 bit mapping. I take it that this sort of problem is not that uncommon and would be really grateful for suggestions of what can be done or where to read up on these topics [sites that are not too hard to understand].

[Volker Halle]

David,

starting with OpenVMS Alpha V7.0, you can use P2 space (up to 4 TB on OpenVMS Alpha) for process-private data storage.

Please have a look at Chapter 10 and 11 of the HP OpenVMS Programming Concepts Manual:

http://h71000.www7.hp.com/doc/82final/5841/5841pro.html#first_page

Volker.

[Hoff · Accepted solution]

You have ten kilograms of data and a five kilogram address space. That's never fun.

--quote the COBOL reference manual--
"The OpenVMS Alpha operating system (as of Version 7.0) and OpenVMS I64 can dynamically allocate data in 64-bit address space. HP COBOL does not support data in 64-bit address space, but limited use of it may be made with the USAGE IS POINTER-64 clause on OpenVMS Alpha and I64. You might need to describe a data item as USAGE IS POINTER-64 if your program interfaces with an application in another language that requires a 64-bit pointer. Then you would use the SET statement or a VALUE clause to assign the address of a static COBOL variable to the pointer variable. The pointer variable can be passed to a routine whose interface definition requires a 64-bit pointer.

You can also use an appropriate system service or the Run-Time Library routine LIB$GET_VM_64 to allocate data and store the address in the pointer variable. Because COBOL does not support dynamic allocation, there is no way to dereference the pointer and access the allocated data. However, you can pass the pointer to other languages that require a 64-bit pointer in 64-bit address space."
--

AFAIK, COBOL lacks 64-bit addressing support.

Technically, you can access P0 space and part of P1 (stack) space here; somewhere between 30 (P0 space) and 31-bit (P0 and P1 space, less the not insubstantial quantity of "other stuff" residing in P1) addressing.

Yes, this quantity of data can be directly addressed in a 64-bit language. With less address space available than necessary, you get to learn what you can have in memory, and manage it directly. (I've been through this on Apple II, RSX-11M & M+, Mac OS X prior to 64-bit, with VAX, and with Alpha prior to V7.0.)

OpenVMS Alpha V7.0 and later can address 64-bit data, but (without access to 64-bit addressing) you can't get at P2 space without (also) mapping that data into P0 space.

Which means you're either mixing your COBOL with a language that can access this quantity of data, or you're segmenting your data and your routines, and (if you're segmenting) you're probably also looking at implementing what amount to overlays; your own application-specific memory management and address-space re-use. With mapping, you're "winking" data sections into and out of 32-bit space as you need to access the data within. Or you can use I/O to load and unload the data into buffers, but (depending on the data access pattern) that's going to consume I/O bandwidth.

Depending on the data access patterns here, you can use routine-level segmentation (stack-local storage) or global storage with file-based or section-based backing (and "winking", for instance), or splitting up into multiple cooperating processes, or other such.

You might well be familiar with this stuff, but (if not) here are some introductory materials related to shareable images and to virtual memory management:

http://labs.hoffmanlabs.com/node/163
http://labs.hoffmanlabs.com/node/228

Stephen Hoffman
HoffmanLabs LLC

[David Ben-Nathan]

Thanks very much for all of that - that's very helpful and appreciated. I'll check it out next week in the office.

[Ian Miller.]

I remember overlays from RSX11M - not good.

Best do this in a language that will deal with 64 bit virtual addresses and load the data into P2 or S2.

[Robert Brooks_1]

I remember overlays from RSX11M - not good.

--

TKB for ever and ever and ever . . .

[Hein van den Heuvel]

>> its an Alpha VMS Cobol program)

What version of OpenVMS ?

>> failures have occurred recently because this limit has been exceeded.

I trust you already acertained that you are hitting the 1GB boundary, but I'd like to verify because so far you did not utter two magic words: VASFULL and FREP0VA
Do you know what those refer to?

>> These arrays keep having to be extended

Is there an end in sight?
Are the arrays 'tight'.
For example, some integers or dates stored as
text by now more appropriatly be store as binaries, not just for space required, but also for the time needed to access them.

> getting around this limit and trying to find out about various options like using 1. global sections or 2. if it's possible to use shareable images.

Probably global section, but those are tedious in Cobol. You may want to use a service routine written in a different language (C, Macro,...) to request access to an array element.

>> I'm not a whizz at this very technical stuff and frankly don't understand what I'm reading on various sites.

Consider engaging a consultant to get you over the hump?

>> dea to load 100 million bytes into a global section - can the system handle it ?

Yes, easily.

>> Also, if the program loaded more than 1000 million bytes, would it then need special calls that use 64 bit mapping.

Yes, correct, and to make live easier for Cobol, you may want to wrap that as suggested earlier.

Or.. you could have multiple sections in memory and map them as needed in the same address windows. Overlays! Yuck!

>> I take it that this sort of problem is not that uncommon

Actually, it is. Folks that are heading that way tend to figure this out early on and opt for alternatives (files!?).

The only times I have run into VASFULL problems it was with RMS global buffers involved, 'laddering' Va address usage... but that is pretty much fixed in VMS V8.2
If RMS is a used (often is for cobol) the you can possibly save a Megabyte or 50 by helping it to allocate all of it's buffers in P1 space, but if the growth it uncontrolled, then saving a megabyte here or ther eis now good enough.

Hope this helps some,
Hein van den Heuvel ( at gmail dot com )
HvdH Performance Consulting.

[David Ben-Nathan]

Again, thanks very much for that response - fortunately as you mentioned the arrays are not tight and a lot can be done to save space - it is a matter of planning ahead though and knowing what options there are.

[John Gillings]

David,
If you're doing work on the data structure, I'd strongly recommend abstracting it into a procedural interface. Hide the physical details behind routine calls to store or fetch data.

That will allow you to change the data structures as required without having to change the application logic.

[John McL]

David, you've not indicated why you need this much data in memory simultaneously, nor why it should continue to grow. I suspect a report generating program but it's not clear.

Are other jobs expected to run on the machine at the same time? The first level trimming by the memory management subsystem could have your working set size bouncing around all over the place and potentially put a big demand on pagefiles and therefore on the I/O subsystem.

You might find that redesigning the architecture of the processing is a big gain without much pain. I'm thinking particularly of some kind of preprocessing of your data so that the task becomes more sequential. I did this years ago and took an 8-hour job down to 6 minutes.

[Willem Grooters]

I've programmed 'dynamic array' in FORTRAN years ago, using LIB$CREATE_ZONE and LIB$GET_VM, rabging form 1000 to 100.000 elements; both allow for 64 bit addresses.
I'm not sure whether this holds for COBOL, but AFAIK, the language does contain the required types.
If you handle the data seqentially, think of using linked lists, that you can position in P2 space.

[Hein van den Heuvel]

COBOL does not do pointer variables / dereferencing.
When using Global sections or GETVM you have to 'cheat' by calling subroutines to pass the address of a structure or by calling LIB$COPY / MOVC3 functions to copy the data into a working-storage variable, to act on it.

There is little point speculating until David (and we) learn pertinent details.

>> These arrays keep having to be extended and we are getting close to the P0 limit and failures have occurred recently because this limit has been exceeded.

"extended" how ? Source code changes ?
"failures" how ? Error messages ? Link time ? Run time, Application or systems level?
"have occured" when ? all the time ? during SORT ? During data load ...

My impression certainly is a Virtual Address issue, hitting the 1GB P0 limit. Folks just do not tend to make that up... but they could, to hide lack of true understanding. It could still be a simple PAGFILQUO or such problem until we hear pertinent details.

Cheers,
Hein.

[John Gillings]

P0 space limits are a real issue. We see far too many processes hit the wall as we scale up workloads. It's just a reminder that OpenVMS is really a 32 bit OS which has had some 64 bit stuff bolted in the middle.

It's a very clever way of extending the address space, BUT no one has yet found a simple way to automagically make a 32 bit program exploit 64 bits. There's not a lot of incentive for software suppliers to go back over old code and rework it to use 64 bit space, then redo all the necessary testing.

I've attached a modified version of a procedure we use to monitor the P0 consumption of processes. I've removed all the site specific stuff, but without much testing. You will want to update the alarm code to suit your site.

[David Ben-Nathan]

Thanks again for all the help so far. fyi, the arrays keep growing because customer base is increasing and also more and more functionality keeps getting added all the time. I have created a routine today that allocates P2 space calling SYS$_REGION_64 and SYS$_EXPREG_64. Can someone tell me what the system service calls are to actually start populating the area reserved with data and also to read that data back. Someone mentioned LIB$MOVC3 but that doesn't seem to take quadword arguments and I take it that there are routines that should be used with the ones I've used above. Also, reading chptr 10 of the VMS Programming Concepts manual, it says that P2 is part of process private space. Some of the flags on the calls for the routines above specify protection of the region created and I was wondering why that's necessary if it's all part of a process's own private space. Also, there seem to be a number of similar system service calls and it's hard to tell them apart [eg LIB$GET_VM, similar routines for creating Zones etc]. One last question - what are the limits on these routines - presumably at some point they could fail because there is not enough contiguous space available in memory ?

[John Gillings]

David,

See routines LIB$CREATE_VM_ZONE_64,
LIB$GET_VM_64 and other related routines:

$ HELP RTL LIB$ LIB$*VM*64

and, for more detail in the LIBRTL reference.

To move data around, use OTS$MOVE3 or OTS$MOVE5, they handle 64 bit addresses for source and destination arguments, and the length argument can be up to 2GB. Note that the length is passed by immediate value.

Have a look through the various porting manuals for details of handing 64 bit values. Beware, this might look very ugly in Cobol. Maybe consider writing your 64 bit stuff in another language, with better support for pointers, and access it as a procedural interface from Cobol.

[Hoff]

Given the underpinnings of a virtual memory system, it is typically entirely feasible to choose double-map the data as required.

And given I could not locate a readily available source code example using the 64-bit system services to double-map virtual memory between P2 space and P0 space, I wrote one:

http://labs.hoffmanlabs.com/node/1413

This is the "winking" technique that I referenced earlier.

There is certainly some overhead here with the system service mapping, but hauling around large tracts of virtual memory (via movc3 or movc5 or otherwise) is going to have its own and substantial issues around memory paging and around processor overhead.

You can't map the whole of P2 space into P0 space, but you can certainly map parts of P2 space and "wink" your buffers into P0 space as needed.

Using this technique as part of an application-defined programming interface (whether shareable image or otherwise) means you can keep a consistent interface for your code, and the COBOL code can be largely oblivious to the use of 64-bit address space. And you will need to add some code written in a 64-bit capable language into the COBOL application here to deal more directly with P2 space.

[David Ben-Nathan]

Thanks very much for all the replies, in particular to Stephen Hoffman - I have taken a look at the routine you wrote and will see if I can work it into a Cobol routine [or maybe a cobol program could call a C routine to handle accessing the P2 memory. Sorry I couldn't reply sooner but I had not time during the week.