PGFLQUOTA limit with NEW - DISPOSE

[john nicoara]

Description:

I have a PASCAL program with a big array of records
(~ 8 milions x 32 bytes = 256 MBytes).
This array is dynamically allocated with NEW.
Each day, this table is new initialized with a DISPOSE and a NEW.

Problem:

By DISPOSE, the value of PGFL not change.
By NEW, if the number of records is smaller as previous day,
the value of PGFL not change.
But if the number of records is greater as previous day,
the value of PGLF duplicate.
Like this, the PGFL reach quickly the quota limit.

Test-Program:

Attached is a small test-program.
In the main-loop, the program ask how many millions records
do allocate (NEW) and DISPOSE.

Environment -> HP Pascal Alpha V6.0-107 on OpenVMS Alpha V8.3

Test-1:
-------
Mio Free Free
Step Rec [MB] PGFLQUOTA Memory [MB] Pagefile [MB]
-----------------------------------------------------------------
Start - - 6064 1012 1617
Allocate 2 61 131232 951 1617
Allocate 1 30 131232 951 1617
Allocate 1 30 131232 951 1617
Allocate 3 91 319024 860 1617
Allocate 3 91 319024 860 1617
Allocate 2 60 319024 860 1617
Allocate 4 122 569408 737 1617
-----------------------------------------------------------------


Test-2:
-------
Mio Free Free
Step Rec [MB] PGFLQUOTA Memory [MB] Pagefile [MB]
-----------------------------------------------------------------
Start - - 6064 1012 1617
Allocate 4 122 256736 737 1617
-----------------------------------------------------------------


Question:

I make something wrong ?
Or is a bug with Virtal-Memory-System in VMS :-)

[John Reagan · Accepted solution]

Pascal's NEW and DISPOSE are simply jackets over the top of LIB$GET_VM and LIB$FREE_VM.

LIB$FREE_VM never gives memory back to the system when it is freed. It keeps it for future use. So looking at current quotas to look for "memory leaks" will lead you astray.

You should use LIB$SHOW_VM instead.

Changing the example to use LIB$SHOW_VM instead of your SHOW_QUOTA/LIB$GETJPI stuff shows that after the Pascal RTL allocates a handful of small items (like the FAB, RAB, NAM, etc. for OUTPUT), all you see are your NEW and DISPOSEs correctly mapped into calls to LIB$GET_VM/LIB$FREE_VM.

As I mentioned, since LIB$VM keeps memory available for future use, continually asking for larger and larger sections will end up fragmenting memory. For various reasons, LIB$VM won't "extend" a previously allocated/freed region.

[Hein van den Heuvel]

John Reagan wrote>> the Pascal RTL allocates a handful of small items (like the FAB, RAB, NAM, etc. for OUTPUT),

In the test program this is ofcourse really minimal. However, in the real application this probably has more impact and may cause ongoing fragmention.
You can reduce that effect, and maybe even make it go away, by using the LINKER OPTION : IOSEGMENT
Pick a large value: 50,000?
Max is currently 64K (hackable in the image header).
For existing images you can try raising the SYSGEN selected DEFAULT: MCR SYSGEN SHOW IMGIOCNT

Seeing this behaviour, you may want to just always start teh program with a very large NEW, large enough for the largest expected load in the next few weeks/months.then dispose that fresh large chunk and start doing the normal work.

Not also, how the test output is further misleading in the Free mem and Pagefile columns. This is because while the memory is allocated, it has not been touched yet. So it is 'very' virtual and not representative of the real application being in action.

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Jon Pinkley]

John,

Is there any way using Pascal features to do a LIB$DELETE_VM_ZONE?

If not, then it would see the only thing he can do is to replace his NEW with calls to LIB$CREATE_VM_ZONE, then LIB$GET_VM specifying the zone-id from the LIB$CREATE_VM_ZONE, and then calling LIB$DELETE_VM_ZONE with zone-id where he is currently using DISPOSE.

If the only NEW and DISPOSE are at the daily initialization, then there shouldn't be too much extra overhead with this approach.

The alternative is to do an initial NEW request with a value larger than you ever expect to need. John, do you know if the NEW actually touches the memory or just maps it? If it never gets faulted in until the program references the memory, requesting more than he needs at program startup time may not be a bad approach. It will still be charged against his pgflquota, but that may be the extent of the penalty vs. the extra work of creating a zone and deleting it.

Jon

[John Gillings]

john,

What you're seeing is a phenomenon called "checker boarding". It's a characteristic of some patterns of virtual memory use where "correct" allocations and deallocations result in continually expanding VM.

The simplest way to understand it is by example. Suppose I have a program which allocates two different sized chunks of memory - one large "L", and one small "S".

The default allocation algorithm for most general VM management routines is a LIFO buffer of free blocks using "first fit".

So, if the pattern of allocations is:

NEW S
NEW L
DISPOSE S
DISPOSE L
NEW S
NEW L
DISPOSE S
DISPOSE L
etc...

Thus, at the time of the second "NEW S", the free L chunk is at the front of the free list, and it's bigger than the S request, so we carve of an S sized chunk, leaving the remainder on the free list. When the next "NEW L" is performed, the partial chunk isn't big enough to satisfy the request, so a whole new allocation is made. As the pattern repeats, we accumulate the remainders which aren't big enough for an L request, but are always behind an L packet when an S request is made.

The increasing size pattern does something similar.

This is not a bug. The algorithm is good for most general use. There is no algorithm which performs perfectly for all possible usage patterns.

What you can do about it is use your knowledge of the way memory is used by your application to optimize VM use. In the case of your big array, instead of NEW and DISPOSE, why not just allocate it once and keep it? If it gets bigger over time, perhaps an array isn't the best choice of data structure. Perhaps a list of some sort?

Remember that, by default, your data is kept in P0 space, which is only 1GB. Your object is a quarter of the entire region, which is a bit tight. Maybe you should think about moving it into P2 space, using 64 bit pointers?

[John Reagan]

The Pascal RTL routines for NEW take the size from the generated code (for schema types as in the posted example, the size could be a run-time value) and adds 8 or more bytes to it. We need to save the real allocated size so we have it again when DISPOSE is called. We need to save the schema discriminants in case the program asks for them. NEW also zeros out the memory.

For NEW, besides the compiler-generated call to PAS$NEW2, it also generates code to apply any initial-state values to the variable. You wouldn't get that if you use somehting other than NEW.

There is also a rare path for people using the very old MARK and RELEASE support (sorta an ancestor of DELETE_ZONE) where we'd allocate even a few more longwords at the top.

So in general, replacing NEW/DISPOSE with direct calls to LIB$GET_VM/LIB$FREE_VM can be difficult.

[Willem Grooters]

Allocating a large amount of memory to store an array of arbitrary size, I used lib$create_vm_zone to get a contiguous address range and allocated memory from that zone using lib$get_vm. After use, one call to lib$delete_vm_zone returns all allocated pages to the system and removes the address space.

Perhaps that's an idea to prevent your problem.

[John Reagan]

Uh, LIB$DELETE_VM_ZONE doesn't delete the address space. The only call to $DELTVA you'll find in any of the LIB$VM routines is in the error path of LIB$GET_VM_PAGE right after it calls $EXPREG. If the expand fails for some event, the error path tries to delete the address space right before it returns LIB$_INSVIRMEM to the caller.

[Willem Grooters]

@John: Documentation says:

The Delete Virtual Memory Zone routine deletes a zone from the 32-bit virtual address space and returns all pages on VAX systems or pagelets on Alpha and I64 systems owned by the zone to the processwide 32-bit page pool.

so the memory will be re-usable (What would happen if:

* CREATE_VM_ZONE
* DELETE_VM_ZONE
* CREATE_VM_ZONE (same size) --> Will that use the same area just deleted?

[john nicoara]

Thank you very much for all you gut information
and help for my Problem.
I have test to use library routines for VM.


Test with LIB$GET_VM / LIB$FREE_VM:

I have try to replace NEW and DISPOSE with LIB$GET_VM and LIB$FREE_VM.
This work exactly same. The only problem is that the internal PASCAL ARRAY structure
is not correctly initialized and PASCAL create a index out of range.
Following PASCAL functions return a wrong value: SIZE, LOWER and UPPER.


Test with LIB$CREATE_VM_ZONE / LIB$DELETE_VM_ZONE

I have try to add CREATE and DELETE of a VM-Zone.
Unfortunately the DELETE_VM_ZONE not give the memory free !!
(or i make an error in my Test-Program).
By next CREATE, reserve ALWAYS new memory,
same if the array is smaller or greater.

Test:
-----
Mio Free
Step Rec [MB] PGFLQUOTA [MB]
-----------------------------------------------------------------
Start - - 8304 4
Allocate 2 61 135328 66
Allocate 1 30 198880 97
Allocate 1 30 262400 128
Allocate 3 91 450912 220
Allocate 4 120 702944 343
Allocate 4 120 891456 435
Allocate 2 62 -> %LIB-F-INSVIRMEM, insufficient virtual memory
-----------------------------------------------------------------

I think that i use your the "workaround" proposed:
- Allocate the array 20% bigger and not need
NEW and DISPOSE for next few weeks/months.

[John Reagan]

RE: Willem

Note it says the 'processwide pool', not delete the address space. The memory is still allocated and under control of the LIB$VM. I can't say for sure without trying, but a DELETE_ZONE followed by a similar CREATE_ZONE may very well use the same memory.

[John Reagan]

RE: John

When Pascal allocates a schema type (aka "run-time sized type") on the heap, it allocates additional storage that is at "negative" offsets from the pointer returned from NEW. That is where we store the discriminant values used by SIZE, UPPER, etc. You should only use NEW to allocate schema types from the heap.

[John Gillings]

re: Willem,

>so the memory will be re-usable (What >would happen if:
>
>* CREATE_VM_ZONE
>* DELETE_VM_ZONE
>* CREATE_VM_ZONE (same size) --> Will that
>use the same area just deleted?

Maybe! You have no control over what asynchronous threads might do between your DELETE and CREATE. They may allocate a chunk of whatever you just deallocated, leaving it too small to satisfy the CREATE request. If the pool hasn't been coalesced with other deallocations, and the requested zone is large, chances are you'll need to expand memory to satisfy the request.

Even if it seems to work most of the time, you can't depend on undocumented, coincident behaviour. If you really want to control virtual memory at that level, you need to drop down a level and work with $CRETVA, $DELTVA and maybe $MGBLSC. Not for the faint of heart!

What this whole issue comes down to is how to manage a data structure which potentially grows over time. Here are a few possibilities...

1) Design a data structure that can be expanded incrementally from discontiguous memory. For example, a list or tree with discrete elements.

2) Deallocate and reallocate a larger entity when necessary - down side is there is a limit to how big these entities can be in 32 bit space. P0 space is only 1GB, and if you need to allocate the new object first, then copy the contents of the old before deallocating it, that means you need twice the space! Not useful for objects over (say) about 100MB because there isn't enough headroom

3) Allocate an object that is as large as you'll ever need and manage the expansion yourself. This is "cheap" as memory you don't touch won't actually exist, BUT again you're limited by the 1GB P0 space.

4) Learn about 64bit pointers and memory regions. This all becomes a non-issue, as the memory regions are MUCH larger tha you're likely to ever need (but you need to make sure you have enough pagefile space to support the size of virtual memory).

re john: Test with LIB$GET_VM / LIB$FREE_VM

Doing your own GET_VM/FREE_VM can be useful in some circumstances, but you need to know how it interacts with your programming language. As John R has said, Pascal does a lot of things for you in the background, which mostly make life much easier and protect you from numerous pitfalls. You need a good reason to switch off the autopilot.

I'm fairly sure Pascal understands 64 bit addressing. This maybe as simple as adding an attribute ([QUAD,ADDRESS64]?) to the pointer of the object.