Virtual address space when running a C language image

[Susan Abercrombie]

Environment: OpenVMS 7.3-2 Alpha and HP C 6.5.

Assumptions: A C program which uses malloc() and calloc() and free(), and which does not execute user-coded system calls to create and delete memory regions.

Are there circumstances under which the virtual address space be reduced while the program image is executing? Or does free() merely make the freed memory available for other use by the program?

I know that the working set size can be reduced if it has been expanded beyond the default. I have not found anything in the docs (or the forums) which says explicitly that it might or it will not happen.

Thanks for enlightenment!

[Hein van den Heuvel]

>> Are there circumstances under which the virtual address space be reduced while the program image is executing?

No, not really.
If the program did $EXPREG (directly or indirectly through $CRMPSC) then the system can shrink on detach if (and only if) it is the highest memory address in use.You can $DETLVA the space and re-use but it will not really shrink the address range.

>> Or does free() merely make the freed memory available for other use by the program?

Yes, Malloc gets from a pool and expands that pool as needed. (LIB$GETVM --> $EXPREG)
FREE gives back to the pool, but not back to the system

>> I know that the working set size can be reduced if it has been expanded beyond the default. I have not found anything in the docs (or the forums) which says explicitly that it might or it will not happen.

If you know the freeed memory will not be re-allocated for different purposes then you coudl call $ADJWSL to trim low usage pages from the working set, hopefully inclusive of that freed memory. That will make PHYSICAL memory available to other usages, but will not help reduce VIRTUAL memory space.

http://h71000.www7.hp.com/doc/732FINAL/4527/4527pro.html#jun_31

Is there any particular problem you are trying to solve?

Regards,
Hein van den Heuvel
HvdH Performance Consulting.

[Susan Abercrombie]

Thanks, Hein!

I started from a question asked by a customer -- whether if he created an object which is larger than any he will actually use, and then reused the object, he could be sure that after he started his program (and had created the object) he could be sure he would not see an increase in memory. The subsequent discussion implied that he didn't want it to shrink and expand back to that size. Limiting the working set and/or page file quota, which is done to the process rather than the image, is viewed as unsatisfactory.

(Of course, the answer necessarily depends on what else the program does.)

The available statistics are more limited than one might wish. Peak virtual address is provided, but that doesn't help with the current status of the currently running image. I was aware of the possibility of changes, both up and down, in physical memory footprint via the working set. But I was unsure about page file or swap file space required during image execution. And I failed to find it in the docs. Thus the question.

I was told that on some of the platforms supported by our product, the high water mark of memory size becomes the minimum for the program, but that on other platforms memory is reclaimed. I lack specifics.

Thanks very much!

[Hoff]

The OpenVMS LIBRTL virtual memory calls that are used by the C RTL tend to grow process virtual address space (VAS), as the calls save the last deallocation for an immediate reallocation; you'll tend to see the process VAS grow over time. You won't typically see VAS decrease; not with the normal C allocation and deallocation sequence. This based on what the C RTL and LIBRTL engineers have indicated over the years.

LIBRTL does work to coalesce the heap storage where it can, but VM zones and other such knowledge that can be embedded by the application programmer can make this whole process much more efficient. And as Hein writes, you just won't see the C RTL or any other calls into the LIBRTL giving back VAS. (Well, except on process rundown.)

C and LIBRTL have certainly gotten (far) better about how the code manages VAS, but I've almost always found it more appropriate to code my own application specific memory routines using the LIB$*VM* routines. There are other reasons to avoid the direct use of malloc and free, in favor of centralizing calls. Not the least of which involves troubleshooting issues with heap corruptions.

The C standard library memory management calls are good and useful, but can be too generic larger applications. The LIB$*VM* calls are far better here, particularly if you can code some knowledge into the calls and memory management.

There are some common coding techniques, too, such as using a specific VM zone for temporary memory, that can simplify deallocating transient memory. You can flush the whole zone near, say, the main command processing code (for a command-oriented tool), and not have to deal with tracking of individual allocations. And the LIB$*VM* calls have usage statistics and zone verification routines, which the C API lacks. Etc...

[John Gillings]

Susan,

> whether if he created an object which is
>larger than any he will actually use, and
>then reused the object, he could be sure
>that after he started his program (and had
>created the object) he could be sure he
>would not see an increase in memory

You need to be fairly careful when dealing with large objects intermixed with small orders. There's a phenomenon called "checkerboarding" where the "wrong" sequence of allocations and deallocations can result in continual growth of virtual memory, even though the application is correctly deallocating everything it allocates. This occurs if a small allocation request is satisfied as a chunk of a large deallocated object, leaving a free chunk that isn't big enough to satisfy another large request. Worst case, you accumulate chunks of memory that aren't quite the right size.

There is no single allocation algorithm that is optimal for all memory use patterns.

If you understand the memory usage of your application, you can use multiple free lists to manage memory in an application specific manner. Using Hoff's advice to centralise allocation/deallocation allows you to track usage, hide the application specific logic, and easily modify your strategies if they're not working.

[Wim Van den Wyngaert]

Also note that malloc will not give you the memory but only the right to use it. The pages are allocated at the time you dirty your pages (=fill it).

Wim

[GuentherF]

"...which is larger than any he will actually use, and then reused the object, he could be sure that after he started his program (and had created the object) he..."

What's the point here? The state of the object's virtual address space after an image restart? Influence on the process' working set when the image is restarted?

/Guenther

[Hoff]

re: memory allocation and virtual memory and paging

Really? Can you confirm that?

I'd expect malloc (and technically the librtl VM calls that provide the actual malloc code) has to read and then fill in the heap linkages, so the page(s) involved are already instantiated, and have arrived in the working set from the free list. AFAIK, malloc can't defer this heap-pointer update task. So at least part of the allocation and its underlying memory page will be in the working set, and valid, and will have already been modified.

When such is required, the malloc code (via librtl) invokes the create virtual address space and the expand region calls and that allocate and instantiate address space. But by the time malloc completes, I'd expect the process virtual address space (image virtual memory) to be enlarged if and as needed, and to have mapped that virtual memory to physical memory, and to have the physical memory faulted into the working set, and fully valid. And at least some of the virtual memory immediately near the user-allocated portion of the region -- the heap linkages -- would be modified, and eventually flushed out to the pagefile or (if you're running your own heap) whatever backing storage is in use, if the memory was then faulted out.

It's these nearby heap pointers that tend to get stomped on, and why there are regular references to malloc and free discussions over in the ancient OpenVMS Ask The Wizard area, and recommendations for "fenceposts" and direct calls to the librtl VM routines.

If you turn on the system diagnostics, the user contents of the heap can be modified on allocation and deallocation. See the POOLCHECK parameter and its bitmask for that. Use POOLCHECK with great caution; it's a reserved or special parameter, and other code can have strange behaviors. Once in a while somebody'd enable POOLCHECK and trip over something odd or wrong completely elsewhere.

I tend to use the librtl VM services directly, as that gives me very direct control over the initialization of the memory. The librtl VM services can initialize newly-allocated memory, and can overwrite newly-deallocated memory. Very handy for spotting a whole class of heap-related bugs. And the librtl VM services affect only my own application image's heap with functions such as the pattern overwrites, and the heap linkage verification calls.

[GuentherF]

As simple as that?

Q: Does calling free() reduce a process' virtual memory usage?

A: No! Never!

Q: Does calling free() reduce a process' physical memory usage?

A: No! Never!

But I smell some more behind the customer's question. Maybe just a misunderstanding.

/Guenther

[Wim Van den Wyngaert]

Hoff : on an AS 500 with 7.3.

At DCL prompt : WS 16000 used 1440, VM 33840 used, Phys.mem 12369 free.

Run, pause after malloc of 51.200.000 bytes : WS 16000 used 2720, VM 141248 used, PM 12331 free.

Continue until filled with 1's : WS 37600, used 37024, VM 141248 used, PM 6084 free.

I interprete this as : added to VM but not to the working set (except for the overhead).

Wim

[Wim Van den Wyngaert]

Just tested free() too.
Frees nothing visible (WS, PM, VM).

Infact, I can still use the *free*d memory after the free.

Wim

[Hein van den Heuvel]

>> I interprete this as : added to VM but not to the working set (except for the overhead).

Wim, it may be time to refresh those VM concepts!

The WS is just a window to look at you pages.
It doesn't matter how many pages there are to look at.
VMS will kindly make the window larger 'while convenient', but it is still only a windows and can be shrunk ($adjwsl) at any time without loosing any memory.
Now the MODIFIED and FREE pages would have going down, and the page page load up while doing this.


>> Just tested free() too.
Frees nothing visible (WS, PM, VM).

It only make the next mallocs free!

It's just bookkeeping in already alloacted memory. Just a promiss that the program will not use that memory again untill re allocated from the private memory through a future malloc. The system memory where it came from was once created trhough $EXPREG and stays. That's what this topic is all about!

>> Infact, I can still use the *free*d memory after the free.

Well duh!
Surely you did not expect otherwise?!
Many programs actually do this.
When they do, it is called 'a bug'

Wim

[Wim Van den Wyngaert]

Hein,

You're right : I should refresh it (again).
The problem is I rarely need the knowledge over here in the bank.

In any case, I would expect that VMS doesn't allow to use memory you don't have (it's free'd, t.i. given back) and I would expect an access violation.

I understand the WS principle but still would expect that after the free() the pages would be marked unused via f$gethpi(ppgcnt).

BTW : you signed with Wim

Wim

[Wim Van den Wyngaert]

During the malloc of 100 MB nothing happens in mon pag.

During the dirty making I get very high page fault rate and equal demand zeo fault rate. The free list drops significantly.

Wim

[Hein van den Heuvel]

The malloc code needs only touch the bytes just before the returned address pointer to store teh size.
If you allocate in chuncks significantly larger than a page, then the bulk of the malloced block does nto get touchs and it just address range.

So there will be a major difference between malloc 1 * 100MB or even 100 * 1 MB versus 100,000 * 1 KB.

Elementary.

Hein.

[Hoff]

Earlier in the thread, there's a comment about malloc and free being quite generic, and potentially too generic for specific requirements. Sometimes even the librtl VM services aren't appropriate.

Wim, you appear to be assuming that the applications and particularly the central librtl virtual memory routines use OpenVMS virtual memory management directly, and not that the librtl VM memory management is layered onto the system virtual memory management. It's layered.

FWIW: there were changes in this area at V7 and in later releases, particularly around various structures having been moved to S2 space and resized far larger. These made VIRTUALPAGECNT and various other structural limits effectively no-ops, except as these structures are mapped to physical storage. This includes changes to the handling of the GBLPAGES parameter, due to the allocation of PTEs from S2 space. See below for a related URL pointer or two.

The librtl VM routines do add blocks of physical pages into the working set as required (and OpenVMS will toss other pages of memory out, as required, and usually into the pagefile or other backing storage), but the heap and structures are layered onto this system virtual memory. The librtl VM routines do allocate and instantiate physical memory as required, and will then overlay heap structures onto that virtual memory.

If you wanted to go at page-level allocations and to allocate memory in blocks of storage of 8192 to 65536 bytes (the size range for a physical memory page), then you could more easily implement access violations. Intercepting and managing memory access akin to the debugger -- for generic heap access, at a granularity below a memory page -- is going to be processor-intensive. But yes, with allocations of at least a memory page (8KB to 64KB), you can let OpenVMS memory management subsystem deal with it all. That's gonna need a whole lot of memory, and potentially trigger checker-boarding and fault-per-access references in virtual memory. TANSTAAFL.

Clearing a page from the middle of the heap doesn't allow the aggregate size of the data structures used to be reduced, and the memory management system can (transparently) take care of reducing physical memory requirements for the working set. Yes, you can make the case for detecting and protecting pages within the heap (once the entire page is freed), but that would add to the complexity of the memory management system, as it doesn't care about pages right now and its garbage collection only allows for coalescing of adjacent unallocated blocks without regard to size or underlying pages.

And you'd still have exposure to cases with a mixture of allocated and unallocated storage all co-resident on the page, and to cases where the heap pointers get stomped on.

The approach I use is to use the librtl VM services directly, and use the poisoning mechanisms available for same. And "fenceposts". This is a compromise, but I really don't want to tangle with my own memory management. And I've extensive experience with implementing my own memory management, and with the weirder corners involved including memory barriers and processor caches and reentrancy. I've also supported the home-grown VM schemes of others. Using librtl VM gets me what I need, and it avoids having to manage the VM myself.

If I really needed it, yes, I'd use VM services directly. But I've found the librtl VM calls and use of VM zones, poisoning and such to be sufficient. Very few applications need or want to deal with $cretva, $deltva, $expreg and friends.

There are various technical discussions around how memory addressing and process management; the model used by OpenVMS certainly isn't the only one around -- there are some pretty slick designs available. Also look around at garbage collection, as that's usually what's involved in freeing memory. For details on how OpenVMS manages its memory and organizes its process address space, see the Internals and Data Structures Manual; the IDSM. The IDSM also has details of how system paged and non-paged pool is managed.

Related URLs:
http://h71000.www7.hp.com/openvms/journal/v5/removing-32-bit-limit.html
http://h71000.www7.hp.com/wizard/wiz_7752.html