Virtual address space when running a C language image

[Wim Van den Wyngaert]

Just tested free() too.
Frees nothing visible (WS, PM, VM).

Infact, I can still use the *free*d memory after the free.

Wim

[Hein van den Heuvel]

>> I interprete this as : added to VM but not to the working set (except for the overhead).

Wim, it may be time to refresh those VM concepts!

The WS is just a window to look at you pages.
It doesn't matter how many pages there are to look at.
VMS will kindly make the window larger 'while convenient', but it is still only a windows and can be shrunk ($adjwsl) at any time without loosing any memory.
Now the MODIFIED and FREE pages would have going down, and the page page load up while doing this.


>> Just tested free() too.
Frees nothing visible (WS, PM, VM).

It only make the next mallocs free!

It's just bookkeeping in already alloacted memory. Just a promiss that the program will not use that memory again untill re allocated from the private memory through a future malloc. The system memory where it came from was once created trhough $EXPREG and stays. That's what this topic is all about!

>> Infact, I can still use the *free*d memory after the free.

Well duh!
Surely you did not expect otherwise?!
Many programs actually do this.
When they do, it is called 'a bug'

Wim

[Wim Van den Wyngaert]

Hein,

You're right : I should refresh it (again).
The problem is I rarely need the knowledge over here in the bank.

In any case, I would expect that VMS doesn't allow to use memory you don't have (it's free'd, t.i. given back) and I would expect an access violation.

I understand the WS principle but still would expect that after the free() the pages would be marked unused via f$gethpi(ppgcnt).

BTW : you signed with Wim

Wim

[Wim Van den Wyngaert]

During the malloc of 100 MB nothing happens in mon pag.

During the dirty making I get very high page fault rate and equal demand zeo fault rate. The free list drops significantly.

Wim

[Hein van den Heuvel]

The malloc code needs only touch the bytes just before the returned address pointer to store teh size.
If you allocate in chuncks significantly larger than a page, then the bulk of the malloced block does nto get touchs and it just address range.

So there will be a major difference between malloc 1 * 100MB or even 100 * 1 MB versus 100,000 * 1 KB.

Elementary.

Hein.

[Hoff]

Earlier in the thread, there's a comment about malloc and free being quite generic, and potentially too generic for specific requirements. Sometimes even the librtl VM services aren't appropriate.

Wim, you appear to be assuming that the applications and particularly the central librtl virtual memory routines use OpenVMS virtual memory management directly, and not that the librtl VM memory management is layered onto the system virtual memory management. It's layered.

FWIW: there were changes in this area at V7 and in later releases, particularly around various structures having been moved to S2 space and resized far larger. These made VIRTUALPAGECNT and various other structural limits effectively no-ops, except as these structures are mapped to physical storage. This includes changes to the handling of the GBLPAGES parameter, due to the allocation of PTEs from S2 space. See below for a related URL pointer or two.

The librtl VM routines do add blocks of physical pages into the working set as required (and OpenVMS will toss other pages of memory out, as required, and usually into the pagefile or other backing storage), but the heap and structures are layered onto this system virtual memory. The librtl VM routines do allocate and instantiate physical memory as required, and will then overlay heap structures onto that virtual memory.

If you wanted to go at page-level allocations and to allocate memory in blocks of storage of 8192 to 65536 bytes (the size range for a physical memory page), then you could more easily implement access violations. Intercepting and managing memory access akin to the debugger -- for generic heap access, at a granularity below a memory page -- is going to be processor-intensive. But yes, with allocations of at least a memory page (8KB to 64KB), you can let OpenVMS memory management subsystem deal with it all. That's gonna need a whole lot of memory, and potentially trigger checker-boarding and fault-per-access references in virtual memory. TANSTAAFL.

Clearing a page from the middle of the heap doesn't allow the aggregate size of the data structures used to be reduced, and the memory management system can (transparently) take care of reducing physical memory requirements for the working set. Yes, you can make the case for detecting and protecting pages within the heap (once the entire page is freed), but that would add to the complexity of the memory management system, as it doesn't care about pages right now and its garbage collection only allows for coalescing of adjacent unallocated blocks without regard to size or underlying pages.

And you'd still have exposure to cases with a mixture of allocated and unallocated storage all co-resident on the page, and to cases where the heap pointers get stomped on.

The approach I use is to use the librtl VM services directly, and use the poisoning mechanisms available for same. And "fenceposts". This is a compromise, but I really don't want to tangle with my own memory management. And I've extensive experience with implementing my own memory management, and with the weirder corners involved including memory barriers and processor caches and reentrancy. I've also supported the home-grown VM schemes of others. Using librtl VM gets me what I need, and it avoids having to manage the VM myself.

If I really needed it, yes, I'd use VM services directly. But I've found the librtl VM calls and use of VM zones, poisoning and such to be sufficient. Very few applications need or want to deal with $cretva, $deltva, $expreg and friends.

There are various technical discussions around how memory addressing and process management; the model used by OpenVMS certainly isn't the only one around -- there are some pretty slick designs available. Also look around at garbage collection, as that's usually what's involved in freeing memory. For details on how OpenVMS manages its memory and organizes its process address space, see the Internals and Data Structures Manual; the IDSM. The IDSM also has details of how system paged and non-paged pool is managed.

Related URLs:
http://h71000.www7.hp.com/openvms/journal/v5/removing-32-bit-limit.html
http://h71000.www7.hp.com/wizard/wiz_7752.html