PGFLQUOTA leak with FTP

[Hari Shankar S]

Thank you Volker and John for the info. Can you just confirm this one thing more?

Is it true that "LIB$FREE_VM never gives memory allocated by LIB$GET_VM back to the system when it is freed. It keeps it for its future use"?

Do we have to use LIB$CREATE_VM_ZONE, then LIB$GET_VM specifying the zone-id from the LIB$CREATE_VM_ZONE, and then call LIB$DELETE_VM_ZONE with zone-id to properply release the memory back to the system?

[Volker Halle]

Hari,

do you know about the OpenVMS Heap Analyzer ? See the HP OpenVMS Debugger Manual:

http://h71000.www7.hp.com/doc/82final/4538/4538pro_contents_003.html#toc_chapter_12

This will allow you to look at each memory allocation/deallocation in detail. There is even a Sample Session (in chapter 12.5), which shows how to find a memory leak.

Volker.

[Volker Halle]

Hari,

if you carefully read the documentation OpenVMS RTL (LIB$) manual, you'll find that all the LIB$*VM* routines are layered on LIB$GET_VM_PAGE and LIB$FREE_VM_PAGE to obtain blocks of virtual memory for use by the LIB$*VM* routines.

The description of LIB$FREE_VM_PAGE says: the page or pagelets are returned to the processwide pool and are available to satisfy subsequent calls to LIB$GET_VM_PAGE.

This implies, that the virtual address space associated with those pages is not being deleted.

There are no $DELTVA calls in the [LIBRTL] facility, except some to handle allocation failures.

Volker.

[Volker Halle]

Hari,

the more recent versions of OpenVMS also ship the LIBRTL symbol table file. Using this, you can obtain the LIB$*VM* statistics counters from any process using SDA.

First check, whether SYS$LIBARY:LIBRTL.EXE and LIBRTL.STB have the same date. If not, you have installed some patch including LIBRTL.EXE, which did not also include the associated .STB file. This will most liekly cause the following to not provide the expected results.

$ ANAL/SYS
SDA> SET PROC TCPIP$FTP_1 ! or any other process
SDA> READ/IMAGE SYS$LIBRARY:LIBRTL

SDA> SHOW SYMB/ALL/VALUE LIB$$GL
...
SDA> SHOW SYMB/ALL/VALUE LIB$$GQ
...
SDA> EXIT

The following symbols represent the LIB$*VM* statistics:

LIB$$GL_GETVM_C - Number of successful calls to LIB$GET_VM
LIB$$GL_FREVM_C - Number of successful calls to LIB$FREE_VM
LIB$$GL_VMINUSE - Bytes still allocated

LIB$$GL_GETPG_C - no. of calls to LIB$GET_VM_PAGE
LIB$$GL_FREPG_C - no. of calls to LIB$FREE_VM_PAGE
LIB$$GL_PGINUSE - no. of VM pages still allocated

and the same as above for the LIB$*VM*_64 calls (allocation from P2 address space):

LIB$$GQ_GETVM_C_64
LIB$$GQ_FREVM_C_64
LIB$$GQ_VMINUSE_64

LIB$$GQ_FREPG_C_64
LIB$$GQ_GETPG_C_64
LIB$$GQ_PGINUSE_64

Using this technique, you can determine, if the expansion of the P0 virtual address space of TCPIP$FTP_1 is actually caused by calls to the LIB$*VM* routines.

Volker.

[Richard W Hunt]

While I cannot speak for the contents of the FTP server process or the exact sequence of calls for LIB$ VM routines, I can say with absolute certainty that this exact problem has been seen since VMS v 2.0 and there are very few solutions. The problem is in the garbage collection routines that are used when you release something to VM.

As described earlier, it is partly due to the "random" allocation of different packet sizes. They might actually be only a few different sizes, but they arrive and release randomly because of the nature of a server process with varying numbers of files to PUT and GET.

My first run-in with the memory "leak" was with a big honkin' FORTRAN program and dynamic text-string allocation using string descriptors. Essentially, the string allocation paradigm cause memory to be very quickly checker-boarded with fragments that were too small to be re-used.

A second symptom occurred after a very long time - the program started spending a LOT of time in the LIB$FREE routine doing a huge number of page faults - because VM had grown high enough to overflow the WQQUOTA + WSEXTENT combined, and on this particular machine, enough users were on the system that BORROWLIM and GROWLIM had become significant factors as well. The long time in the VM routines came about because when you DID try to free up a chunk of memory, the free-up routine had to step through the structures of each previously released memory chunk to see if it could merge the chunk being released to any already-released chunks. It actually tries to merge adjacent released chunks. But that means it has to identify and merge the released chunk's neighbors. So it thrashed the system with tons of page faults stepping through a gazillion little memory pool fragments.

At the time my solution was simple. Rather than worry about releasing memory, just let the program EXIT. That way, it would release its private pool of memory wholesale rather than retail.

This problem is really a flaw ... well, a feature ... of the algorithm as applied to random-arrival, random-departure, varying sized memory chunks.

My suggestion is this: If there is a time to schedule it such that you are pretty sure no one will get tripped up by it, terminate the FTP server and restart it. That will reset the size of the process.

I have an idle-job checker. It runs every so often and is allowed to decide if the FTP server has been idle. I also know from system profiling when it is and isn't safe to whack that process if I need to. It might not be possible for you to profile it, but short of totally re-inventing the VM chunk allocation and reclamation algorithm, I don't know how you would be able to do any better with the code. Short of doing a one-size-fits-all method by creating fixed-size lookaside lists that are as big as you'll ever need, and just recycle them.

[Hari Shankar S]

This query is closed.