PGFLQUOTA leak with FTP

[Hari Shankar S]

Can anyone help me out with the deleting the virtual memory once used?

What is the difference between LIB$FREE_VM and LIB$DELETE_VM_ZONE ?

Is it true that "LIB$FREE_VM never gives memory back to the system when it is freed. It keeps it for future use." ?

If it is, what is the way out of this?

[Volker Halle]

Hari,

there is NOTHING you can do about TCPIP$FTP_1, if this process would actually have a memory leak and therefore would be consuming it's pagefile quota over time.

If you can prove that this is true and you can prove - or at least demonstrate - this case, document your findings and raise a call with HP. Only TCPIP engineering could do anything about this, if there is a real problem.

Don't waste your time thinking about LIB$*VM* routines, as they all require access to the source code of the application, i.e. TCPIP$FTP_SERVER.EXE.

As a workaround, you could restart the TCPIP FTP service, if the remaining pagefile quota approaches ZERO. Use @SYS$STARTUP:TCPIP$FTP_SHUTDOWN.COM and TCPIP$FTP_STARTUP.COM. Note that this will temporarily disrupt FTP traffic.

You could also raise the SYSGEN parameter PQL_MPGFLQUOTA, if you don't want to restart FTP every once in a while.

I've tested a simple FTP connection to our OpenVMS Alpha V8.2 TCPIP V5.5 system and the remaining pagefile quota of TCPIP$FTP_1 was the SAME before and after the connection.

You should be able to prove your point by:

$ SHOW PROC/QUOTA/ID=
$ DIR/FTP localhost::login.com
$ SHOW PROC/QUOTA/ID=

If you do this 100 times and if the decrease in pagefile quota remains constant for each FTP connection, you've proven your point. You could then even predict, when TCPIP$FTP_1 is going to fail and restart (see above) the FTP service early enough.

Volker.

[Wim Van den Wyngaert]

The leak doesn't seem present on 5.3 eco 2.

Also check sys$sysddevice:[tcpip$ftp]tcpip%ftp_run.log for messaages. And opcom.

Wim

[Hari Shankar S]

Volker

Let us assume that I have access to the codes. No questions on that please. I am not at liberty to talk more on that.

Please tell me more about why the problem might have occured and also about the LIB$*VM* routines

[Volker Halle]

Hari,

a typical scenario for a 'virtual memory leak' like this would be, if the FTP server allocates some virtual memory (via malloc or an explicit LIB$GET_VM call) and 'forgets' to deallocate this data after finishing the new connection or starting the child process to handle the new connection. Or it deallocates the memory, but not the whole chunk but only a subset of the allocated memory.

You need to look at the code to determine, where this may happen. If you look at the data at the end of P0 space, you may be able to guess from the contents of memory, which data structures are being allocated and check the code for these types of data structures.

All of this requires familiarity with the code itself !

You could use LIB$SHOW_VM calls to collect and display statistics about the no. of bytes allocated/freed etc. Look at the OpenVMS documentation (HP OpenVMS RTL Library (LIB$) Manual) on how to use the LIB$*VM* calls.

Volker.

[Hoff]

Consider open-sourcing the particular ftp source code involved here, and we'll look at and fix it.

lib$get_vm and malloc() are only one of many ways that an application can leak memory.

[GuentherF]

Hi Hari!

Since you own that code, put a wrapper around all the alloc and dealloc calls and you'll see where the deallocs are missing.

/Guenther

[John Gillings · Accepted solution]

Hari,

> Can anyone help me out with the deleting
> the virtual memory once used?

It's unlikely to be applicable in your case, but for completeness, here's the theory:

Virtual memory is created by expanding a region with $EXPREG. For 32 bit user mode applications, and the default heap, the only region you're dealing with is P0. LIB$GET_VM will $EXPREG on your behalf as required.

To delete virtual memory you use $DELTVA, BUT, you need to be certain that the pages being deleted won't be touched, and, if the pages aren't at the end of the region, they won't be available for $EXPREG to reuse, so there's no benefit. All you do is lose memory.

Memory created by LIB$GET_VM for the heap is never deleted because it's too difficult to manage allowing the heap to expand and contract. However, memory that is freed with LIB$FREE_VM can be reallocated.

Even assuming your code uses GET_VM and FREE_VM correctly, it is still possible to get VM leakage with pathological patterns of allocations and deallocations. The simplest way to illustrate this is with the default allocation algorithm (first fit):

allocate small
allocate big
deallocate small
deallocate big
(repeat many times)

Since the "big" object is at the front of the list, it's used to satisfy the next small request, leaving a free block too small to satisfy the next big request. Although the code is correct, VM will expand, with a free list alternating a small object and big-small object until it hits a limit. This type of heap fragmentation is called "checkerboarding"

If possibly, reordering the sequence to:

allocate small
allocate big
deallocate big
deallocate small

will fix the problem. Alternatively, seggregate big and small objects into different heaps, or manage your own "lookaside" free lists of standard sized objects. Another possibility is to use the slower "best fit" algorithm when allocating memory.

All this is academic if you don't have direct control over trhe code in question.

[Hari Shankar S]

Hoff,

I am afraid that I dont have the authority to open source the code.

[Volker Halle]

Hari,

as this seems to be an easily reproducable problem with TCPIP V5.6 ECO 3 and you have access to the sources, did you also go the route to try with ECO 2 and if the problem is not reproducable with that version, look for changes in the source code ?

Volker.