topic Re: PGFLQUOTA leak with FTP in Operating System - OpenVMS

PGFLQUOTA leak with FTP

Hari Shankar S — Tue, 03 Mar 2009 13:05:23 GMT

It has been found that the PGFLQUOTA of FTP never decreases. When new FTP connections are established the PGFLQUOTA is consumed but when the connections are closed the PGFLQUOTA is not decreased. As times passes, more and more writeable pages are gradually
added to the end of p0 space. If the system is left on long enough, it could eventually run out of quota.

Does any one have any idea why this problem occurs?

Re: PGFLQUOTA leak with FTP

Hoff — Tue, 03 Mar 2009 13:48:39 GMT

Looks to be a bug in the code. A leak. There are various reasons why this can arise within an application, but there's little that can be done to resolve this case without access to the source code. (Eons ago, there was a bug within sys$creprc that caused a four-page leak, for instance.)

This reply assumes that this is the TCP/IP Services product (there are several IP stacks around) and the current version and current ECO, and that this arises on a recent or current version of OpenVMS Alpha or OpenVMS I64, and that this is involving the ftp server and not the ftp client.

Assuming this is the current version and ECO of the TCP/IP Services product, log a problem report with HP with a reproducer.

In the short term, you could periodically restart the FTP server; that should clear this up. (If you have the source code to ftp available, that's another discussion entirely.)

Re: PGFLQUOTA leak with FTP

Wim Van den Wyngaert — Tue, 03 Mar 2009 14:17:29 GMT

Don't get it. FTP client and server are both alive only during a session. Do you mean that you keep the session alive for a long time ?

Wim

Re: PGFLQUOTA leak with FTP

Hari Shankar S — Tue, 03 Mar 2009 14:38:12 GMT

The FTP server is an OpenVMS system and the client may be any other. The OVMS system is kept on without restarting the system or the service. The clients may come and go as they like. Even though the FTP connection is closed, the PGFLQUOTA is not decreased.

Re: PGFLQUOTA leak with FTP

Hari Shankar S — Tue, 03 Mar 2009 14:39:23 GMT

Hoff,
Can you please explain to me what all could be reasons for the memory leak?

Re: PGFLQUOTA leak with FTP

Volker Halle — Tue, 03 Mar 2009 16:51:49 GMT

Hari,

which process are your referring to ? TCPIP$FTP_1 ? Which version and architecture of OpenVMS and which version of TCPIP ?

$ UCX SHOW VERS ! should give info

For each FTP client connection to the OpenVMS system, a new network process will be created named TCPIP$FTPCnnnnn. This process terminates, if the FTP connection is terminated by the client. Only the FTP Server TCPIP$FTP_1 process stays around.

If there would be a 'memory' leak affecting PGFLQUOTA, this could only exist in the FTP server process.

What symptoms of this leak are you seeing ?
Are you using SHOW PROC/QUOTA/ID=ypid-of-FTP-server> ?

The Paging file quota: value shown is the remaining page file quota for this process. If some more is allocated for each FTP connection and not de-allocated or even re-used, then you would see pagefile quota decreasing over time.

Volker.

Re: PGFLQUOTA leak with FTP

Hoff — Tue, 03 Mar 2009 20:00:26 GMT

>Can you please explain to me what all could be reasons for the memory leak?

Resolving a leak requires source code access, or a whole lot more time and effort in reverse engineering this than this particular bug is probably worth. Do you have source code access to the particular ftp daemon here? (If you don't have that source code, there's not that much that can be done here short of a very substantial investment in software engineering. Other than forwarding this problem report over to HP support with a reproducer.)

As for the trigger, this isn't a small topic area. There are a vast number of triggers here. I have presented day-long courses on advanced programming tips and techniques and tools specifically for OpenVMS tailored for various sites. And that time only begins to cover some of what's involved here. And which introduces how to try to avoid these leaks (and a close cousin to leaks, the heap corruption), and how to more quickly find them when they arise. For an overview of some of what has been in the training in this specific topic area, see http://64.223.189.234/node/401 and some of the articles referenced there.

Most any dynamically-allocated data structure can be lost during processing, and there can be fragmentation of the heap that leads to increased virtual memory use. The sys$creprc flaw mentioned earlier leaked pagefile; the system service was not releasing all of what it had consumed. There are many other operations where the application must perform some sort of tracking and release; neither OpenVMS nor C do not offer any form of garbage collection, though there are ways to incorporate some techniques that avoid the need to track memory.

Re: PGFLQUOTA leak with FTP

John Gillings — Tue, 03 Mar 2009 20:55:28 GMT

Hari,

In general the PAGFILCNT (=remaining PGFLQUOTA) of a process does not increase. Although it's possible to delete virtual memory, it tends to be rarely done, because there's very little benefit, and the conditions under which you can do it are fairly strict and difficult to achieve. For most processes VIRTPEAK is monotonic increasing and PAGFILCNT is therefore monotonic decreasing.

A long lived process may continue to grow even if it doesn't have a leak. It just means its work load is gradually expanding. Hopefully the rate of growth diminishes over time.

Monitor the process long term and look at the shape of the growth curve. If it's linear, you probably are dealing with a leak (but if you don't have access to the source code, there's little you can do about it). If the growth is asymptotic, there's probably natural and benign.

>If the system is left on long enough,
>it could eventually run out of quota.

Possibly true, but if it hasn't happened yet, and the projected point at which it will happen is a long way into the future, it's not something you should be too concerned about.

Re: PGFLQUOTA leak with FTP

Hari Shankar S — Wed, 04 Mar 2009 05:14:15 GMT

Volker,

>which process are your referring to ? TCPIP$FTP_1 ?
Yes, I am referring to the TCPIP$FTP_1 process.

>Which version and architecture of OpenVMS and which version of TCPIP ?
OpenVMS is Alpha V8.3-1h1 and TCP/IP is V5.6 Eco3 with the latest patches installed.

>What symptoms of this leak are you seeing ?
If there is a lot of incoming FTP with my system , %PGFLQUOTA_CNT can grow, I mean, the quota is partially consumed. When all those FTPs (open channels) with my system finish, the channels are released , the process TCPIP$FTPC000XXX DIES ,AND ONLY TWO CHANNELS BG'S, OPENED AT TCPIP$FTP_1. BUT PAGFILCNT of TCPIP$FTP_1 is not decreased

>Are you using SHOW PROC/QUOTA/ID=ypid-of-FTP-server> ?
The information about the leak is gotten from SDA formatting the JIB (Job Information Block) of the TCPIP$FTP_1 process and displaying with F$GETJPI and PAGFILCNT

Re: PGFLQUOTA leak with FTP

Hari Shankar S — Wed, 04 Mar 2009 05:22:16 GMT

John,

>Although it's possible to delete virtual memory, it tends to be rarely done
How is this done?

>Monitor the process long term and look at the shape of the growth curve.
I can see that each 48 or 54 minutes the consumption of PGFLQUOTA grows. But there is not an explanation for that , becase the number of FTP session DON'T GROW.

>but if it hasn't happened yet, and the projected point at which it will happen is a >long way into the future
It has happened.

Re: PGFLQUOTA leak with FTP

Hari Shankar S — Wed, 04 Mar 2009 05:32:42 GMT

Can anyone help me out with the deleting the virtual memory once used?

What is the difference between LIB$FREE_VM and LIB$DELETE_VM_ZONE ?

Is it true that "LIB$FREE_VM never gives memory back to the system when it is freed. It keeps it for future use." ?

If it is, what is the way out of this?

Re: PGFLQUOTA leak with FTP

Volker Halle — Wed, 04 Mar 2009 07:00:29 GMT

Hari,

there is NOTHING you can do about TCPIP$FTP_1, if this process would actually have a memory leak and therefore would be consuming it's pagefile quota over time.

If you can prove that this is true and you can prove - or at least demonstrate - this case, document your findings and raise a call with HP. Only TCPIP engineering could do anything about this, if there is a real problem.

Don't waste your time thinking about LIB$*VM* routines, as they all require access to the source code of the application, i.e. TCPIP$FTP_SERVER.EXE.

As a workaround, you could restart the TCPIP FTP service, if the remaining pagefile quota approaches ZERO. Use @SYS$STARTUP:TCPIP$FTP_SHUTDOWN.COM and TCPIP$FTP_STARTUP.COM. Note that this will temporarily disrupt FTP traffic.

You could also raise the SYSGEN parameter PQL_MPGFLQUOTA, if you don't want to restart FTP every once in a while.

I've tested a simple FTP connection to our OpenVMS Alpha V8.2 TCPIP V5.5 system and the remaining pagefile quota of TCPIP$FTP_1 was the SAME before and after the connection.

You should be able to prove your point by:

$ SHOW PROC/QUOTA/ID=
$ DIR/FTP localhost::login.com
$ SHOW PROC/QUOTA/ID=

If you do this 100 times and if the decrease in pagefile quota remains constant for each FTP connection, you've proven your point. You could then even predict, when TCPIP$FTP_1 is going to fail and restart (see above) the FTP service early enough.

Volker.

Re: PGFLQUOTA leak with FTP

Wim Van den Wyngaert — Wed, 04 Mar 2009 07:16:44 GMT

The leak doesn't seem present on 5.3 eco 2.

Also check sys$sysddevice:[tcpip$ftp]tcpip%ftp_run.log for messaages. And opcom.

Wim

Re: PGFLQUOTA leak with FTP

Hari Shankar S — Wed, 04 Mar 2009 07:25:21 GMT

Volker

Let us assume that I have access to the codes. No questions on that please. I am not at liberty to talk more on that.

Please tell me more about why the problem might have occured and also about the LIB$*VM* routines

Re: PGFLQUOTA leak with FTP

Volker Halle — Wed, 04 Mar 2009 16:49:57 GMT

Hari,

a typical scenario for a 'virtual memory leak' like this would be, if the FTP server allocates some virtual memory (via malloc or an explicit LIB$GET_VM call) and 'forgets' to deallocate this data after finishing the new connection or starting the child process to handle the new connection. Or it deallocates the memory, but not the whole chunk but only a subset of the allocated memory.

You need to look at the code to determine, where this may happen. If you look at the data at the end of P0 space, you may be able to guess from the contents of memory, which data structures are being allocated and check the code for these types of data structures.

All of this requires familiarity with the code itself !

You could use LIB$SHOW_VM calls to collect and display statistics about the no. of bytes allocated/freed etc. Look at the OpenVMS documentation (HP OpenVMS RTL Library (LIB$) Manual) on how to use the LIB$*VM* calls.

Volker.

Re: PGFLQUOTA leak with FTP

Hoff — Wed, 04 Mar 2009 18:16:25 GMT

Consider open-sourcing the particular ftp source code involved here, and we'll look at and fix it.

lib$get_vm and malloc() are only one of many ways that an application can leak memory.

Re: PGFLQUOTA leak with FTP

GuentherF — Wed, 04 Mar 2009 18:24:07 GMT

Hi Hari!

Since you own that code, put a wrapper around all the alloc and dealloc calls and you'll see where the deallocs are missing.

/Guenther

Re: PGFLQUOTA leak with FTP

John Gillings — Wed, 04 Mar 2009 22:36:55 GMT

Hari,

> Can anyone help me out with the deleting
> the virtual memory once used?

It's unlikely to be applicable in your case, but for completeness, here's the theory:

Virtual memory is created by expanding a region with $EXPREG. For 32 bit user mode applications, and the default heap, the only region you're dealing with is P0. LIB$GET_VM will $EXPREG on your behalf as required.

To delete virtual memory you use $DELTVA, BUT, you need to be certain that the pages being deleted won't be touched, and, if the pages aren't at the end of the region, they won't be available for $EXPREG to reuse, so there's no benefit. All you do is lose memory.

Memory created by LIB$GET_VM for the heap is never deleted because it's too difficult to manage allowing the heap to expand and contract. However, memory that is freed with LIB$FREE_VM can be reallocated.

Even assuming your code uses GET_VM and FREE_VM correctly, it is still possible to get VM leakage with pathological patterns of allocations and deallocations. The simplest way to illustrate this is with the default allocation algorithm (first fit):

allocate small
allocate big
deallocate small
deallocate big
(repeat many times)

Since the "big" object is at the front of the list, it's used to satisfy the next small request, leaving a free block too small to satisfy the next big request. Although the code is correct, VM will expand, with a free list alternating a small object and big-small object until it hits a limit. This type of heap fragmentation is called "checkerboarding"

If possibly, reordering the sequence to:

allocate small
allocate big
deallocate big
deallocate small

will fix the problem. Alternatively, seggregate big and small objects into different heaps, or manage your own "lookaside" free lists of standard sized objects. Another possibility is to use the slower "best fit" algorithm when allocating memory.

All this is academic if you don't have direct control over trhe code in question.

Re: PGFLQUOTA leak with FTP

Hari Shankar S — Thu, 05 Mar 2009 07:42:37 GMT

Hoff,

I am afraid that I dont have the authority to open source the code.

Re: PGFLQUOTA leak with FTP

Volker Halle — Thu, 05 Mar 2009 07:43:39 GMT

Hari,

as this seems to be an easily reproducable problem with TCPIP V5.6 ECO 3 and you have access to the sources, did you also go the route to try with ECO 2 and if the problem is not reproducable with that version, look for changes in the source code ?

Volker.