topic Re: Memory leak in Operating System - OpenVMS

Memory leak

Wim Van den Wyngaert — Tue, 14 Oct 2008 14:39:36 GMT

We have a program running on a 2 CPU GS160.
It's not changed since 2003 and is doing TCP (Reuters Sink) and DECnet (other VMS nodes) communications.

Since a few days the load is heavy due to the heavy activity on the stock exchanges. Under this heavy load, it starts consuming a lot more memory and after a few minutes it goes out of memory (normally +- 600 MB for the whole process tree, now going to 1500 MB).
The process tree is restarted every day and each time the problem comes back.

I included the PSDC samping report taken when the process goes from 750 MB to 1500 MB. The process is named FOE_RGS_SRV (and consumes the cpu together with FOE_POS_SRV).

Is anyone able to make something of it ?
(VMS 7.3, TCP 5.3 eco 2, decnet 7.3 eco 3)

Wim

Re: Memory leak

labadie_1 — Tue, 14 Oct 2008 15:03:38 GMT

Before you find the leak, you can add in your program a regular (every 2 hours ?) call to $purgws. Be prepared for more pagefile utilisation.

Of course this is a temporary workaround.

Re: Memory leak

Andy Bustamante — Tue, 14 Oct 2008 15:23:14 GMT

What's the QBB/CPU/memory layout? Do you use global pages in the process tree?

Starting in 7.3, global pages are mapped allocating pages across QBBs. This led to an application running multiple processes against global pages creating heavily fragmented global allocations during the day. Eventually, performance deteriorated and the application hung requiring a restart.

HP recommended application changes to way global pages were allocated. The end solution adopted was moving to GS-1280s. Updating the hardware layout may mitigate the issue in your case.

Andy

Re: Memory leak

Hoff — Tue, 14 Oct 2008 15:51:06 GMT

An application that's been running perfectly since 2003 is not an application that is immune to latent bugs.

Application (and system) load is one of the classic and salient triggers for exposing bugs and race conditions and leaks.

Identify what memory resource(s) are leaking, and work from there. This can involve digging around in the process data structures, and in the process address range. (If restarting the application cures these, then it's usually a process private leak. That doesn't, however, mean it's your code or HP code.)

Your attachment shows PC samplings, and those are not on point for a memory leak; there's not a correlation between cold or hot PC ranges and memory use. Yes, you do have to access the range to get the leak, but the range of code doesn't have to be hot.

Small leaks in hot code and big leaks in cold code can ruin your uptime statistics. And nothing says there is just one leak. Though big leaks in hot code are usually pretty obvious.

There have been various leaks in OpenVMS and TCP/IP Services and other products remediated over the years; if you're not current...

Re: Memory leak

Hein van den Heuvel — Wed, 15 Oct 2008 00:33:17 GMT

Please explain to your management that it is time to 'pay the piper'.

This is what you get for sticking with an old OS version on an old platform.

Good luck! (you'l need some :-)

Regards
Hein
(geen punten nodig voor dit advies :-(

Re: Memory leak

Wim Van den Wyngaert — Wed, 15 Oct 2008 05:30:50 GMT

Then I'll ask a different question.

52$ of the time is taken by SYSTEM_PRIMITIVES_MIN / MMG_STD$ALLOC_SYSTE

What is this ? It can't be that the program is in "alloc" for 52% of the time, I hope.

The problem is that it will be difficult to get a correction for a few months that are left. So, if it could be solved by a reboot I would be very happy.

Wim

Re: Memory leak

Willem Grooters — Wed, 15 Oct 2008 11:47:22 GMT

A reboot won't help much if the problem is within the software. It will keep consuming memory until you boot it again unless you find the hole and fix it.
Memory fragmentation might be one cause; but given your description, I would look for the code executed in those new minutes and concentrate in the code tree that is executed. also check for asymnconous code that may get triggered and allocates chunks of memeory.

Re: Memory leak

kari salminen — Wed, 15 Oct 2008 12:18:10 GMT

If your application is using LIB$GET_VM / LIB$FREE_VM, a memory leak bug in LIBRTL.EXE
may be the cause, it got fixed in VMS 7.3-2 LIBRTL ECO 2, I don't know if there is an ECO for VMS 7.3

A LIB$GET_VM may expand the process region when there are sufficient contiguous bytes in the memory zone to satisfy the request.

ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.3-2/VMS732_LIBRTL-V0200.txt

You may find the PQUOTA tool useful for analyzing memory leaks, latest version V2.0 runs on VAX, Alpha and Itanium,

http://vms.process.com/scripts/fileserv/fileserv.com?PQUOTA

Re: Memory leak

Hoff — Wed, 15 Oct 2008 14:13:23 GMT

There are two choices: either find and fix the leak, or use the $purgws/restart sequence and a large pagefile and/or a reboot when the stock market gets busy.

It may be cheaper to throw some disk storage (pagefile) and some quota and some memory at this case; to buy enough headroom for the daily restart.

If following the former path, the intrepid explorer needs to first find what structures are being leaked. This can be through examination or through instrumentation.

One obvious variation here is to speed up the migration off of OpenVMS.

Re: Memory leak

Robert Gezelter — Wed, 15 Oct 2008 14:33:10 GMT

Wim,

As Hoff mentioned, "long running" does not necessarily imply "no latent problems".

Without knowing how the application is structured, it is difficult to guess. Having done similar applications in the past, I can see many situations where such a thing could happen.

In this situation, I would suggest both palliative measures and a longer-term fix. For palliative measures, a larger page file is definitely a start, and an automated restart at a quiet time.

For a longer term fix, it might be useful to get one or more sets of process dumps to identify the nature of the "memory leak". It is not unlikely that it is a small, discrete fix.

- Bob Gezelter, http://www.rlgsc.com

Re: Memory leak

Richard W Hunt — Wed, 15 Oct 2008 16:19:56 GMT

When I last saw a problem like this on my system (and referencing the Allocate System Memory) function, it was caused by something growing its working set because of a string processing issue.

By any chance are those two programs written in a language for which the string paradigm is that a string is actually a descriptor that points somewhere in the program heap? (As opposed to FORTRAN-like, where strings are pre-allocated and fixed length.)

The problem was "thrashing" the program's scratchpad. You would probably also see a sudden increase in paging/swapping activity just before this problem reared its ugly head.

Re: Memory leak

Hoff — Wed, 15 Oct 2008 16:31:17 GMT

Richard is referring to modifications to a dynamic string descriptor; erroneously writing to the descriptor as if it were a static descriptor.

That sequence can certainly result in memory loss, but it's typically a sort of more continuous leak. That class of bug is not (usually) a load-activated bug, though that class of bug could easily be secondary to another bug.

I posted the general code review list over in http://h71000.www7.hp.com/wizard/wiz_1661.html and some other threads referenced there.

Do ramp up on the new platform, too -- whatever that might be. Life's too short to stay grumpy, and I'm inferring you've got a case of the grumpies today. :-)

Re: Memory leak

John McL — Thu, 16 Oct 2008 02:06:06 GMT

You've said nothing about the architecture of the application but that might be the cause of the problem.

Is it AST driven with a lot of network I/O and using a ring buffer that might be stressed by a heavy load? It's feasible that an initial buffer allocation is insufficient and further allocations are made, but the "expand" flag is not being cleared after an expansion is made. Perhaps previous input rates have never been enough to trigger this action (i.e. the buffer contents are processed fast enough so that the buffer never needed expansion) and the bug has not previously been exposed. If you are really lucky you'll have monitoring tools that tell you how many buffers are allocated and used.

Of course I might be barking up the wrong tree because I'm only guessing at how an application that processes Borse data might be structured.

Re: Memory leak

labadie_1 — Thu, 16 Oct 2008 06:05:40 GMT

A tool like Callmon may help you

http://mvb.saic.com/freeware/vmslt96a/lelegard/callmon_.ada

Re: Memory leak

Volker Halle — Thu, 16 Oct 2008 11:55:57 GMT

Wim,

is the application using PTHREADS ? Use ANAL/SYS and SDA> SET PROC FOE_RGS_SRV
Then try SDA> PTHREAD VM

If there is no error message, because the process is not threaded, is there any lookaside list with a substantial amount of packets ?

Volker.

Re: Memory leak

John McL — Thu, 16 Oct 2008 23:37:59 GMT

You might also find the Tech Journal No. 7 article "Faking it with Open VMS Shareable Images" by John Gillings useful especially if you think the problem might be an errant LIB$GET_VM call.

The article is online at http://h71000.www7.hp.com/openvms/journal/v7/faking_it_with_openvms_shareable_images.html and the code seems to be downloadable.

Re: Memory leak

Wim Van den Wyngaert — Mon, 20 Oct 2008 07:33:06 GMT

All I know it's written in C.

Volker : pthread is unknown command in 7.3. Show proc/thr says "1 thread". May be retry when NYSE opens.

Nice article of John G. We used something simular on HP3000 to fool the verification of the license date of a certain product.

Labadie : ada ...

The applidcation boys will not look at the code and we now restart the process when it gets mad. BTW : when it gets mad it only takes a few minutes before the 1.5 GB is taken. Increasing it would only delay the problem a little.

Wim

Re: Memory leak

Volker Halle — Mon, 20 Oct 2008 07:51:13 GMT

Wim,

the SYS$SHARE:PTHREAD$SDA.EXE extension should be available since OpenVMS V7.2-1 ...

You can also check with SDA> SHOW PROC/CHAN, if PTHREAD$RTL is an activated image for this process.

Volker.

Re: Memory leak

Robert Gezelter — Mon, 20 Oct 2008 08:13:11 GMT

Wim,

"The applidcation boys will not look at the code and we now restart the process when it gets mad. BTW : when it gets mad it only takes a few minutes before the 1.5 GB is taken. Increasing it would only delay the problem a little."

I have had a few of those at clients over the years. Regrettably, the solution has often been to identify the failing code independently, and propose a fix. Not the best way to work, but it can be the most effective way to deal with organizational politics.

- Bob Gezelter, http://www.rlgsc.com

Re: Memory leak

Wim Van den Wyngaert — Mon, 20 Oct 2008 10:47:12 GMT

Bob,

Did that once (network connection was not closed). But after several years it's still not in production because of testing requirements.

In November we will have DRP tests and then I can reboot the node. May be it gets solved that way.

Volker,

I shortened the command to PTHR but this didn't work. When I typed it in full it worked. Sorry. Same conclusion : 1 thread.

Wim