Review of memory leak tools on OpenVMS: Heap Analyzer, etc.

[Ben Armstrong]

I would like to know what tools people are reaching for to debug memory leaks in their own code on OpenVMS. Because it is readily available, we have used Heap Analyzer (currently on OpenVMS V7.3-2) on a few occasions, but to say we have not been impressed by its performance and ease-of-use would be the understatement of the year.

Heap Analyzer Abyssmal Performance
----------------------------------

Like this poster http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1093719 who complains about a hundredfold slowdown, we have experienced similar results, except in our case we've estimated it runs 8500 times slower.

Here's how I arrived at this figure. I am trying to debug an application that in production takes about 3.5 hours to run on a DS15. The main execution loop will run about 200,000 iterations, averaging about 0.063 seconds per iteration. When I run the heap analyzer on it (albeit on a DS10 617 Mhz test system instead of a DS15) it takes about an hour just to start the application and go through the first iteration. Subsequently, it takes 18 minutes per iteration. At this rate, the application will finish in 7 years, 17,000 times slower than our production system! If you consider a DS15 to be about twice as fast as a DS10, that would mean running the heap analyzer makes the application run 8500 times slower.

Heap Analyzer Serious Usability Issues
--------------------------------------

The interactive nature of this tool is beguiling, but it comes at a tremendous cost, especially when you factor in the performance problems. We would be much happier with a tool that would collect information about segment allocations and just dump it into a table. Points for providing a tool to examine such a dump afterwards and display charts/jump to specific lines of code in the traceback for each allocation, etc.

But overlooking this fundamental problem for the moment, here are a hatful of niggling issues that dogged our debugging effort this time around:

The memory map region of the user interface does strange things when you zoom in to the 'extremely close' view. All of a sudden I can no longer scroll to the very top of the heap to see the most recent allocations. Yet if I zoom out, the individual segments are so small I can't distinguish them from each other.

I have not yet figured out how to cut-and-paste the "Display Segment" from the "Information Window". I would like to be able to do the same from the source window too so I can communicate details to the rest of my team. This is especially crucial considering the high overhead to set up each test case (i.e. it is impractical to ask the other team members to rerun the tests).

And finally, much time was wasted when we first started using the tool following our notes from the last time we tried it. Using both a Linux workstation and a Windows workstation, each of which were running an X server, we tried to set display/create to display the debugger window in the X server. This seemed to work for a time, but at some point right-clicking on the segments to display them failed to bring up the context-sensitive menu so we could examine them. We are not sure what happened but have not been able to make this work since then. To workaround the problem, we need to operate directly on the system console in the frigid machine room (something which one team member who is not even located in the same country as the development server cannot do).

Other tool recommendations?
---------------------------

So we've about had it with this tool. It is still useful in a very limited way, but it is not at all suitable for regular use and I would only ever use it again as a last resort. What memory leak detection tool(s) do you use regularly in your own shop on OpenVMS? Has anyone used the Heap Analyzer themselves and been *happy* with it?

Thanks,
Ben

[Ben Armstrong]

Oh, and just in case you wondered, no I did not have the "sync" checkbox checked, so as I understand it, the analyzer should be running flat out, (mostly?) unconstrained by DECWindows performance.

[Hoff · Accepted solution]

I do only occasionally use the heap analyzer for my own applications, preferring to start with defensive coding practices. To wit:

http://64.223.189.234/node/401
http://64.223.189.234/node/260

Options include HA, fake VM and (most commonly) the use of centralized memory management routines and fenceposts.

In several cases, I've retrofit wholly new application memory management into unstable applications and with great success.

If you have a support contract, any product enhancement suggestions or requests should be directed to HP via the support center; postings here may or may not be seen and acted upon.

[labadie_1]

Ben

There used to be a useful article

[OpenVMS] Example C - How to Build a C File From C Heap Routine Traces

located at
http://h18000.www1.hp.com/support/asktima/operating_systems/009D5576-DEA17C80-1C0186.html

but unfortunately, it is no longer available (rants: like many useful articles).

[Ben Armstrong]

Thanks. Helpful advice and interesting articles which I will review with my team.

If I did not already consider HA to be beyond redemption, I might expend the further effort (would at least require I re-test on a supported version of the software, don't you think?) to make my complaints through proper channels. But given the time we've wasted on this to date and the pain involved in running the software *again*, I'm more inclined now to focus on implementing some of your suggestions than to throw more time at HA.

Ben

[John Gillings]

Ben,

HA isn't really an option in my environment as it requires a DECwindows session (at least that's my understanding). Although I guess I could run an X server on my desktop, it's a bit too much of a fiddle for my liking.

For debugging memory issues I've recently been using my FAKE_RTL on LIBRTL. Latest version is attached, it has added support for tracing virtual memory. I've also been working on full 64 bit support. It seems to be working, but please let me know if you find any problems.

Using a fake RTL WILL give a performance hit, but hopefully not quite in the 1800x range! You won't get real time analysis, just a (potentially very large!) text file that you can analyze afterwards.

To configure for just VM analysis, run FAKE_RTL against LIBRTL. Edit the generated FAKE_LIBRTL.MAR and change all the flags from "DoLog" to "NoLog" (or just change the definition of DoLog to 0). Now, for all the routines you're interested in set the Log and VM flags to 1.

To use:

$ DEFINE FAKE_DUMPARGS TRUE
$ DEFINE FAKE_ARGDUMP where-you-want-your-logfile
$ DEFINE REAL_LIBRTL SYS$SHARE:LIBRTL
$ DEFINE FAKE_LIBRTL your-fake-rtl

Now run the program you want to analyse.

[Jon Pinkley]

John,

Can you please add the attachement, it seems that ITRC didn't get it.

Thanks,

Jon

[John Gillings]

my first post required a retry, and I didn't notice the attachment had been dropped.

Having now retried 5 times, it seems ITRC doesn't like my attachment.

The SUBMIT takes a long time, then stops on a blank page. It's not a huge file, and just plain text, with filetype .TXT so I don't know why it's a problem.

Send me email if you want a copy. Just my name with @ in the middle and .com on the end.

[John Gillings]

one more try

[Guy Peleg]

My preferred tool is the PRF extension
in SDA. It can do PC sampling without
any changes to the code/application.

It works from SDA and can monitor the
entire system or a specific process.

Another useful tool is system service logging. It has its limitations but has
been proven very useful for me in the
past troubleshooting performance problems.

fwiw,

Guy Peleg
Maklee Engineering
http://www.maklee.com

[Jon Pinkley]

Guy,

Nice you see you are back, it's been a while since I've you here.

Was your recommendation of PRF related to memory leak detection, or were you just recommending it in response to the "Other tool recommendations?" question?

Perhaps you were suggesting its use in determining why the Heap Analyzer is slow.

Or perhaps I am just missing something that is obvious. If so, how would PRF be used as a memory leak detection tool?

Jon

[Guy Peleg]

Guy,

Nice you see you are back, it's been a while since I've you here.

>>> I have been a silent observer of this
>>> forum lately.

Was your recommendation of PRF related to memory leak detection, or were you just recommending it in response to the "Other tool recommendations?" question?

>>> Personally, I do not like the heap >>>anaylzer due to all the problems
>>> encountered by Ben. I prefer to use PRF
>>> count the number of times memory >>>allocation/deallocation rountines been
>>> visited, maybe less trivial but less
>>> overhead.

Guy Peleg
Maklee Engineering
http://www.maklee.com

[Ben Armstrong]

In the end, HA did point us in the right direction, though soon after finding a suspicious routine we switched to Nvwa's debug_new.cpp because it was many times faster:

http://nvwa.sourceforge.net/doc/debug__new_8cpp.html

Before entirely giving up on HA, here are some things that helped:

- Increase the pagefile quota of the process running HA from 300,000 to 600,000, as otherwise I would run out of memory.

- Hide/collapse all images, etc. that were uninteresting. EXPREG and LIBRTL were where all of the activity was happening. This helped with the aforementioned issue with being able to scroll to the very bottom when zoomed in.

- Make liberal use of "SET BREAK foo DO (something; GO)" in order to see what's going on without having to watch the HA session. For instance, for my timings I did a SPAWN SHOW TIME at certain breakpoints.

Thanks again to everyone who answered. We have now identified and fixed the leak. I think we're going to continue to use Nvwa, given the success we've had with it so far.

Ben

[Hoff]

Guy's PRF comment reminds me; I've also used the basic FAKE_VM technique that John Gillings described in one of the VTJs, and (in addition to centralization and fenceposts) DECset PCA tool and the PCS tracing extension.

(That the existing OpenVMS memory management libraries aren't specifically instrumented is rather unfortunate. We all seem to end up rolling our own solutions on OpenVMS. Some of the tools around - at customer sites and in HP - are pretty good, but none of them seem to have been productized, or even distributed.)

(Dig through the available SDA extensions, too. Both the documented and undocumented - there are point solutions to a number of problems buried among those.)

And do look at the LLVM/Clang static analyzer. Or at IBM Purify. Or any of the other code analysis tools that are around.