topic Re: debugging memory leak in a pthread program in Operating System - OpenVMS

debugging memory leak in a pthread program

Guinaudeau — Tue, 03 Jan 2006 23:12:25 GMT

hello,

we developed for customers a critical application running 24/24 on OpenVMS 7.3-x using posix threads (precisely, several processes in the app are DCE RPC servers).

For more precision, our systems are configured SYSGEN MULTITHREAD=1, and the RPC server programs are linked :

%THREADCP-I-MKT, multiple kernel threads are disabled
%THREADCP-I-UPC, upcalls are disabled

I was informed thru people of this forum (eg, thread 865731) about the new V7.3-2 feature :

$ SET PROC/DUMP=NOW/ID=

We have currently troubles with a memory leak within our application, and situation was yesterday very bad for one very active RPC server in the app.

Normal process values : pagefile usage < 250 Mbytes. Probably due to a memory leak, process pagefile usage ~ 420 Mbytes and VIRTPEAK ~ 540
Mbytes !

We decided with our customer a restart of the server process, but before doing it, to dump it using the SET PROC/DUMP=NOW command.

I holed the dump to our development site, on a very close system (same VMS patch level is probably mandatory). Copied the executable and the application shareable images into the login directory for the analyzer
to find them.

I try to look with the SDA PTHREAD extension, and i discovered something suspect :

DBG> pthread vm
lookaside 0 (32 bytes; obj-name) 585866 in use, 1 free
lookaside 1 (256 bytes; hash-bucket) 187 in use, 0 free
lookaside 2 (384 bytes; rwb, mub, cvb) 586318 in use, 0 free
lookaside 3 (4096 bytes; tsd-array) 0 in use, 0 free
lookaside 4 (4288 bytes; mu-meter) 0 in use, 0 free
lookaside 5 (4352 bytes; cv-meter) 0 in use, 0 free
lookaside 6 (8192 bytes; tcb) 0 in use, 0 free

On a very few active RPC server these are :

Process name: BNASR0 Extended PID: 20200184 Thread data: "vm"
-------------------------------------------------------------------------
lookaside 0 (32 bytes; obj-name) 2323 in use, 1 free
lookaside 1 (256 bytes; hash-bucket) 82 in use, 0 free
lookaside 2 (384 bytes; rwb, mub, cvb) 2338 in use, 0 free
lookaside 3 (4096 bytes; tsd-array) 0 in use, 0 free
lookaside 4 (4288 bytes; mu-meter) 0 in use, 0 free
lookaside 5 (4352 bytes; cv-meter) 0 in use, 0 free
lookaside 6 (8192 bytes; tcb) 0 in use, 0 free

The total size of the lookaside lists 0 and 2 is already
(384*586318 + 32*866) ~ 225 Mbytes

Where could i find the lookaside lists and examine some buffers to
possibly detect an error pattern ? PTHREAD HELP is very little help
for this. Could someone bring informations about this SDA extension ?

I have also discovered with "pthread threads" a series of threads
which sound suspect, although i should check with the developer
details (may be they are normal in a very active server sometimes).

I might try (if i suspect some threads) to examine the stack of some
of them using SET TASK, but i am not familiar with that.

Thanks for any help

Louis Guinaudeau

Re: debugging memory leak in a pthread program

Travis Craig — Wed, 04 Jan 2006 20:58:55 GMT

Louis,

I'm sorry I don't have any answers for your specific questions. We do have a multithreaded RPC server application, though, and thought I would share a couple of experiences we have had.

First, for one type of problem we had, probably a race condition with inadequate locking between threads, we also set the MULTITHREAD Sysgen parameter to 1 and that helped, so I think you are on a good track there (assuming you have a multiprocessor).

Second, we used to link the executable without the /THREADS=(MULTIPLE_KERNEL_THREADS,UPCALLS) qualifier and we saw problems with the execution of our program. I don't remember what went wrong, but I think it might have hung and the problem might have involved the fact that we were doing significant work in AST routines, too. H-P recommends against doing anything significant in an AST routine if you are using Pthreads and I heartily agree with them, but we have legacy VMS-specific code that we are not going to change at this point. In any case, when we started linking for multiple kernel threads and upcalls, that particular problem cleared up and our application has been able to run.

I'm sorry I don't have anything for you on memory leaks or further analysis of your dump. I tried the new dump feature on our RPC server app and found nothing new for you there.

--Travis Craig

Re: debugging memory leak in a pthread program

Guinaudeau — Thu, 05 Jan 2006 03:25:53 GMT

Travis,

thanks for your answer.

outside the scope of my memory leak troubles, i would appreciate to share experiences with people like you, when i understand your experience, because our VMS application (a run-time SCADA + motif based man-machine interface) was originally purely VMS "legacy" (?) software, using VMS resources (mailboxes, lock [mostly locally and lock convert => fast ENQ ops]) for the synchronisation between [mono-threaded + AST] processes. now the app is a mix between both types of processes, those pure VMS and the new coming RPC servers

if you accept to share some experiences, would be helpfull if you have them already done :

- traceback adaptation to PTHREAD process : we had traceback on the original app, cannot have a reasonable traceback in the RPC servers

- PTHREAD SDA extension : difficult to train on the job that feature, or am i missing the right documentation, or am i not enough competent on development side (i am sysadmin and should help maintenance of these codes)

i may either open a new thread for these issues, or communicate per mail directly.

anyway, thanks for the answer.

louis

Re: debugging memory leak in a pthread program

Ian Miller. — Thu, 05 Jan 2006 05:15:08 GMT

I have not seen any documentation for the PTHREAD SDA extension apart from the help text.

You should read the appendicies of the pthreads manual
http://h71000.www7.hp.com/doc/73final/6493/6101pro_031.html#vms_appendix
http://h71000.www7.hp.com/doc/73final/6493/6101pro_033.html#debugging_threads_appendix

Re: debugging memory leak in a pthread program

Galen Tackett — Thu, 05 Jan 2006 06:22:18 GMT

Ian,

> http://h71000.www7.hp.com/doc/73final/6493/6101pro_033.html#debugging_threads
_appendix

When I tried this link the web server mumbles something about "The requested document "http://h71000.www7.hp.com/doc/73final/6493/6101pro%3CBR%3E_033.html" was not found on this server."

The first link worked okay though.

Galen

Re: debugging memory leak in a pthread program

Ian Miller. — Thu, 05 Jan 2006 06:34:28 GMT

What can I say but the link works for me (Windows XP, Firefox V1.5 and IE6).

Re: debugging memory leak in a pthread program

Guinaudeau — Thu, 05 Jan 2006 18:53:46 GMT

Ian :

i know the first document, i looked some parts of it (trying to implement the traceback, see previously). but i should have a look again at it focusing on the trouble.

i knew but forgot to look at the 2nd one, will.

now now the priorities are elsewhere, so wait a minute.

also the recent information i should look at : SDA extension sources available in licensed source CD-ROM => until now we never used (thought) about such a thing, but if it helps to better use SDA PTHREAD extension, at least because one catches some structures and links in the process dump ??? i should at least examine this.

Galen :

have a look to the exact command you catch when click on the web reference. i use IE6 and i got from time to time an error like "page not found" too.

Just i look at the page URL and it was :

javascript&colon;openExternal('http://h71000.www7.hp.com/doc/73final/6493/6101pro_031.html')

i just edit manually into

http://h71000.www7.hp.com/doc/73final/6493/6101pro_031.html

and that's it.

i could probably improve my IE6 settings to avoid this, but it is not enough frequent and i didn't until now took time to find the error.

Re: debugging memory leak in a pthread program

Travis Craig — Fri, 06 Jan 2006 22:05:02 GMT

Louis,

Yes, I'll look at your reply more closely when I have a few minutes and am willing to see what useful information we can exchange if you want to start another thread for it.

--Travis

Re: debugging memory leak in a pthread program

Guinaudeau — Mon, 09 Jan 2006 06:17:28 GMT

Travis :

any help is welcome. thanks in advance. i am also taken by other tasks and can / should be patient for that process dump to be analysed further ...

another issue is the traceback in case of thread cleanup (most of our cases) or fatal error of such a process : if you have any experience (and are ready to share it) would be of interest for us. we miss this feature where we have PTHREAD processes, and that should be possible under VMS. The VMS condition frame does exist and is valuable, but is it that the frame should be evaluated in special way due to the multi-threading and the stack peculiarity ?

louis

Re: debugging memory leak in a pthread program

Galen Tackett — Tue, 10 Jan 2006 06:16:15 GMT

Louis,

HP has a debugging tool called Visual Threads that might interest you. It's available free of charge if you join the Developer & Solution Partner Program (DSPP), and there's no charge for individual membership.

I have never really tried it out beyond installing it and seeing that it appeared to operate. But it allegedly offers some pretty nice capabilities for dealing with thread-based application debugging.

You can read more about it here: http://h21007.www2.hp.com/dspp/tech/tech_TechSoftwareDetailPage_IDX/1,1703,5077,00.html

You can get to the registration page for DSPP from:
http://h21007.www2.hp.com/dspp/tech/tech_TechSoftwareDetailPage_IDX/1,1703,5062,00.html

Re: debugging memory leak in a pthread program

Guinaudeau — Tue, 10 Jan 2006 07:12:34 GMT

galen,

DSPP is probably the new brand name for CSA alias ... as a software house, we are anyway HP partners.

i will look at quoted links about this debugger. i red the name, but did not really try to go further. i thought i would have some other infos (eg, to analyse this lookaside list) thru SDA extension PTHREAD.

the memory consumption displayed from it is awfull in our opinions, something should be around this, would we find the code(s) which allocate them.

i will look also at another way for that leak, that is : should examine the map of the listing. for the VMS concern, eg, some DECC library run-time code alike DECC$GETENV implicitly allocate and dont deallocate (at least for the life of the corresponding thread). it should have limited effects. i can provide here link informations, possibly map listing, if someone has any suggestion around the link, esp and run-time libraries called we should carefully look at.

louis

Re: debugging memory leak in a pthread program

Volker Halle — Tue, 17 Jan 2006 10:33:15 GMT

Louis,

SDA extension sources available in licensed source CD-ROM

the PTHREAD$DBGSHR listings are not part of the OpenVMS source listings CD (at least not in V7.3-1).

Volker.

Re: debugging memory leak in a pthread program

Ian Miller. — Tue, 17 Jan 2006 11:45:46 GMT

On the V7.3-2 source listings CD in the [PTHREAD] directory you will find various files thd*.* which are the sources for pthread$dbgshr

This directory is not present on the V7.3-1 CD.

Re: debugging memory leak in a pthread program

Guinaudeau — Tue, 17 Jan 2006 11:53:45 GMT

volker :

do you mean : to understand which informations PTHREAD SDA extension provide, this CD-ROM would be no much help ?

practically, i got the infos of the lookaside list (see previously) and would like to understand which threads allocate these buffers (0 buffers are free), where are they located, look at their contents etc ...

would this CD-ROM gives me enough about the structures and so on to find this ? or i better would have to do this with Visual Pthread ?

yours

louis

Re: debugging memory leak in a pthread program

Volker Halle — Tue, 17 Jan 2006 12:06:49 GMT

Louis,

as Ian has indicated, the PTHREAD sources would be on the V7.3-2 (or higher) source listing CDs. With that information, it might be possible to find the lookaside lists in question and look at their contents.

It will still be a major task to go through the source listings to understand those structures and how they are being allocated/deallocated. This will all be pthreads internals...

It may be more efficient to order PTHREAD engineering support from HP than trying to buy the source CDs and do it yourself.

Volker.

Re: debugging memory leak in a pthread program

Travis Craig — Fri, 20 Jan 2006 04:47:20 GMT

Louis,

I have looked more at your reply, but have no further insight or wisdom to offer for it.

I'm still willing to consider any new questions you have and/or share some experience in a new thread when you find time to do that. I also work on SCADA systems that have evolved over the years, mostly to UNIX/Linux and Windows-based systems now, but with a few of our established users still on OpenVMS.

--Travis

Re: debugging memory leak in a pthread program

Volker Halle — Tue, 31 Jan 2006 07:36:50 GMT

Louis,

from looking at your PHTREAD$SDA data, it seems that the number of packets used from the lookaside list and the used packets seem to be closely related. The same is true for the other RPC server data you've shown.

This might imply that some piece of code seems to allocate one packet from each list for a certain call/operation and not return them.

Could you correlate this number of non-returned packets with the execution count of some of your threads or with the no. of certain operations in your application ?

If you suspect a certain routine to behave incorrectly, could you try to write a little example program, create a thread calling that routine and execute it lots of time ?

What you also can do in your dump, is look around with SDA> EXA addr;100 at various addresses (outside the address range of the activated images as shown by SHOW PROC/IMA) looking for unusual or repeating patterns. If such a large portion of the process address space is used by the same kind of structures, you might spot something.

Volker.

Re: debugging memory leak in a pthread program

Volker Halle — Sun, 05 Feb 2006 03:26:57 GMT

Louis,

here is another SDA> PTHREAD command to try:

SDA> PTHREAD show -a

will print the known object lists. From this info, we can deduct the type of packets in the lookaside 2 queue:

rwb = rwlock objects
mub = mutex objects
cvb = condition variable objects

Once we would be able to get the address of one of those queues, the PTHREAD SQUEUE command will be able to follow and format those queues.

Volker.

Re: debugging memory leak in a pthread program

Volker Halle — Sun, 05 Feb 2006 03:46:19 GMT

Louis,

SDA> PTHREAD vm -cf

will list more details about the various lookaside lists (including high water mark and trend).

Volker.

Re: debugging memory leak in a pthread program

Volker Halle — Sun, 05 Feb 2006 08:23:49 GMT

Louis,

after a little bit of SDA> SEARCH, I've finally been able to locate the data structures describing the VM lookaside lists in the PTHREAD$RTL image in memory, but BAD LUCK - they do not contain a queue of 'blocks in use'. The only queue listheads seem to be used for 'free blocks'.

The problem now is how to find those data strucutres, which are 'in use'. There are lots of them in your dump...

Let's see if using any of the other SDA> PTHREAD commands given in my 2 previous replies shed any more light on this problem.

Volker.