- Community Home
- >
- Servers and Operating Systems
- >
- Operating Systems
- >
- Operating System - OpenVMS
- >
- Memory leak
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Forums
Discussions
Discussions
Discussions
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2008 07:39 AM
10-14-2008 07:39 AM
Memory leak
It's not changed since 2003 and is doing TCP (Reuters Sink) and DECnet (other VMS nodes) communications.
Since a few days the load is heavy due to the heavy activity on the stock exchanges. Under this heavy load, it starts consuming a lot more memory and after a few minutes it goes out of memory (normally +- 600 MB for the whole process tree, now going to 1500 MB).
The process tree is restarted every day and each time the problem comes back.
I included the PSDC samping report taken when the process goes from 750 MB to 1500 MB. The process is named FOE_RGS_SRV (and consumes the cpu together with FOE_POS_SRV).
Is anyone able to make something of it ?
(VMS 7.3, TCP 5.3 eco 2, decnet 7.3 eco 3)
Wim
- Tags:
- leak
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2008 08:03 AM
10-14-2008 08:03 AM
Re: Memory leak
Of course this is a temporary workaround.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2008 08:23 AM
10-14-2008 08:23 AM
Re: Memory leak
Starting in 7.3, global pages are mapped allocating pages across QBBs. This led to an application running multiple processes against global pages creating heavily fragmented global allocations during the day. Eventually, performance deteriorated and the application hung requiring a restart.
HP recommended application changes to way global pages were allocated. The end solution adopted was moving to GS-1280s. Updating the hardware layout may mitigate the issue in your case.
Andy
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2008 08:51 AM
10-14-2008 08:51 AM
Re: Memory leak
Application (and system) load is one of the classic and salient triggers for exposing bugs and race conditions and leaks.
Identify what memory resource(s) are leaking, and work from there. This can involve digging around in the process data structures, and in the process address range. (If restarting the application cures these, then it's usually a process private leak. That doesn't, however, mean it's your code or HP code.)
Your attachment shows PC samplings, and those are not on point for a memory leak; there's not a correlation between cold or hot PC ranges and memory use. Yes, you do have to access the range to get the leak, but the range of code doesn't have to be hot.
Small leaks in hot code and big leaks in cold code can ruin your uptime statistics. And nothing says there is just one leak. Though big leaks in hot code are usually pretty obvious.
There have been various leaks in OpenVMS and TCP/IP Services and other products remediated over the years; if you're not current...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2008 05:33 PM
10-14-2008 05:33 PM
Re: Memory leak
This is what you get for sticking with an old OS version on an old platform.
Good luck! (you'l need some :-)
Regards
Hein
(geen punten nodig voor dit advies :-(
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-14-2008 10:30 PM
10-14-2008 10:30 PM
Re: Memory leak
52$ of the time is taken by SYSTEM_PRIMITIVES_MIN / MMG_STD$ALLOC_SYSTE
What is this ? It can't be that the program is in "alloc" for 52% of the time, I hope.
The problem is that it will be difficult to get a correction for a few months that are left. So, if it could be solved by a reboot I would be very happy.
Wim
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 04:47 AM
10-15-2008 04:47 AM
Re: Memory leak
Memory fragmentation might be one cause; but given your description, I would look for the code executed in those new minutes and concentrate in the code tree that is executed. also check for asymnconous code that may get triggered and allocates chunks of memeory.
OpenVMS Developer & System Manager
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 05:18 AM
10-15-2008 05:18 AM
Re: Memory leak
If your application is using LIB$GET_VM / LIB$FREE_VM, a memory leak bug in LIBRTL.EXE
may be the cause, it got fixed in VMS 7.3-2 LIBRTL ECO 2, I don't know if there is an ECO for VMS 7.3
A LIB$GET_VM may expand the process region when there are sufficient contiguous bytes in the memory zone to satisfy the request.
ftp://ftp.itrc.hp.com/openvms_patches/alpha/V7.3-2/VMS732_LIBRTL-V0200.txt
You may find the PQUOTA tool useful for analyzing memory leaks, latest version V2.0 runs on VAX, Alpha and Itanium,
http://vms.process.com/scripts/fileserv/fileserv.com?PQUOTA
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 07:13 AM
10-15-2008 07:13 AM
Re: Memory leak
It may be cheaper to throw some disk storage (pagefile) and some quota and some memory at this case; to buy enough headroom for the daily restart.
If following the former path, the intrepid explorer needs to first find what structures are being leaked. This can be through examination or through instrumentation.
One obvious variation here is to speed up the migration off of OpenVMS.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 07:33 AM
10-15-2008 07:33 AM
Re: Memory leak
As Hoff mentioned, "long running" does not necessarily imply "no latent problems".
Without knowing how the application is structured, it is difficult to guess. Having done similar applications in the past, I can see many situations where such a thing could happen.
In this situation, I would suggest both palliative measures and a longer-term fix. For palliative measures, a larger page file is definitely a start, and an automated restart at a quiet time.
For a longer term fix, it might be useful to get one or more sets of process dumps to identify the nature of the "memory leak". It is not unlikely that it is a small, discrete fix.
- Bob Gezelter, http://www.rlgsc.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 09:19 AM
10-15-2008 09:19 AM
Re: Memory leak
By any chance are those two programs written in a language for which the string paradigm is that a string is actually a descriptor that points somewhere in the program heap? (As opposed to FORTRAN-like, where strings are pre-allocated and fixed length.)
The problem was "thrashing" the program's scratchpad. You would probably also see a sudden increase in paging/swapping activity just before this problem reared its ugly head.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 09:31 AM
10-15-2008 09:31 AM
Re: Memory leak
That sequence can certainly result in memory loss, but it's typically a sort of more continuous leak. That class of bug is not (usually) a load-activated bug, though that class of bug could easily be secondary to another bug.
I posted the general code review list over in http://h71000.www7.hp.com/wizard/wiz_1661.html and some other threads referenced there.
Do ramp up on the new platform, too -- whatever that might be. Life's too short to stay grumpy, and I'm inferring you've got a case of the grumpies today. :-)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 07:06 PM
10-15-2008 07:06 PM
Re: Memory leak
Is it AST driven with a lot of network I/O and using a ring buffer that might be stressed by a heavy load? It's feasible that an initial buffer allocation is insufficient and further allocations are made, but the "expand" flag is not being cleared after an expansion is made. Perhaps previous input rates have never been enough to trigger this action (i.e. the buffer contents are processed fast enough so that the buffer never needed expansion) and the bug has not previously been exposed. If you are really lucky you'll have monitoring tools that tell you how many buffers are allocated and used.
Of course I might be barking up the wrong tree because I'm only guessing at how an application that processes Borse data might be structured.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-15-2008 11:05 PM
10-15-2008 11:05 PM
Re: Memory leak
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2008 04:55 AM
10-16-2008 04:55 AM
Re: Memory leak
is the application using PTHREADS ? Use ANAL/SYS and SDA> SET PROC FOE_RGS_SRV
Then try SDA> PTHREAD VM
If there is no error message, because the process is not threaded, is there any lookaside list with a substantial amount of packets ?
Volker.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-16-2008 04:37 PM
10-16-2008 04:37 PM
Re: Memory leak
The article is online at http://h71000.www7.hp.com/openvms/journal/v7/faking_it_with_openvms_shareable_images.html and the code seems to be downloadable.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 12:33 AM
10-20-2008 12:33 AM
Re: Memory leak
Volker : pthread is unknown command in 7.3. Show proc/thr says "1 thread". May be retry when NYSE opens.
Nice article of John G. We used something simular on HP3000 to fool the verification of the license date of a certain product.
Labadie : ada ...
The applidcation boys will not look at the code and we now restart the process when it gets mad. BTW : when it gets mad it only takes a few minutes before the 1.5 GB is taken. Increasing it would only delay the problem a little.
Wim
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 12:51 AM
10-20-2008 12:51 AM
Re: Memory leak
the SYS$SHARE:PTHREAD$SDA.EXE extension should be available since OpenVMS V7.2-1 ...
You can also check with SDA> SHOW PROC/CHAN, if PTHREAD$RTL is an activated image for this process.
Volker.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 01:13 AM
10-20-2008 01:13 AM
Re: Memory leak
"The applidcation boys will not look at the code and we now restart the process when it gets mad. BTW : when it gets mad it only takes a few minutes before the 1.5 GB is taken. Increasing it would only delay the problem a little."
I have had a few of those at clients over the years. Regrettably, the solution has often been to identify the failing code independently, and propose a fix. Not the best way to work, but it can be the most effective way to deal with organizational politics.
- Bob Gezelter, http://www.rlgsc.com
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 03:47 AM
10-20-2008 03:47 AM
Re: Memory leak
Did that once (network connection was not closed). But after several years it's still not in production because of testing requirements.
In November we will have DRP tests and then I can reboot the node. May be it gets solved that way.
Volker,
I shortened the command to PTHR but this didn't work. When I typed it in full it worked. Sorry. Same conclusion : 1 thread.
Wim
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 04:22 AM
10-20-2008 04:22 AM
Re: Memory leak
some things in this story do not add up in MY logic. (But then, in yours neither, I guess).
So, this app is (a. o.) processing NYSE transactions for ING bank, right?
Well, all is well then, I believe.
Only yesterday ING had to get an emergency state loan of just EUR 10 G (about USD 13 G, ie, $13.000.000.000)
A loan at a, well, "friendly" intrest of ___ 8.5 ___ % !!! ( if non-official publications hold some truth)
HOW can your management rhyme this to __NOT__ doing everything necessary to get the software right, AND AS QUICKLY AS POSSIBLE???
Can you explain to someone in accountancy that repairing this will cost MUCH less than even ONE day of interest on that loan alone?
If _I_ were a shareholder, I would publicly declare this an unparallelled case of mismanagement, which calls for IMMEDIATE curative action.....
Just a thought though.
And yes, ING is alsothe bank that processes MY salary.
In spite everything, anyhow:
Proost.
Have one on me.
jpe
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 04:44 AM
10-20-2008 04:44 AM
Re: Memory leak
the fact, that the process only has ONE thread does not mean, that it can't be using DECthreads (pthreads).
Only if SDA> PTHREAD VM returns
Process "xxx" (PID ppp) is not threaded
then you know, that pthreads is not in use.
Volker.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 05:53 AM
10-20-2008 05:53 AM
Re: Memory leak
Then it's using pthreads. I think the program is capable of threads but not using it (just tried sda commands during heavy activity). It has a channel to pthread$rtl in show proc/chan.
Wim
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 07:02 AM
10-20-2008 07:02 AM
Re: Memory leak
then what does SDA> PTHREAD VM report ?
Any lookaside list with lots of packets ?
Volker.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2008 11:48 PM
10-20-2008 11:48 PM
Re: Memory leak
-------------------------------------------------------------------------
lookaside 0 (112 bytes; rwb, cvb, mub) 1 in use, 0 free
lookaside 1 (2120 bytes; cv-meter) 0 in use, 0 free
lookaside 2 (3184 bytes; mu-meter) 0 in use, 0 free