topic Re: Vitual memory increasing in Operating System - HP-UX

Vitual memory increasing

Satish Kumar Malisetti — Wed, 07 Apr 2010 07:23:17 GMT

All,

in my application i used start a process it occupies virtual memory and while processing the files with the same PID it is increasing the virtual memroy size

ex:

i have started process x occupied memory 1 GB
under that i have processed multiple files it is increasing the virtual memory 1.04 GB , i have verifed the open handles before and after processing files is same and the oracle session is same

can any body suggest me to how to know the appended memory ?

Re: Vitual memory increasing

James R. Ferguson — Wed, 07 Apr 2010 10:56:47 GMT

Hi:

First, a process that 'malloc()'s memory and then subsequently 'free()'s that memory doesn't see the freed memory returned to the operating system until the process terminates. Instead, the freed memory is returned to the process's "heap" for reuse.

This is by design to reduce the overhead of memory allocation and deallocation and to reduce memory fragmentation.

I'm not sure the 4% increase you see is significant. Memory leaks and heap fragmentation will lead to growth in a process's size.

See this thread, with particular regard to the 'mallinfo(3)' and 'wdb':

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1351858

Regards!

...JRF...

Re: Vitual memory increasing

Michael Steele_2 — Wed, 07 Apr 2010 11:33:52 GMT

Hi

To find a memory leak you need a script executed in cron every 20 minutes or so. This command is the meat of the script.

UNIX95=1 ps -ef -o vsz,pid,ppid,state,wchan,args | sort -rn | head 10 >> outfile

You are finding biggest 'vsz' consumers and by taking a report every 20 minutes you are observing any growth.

Re: Vitual memory increasing

Bill Hassell — Wed, 07 Apr 2010 12:11:45 GMT

You can't fix program x problems with HP-UX changes. The process is growing because of how it was designed. Whether that growth is correct or not requires information from the programmer. As mentioned, a 4% growth is insignificant. When the program grows to 1.5 to 2 GB, you definitely need to contact the programmer and get the code fixed (if necessary). After all, the programmer may tell you that the program is correctly designed to grow to 50 GB under some circumstances.

Re: Vitual memory increasing

Dennis Handly — Wed, 07 Apr 2010 13:25:02 GMT

>can anybody suggest me to how to know the appended memory?

As JRF suggested, you can look at my replies in that thread.
Ideally, for every file you process, you should come back to some constant state. But if your files are larger and need more heap, you would keep growing.
But if you processed that "large" file first, it should grow to that value right away. (I'm not sure if that experiment is easy to do for you?)
Except if you have heap fragmentation, then the small requests split up the big blocks so nobody can use them.

>Bill: You can't fix program x problems with HP-UX changes.

Some times you can with something like MallocNextGen. Especially with heap fragmentation, it really isn't the program's fault.

>the programmer may tell you that the program is correctly designed to grow

In this case, the program must be remembering some state for each file.

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 06:21:10 GMT

I have tried the with malloc env varibles as posted in above link

export _M_ARENA_OPTS=1:2024
export _M_SBA_OPTS=65536:50:256

but still my virtula memory is increasing after processing file 7MB per file and

i can't test it based on huge file first and small file later coz i can't expect the upstream file sizes

Any suggessions please

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 06:25:25 GMT

Bill ,

I am from the development team itself ,
as per the design virual memory should not increase , if it reaches around 8GB then component/particular process automatically go down

Re: Vitual memory increasing

Dennis Handly — Thu, 08 Apr 2010 06:49:33 GMT

>but still my virtual memory is increasing after processing file 7MB per file

Then you need to first assume you have a memory leak and use gdb to find them.

>as per the design virtual memory should not increase

Then if you don't have leaks, you have heap fragmentation. Have you tried MallocNextGen?

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 06:52:57 GMT

can i have any links how to use
mallocnextgen and gdb please

Re: Vitual memory increasing

Dennis Handly — Thu, 08 Apr 2010 07:26:58 GMT

>can I have any links how to use mallocnextgen and gdb

Did you look at all the URLs in thread JRF listed above?

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 07:45:37 GMT

Dennis,

I am from development background , i am unable to get clearly how to use these

please could you expalin with a example

Re: Vitual memory increasing

Dennis Handly — Thu, 08 Apr 2010 08:01:46 GMT

>I am unable to get clearly how to use these

Have you read the WDB document: Debugging dynamic memory usage errors using HP WDB
http://www.hp.com/go/wdb

>please could you explain with a example

Should be some examples above.

You could call a function like this, for every file you process:
#include
#include
void printMallocInfo(const char* title) {
struct mallinfo info;
info = mallinfo();
printf("%s\n",title);
printf("Memory allocation info:\n");
printf(" total space in arena = %d\n", info.arena);
#ifdef DETAILS
printf(" number of ordinary blocks = %d\n", info.ordblks);
printf(" number of small blocks = %d\n", info.smblks);
printf(" space in holding block headers = %d\n", info.hblkhd);
printf(" number of holding blocks = %d\n", info.hblks);
printf(" space in small blocks in use = %d\n", info.usmblks);
printf(" space in free small blocks = %d\n", info.fsmblks);
printf(" space in ordinary blocks in use = %d\n", info.uordblks);
printf(" space in free ordinary blocks = %d\n", info.fordblks);
printf(" keep option space penalty = %d\n", info.keepcost);
#else
printf(" space in use = %d\n",
info.usmblks + info.uordblks);
printf(" space free = %d\n",
info.fsmblks + info.fordblks);
#endif
}

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 08:56:36 GMT

Dennis,

i can't this function from my application/while file processing it it not supporting c functions or calls , it is seperate scripting language dsd

how to proceed on this ?

please help me

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 09:47:47 GMT

dennis,

(const char* title)

colud you please let me know what is this ?

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 11:53:08 GMT

Dennis,

I have set the env variables as given in thread http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1270727289258+28353475&threadId=1360331

set unwindonsignal on
set heap-check leaks on
set heap-check frame-count 16

and ran the application after i checked with gdb

it is showing no processes are running

cbh10702 $ gdb
HP gdb 6.0 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.0 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.

(gdb) info leak
There is no running process.

do i need to call this gdb from the application ?

please suggest me

Re: Vitual memory increasing

Satish Kumar Malisetti — Thu, 08 Apr 2010 12:00:05 GMT

Dennis,

When i am ran the gdb with particular pid

gdb -pid 4395

i got the messages like

Attaching to process 4395
Reading symbols from /fwdev/home/fwprod1/data/mediation/product/bin/cor...
warning: Load module /fwdev/home/fwprod1/data/mediation/product/bin/cor has been stripped.
Debugging information is not available.

warning: Load module /fwdev/home/fwprod1/data/mediation/product/lib/libfworks.sl has been stripped.
Debugging information is not available.

attached the log for the same

please suggest me to proceed on this?

Re: Vitual memory increasing

Dennis Handly — Fri, 09 Apr 2010 06:33:39 GMT

>When I ran the gdb with particular pid
>please suggest me to proceed on this?

Did you read the wdb document?
You won't be able to attach to a process instead of starting off in gdb directly:
(gdb) set heap-check on
librtc is not loaded: Either use -leaks command line option, set heap-check before starting the program, or link librtc explicitly

Re: Vitual memory increasing

Satish Kumar Malisetti — Fri, 09 Apr 2010 08:40:15 GMT

Dennis,

I have tried in different way of testing my application

invoked the scripting and allocated memory to the paricular process and deallocated but virtual memory is incresing but not decreasing or returning

ex:
cbh10702 $ dsd
DSD> getpid
14581
DSD> getppid
10481
DSD> allocate 1200000
DSD> allocated
1200000
DSD> deallocate 1200000
DSD> allocated
0

cbh10702 $ UNIX95= ps -ex -o sz=Kbytes -o vsz=Kbytes -o ruser,pid,ppid | grep 10481
651 2624 fwsup01a 14581 10481
128 548 fwsup01a 10481 10480
cbh10702 $ UNIX95= ps -ex -o sz=Kbytes -o vsz=Kbytes -o ruser,pid,ppid | grep 10481
939 3776 fwsup01a 14581 10481
128 548 fwsup01a 10481 10480
cbh10702 $ UNIX95= ps -ex -o sz=Kbytes -o vsz=Kbytes -o ruser,pid,ppid | grep 10481
939 3776 fwsup01a 14581 10481
128 548 fwsup01a 10481 10480

after deallocation also its is not decreased the size of virtual memory

is this memory leak ?

please suggest

Re: Vitual memory increasing

Satish Kumar Malisetti — Fri, 09 Apr 2010 08:42:33 GMT

after complete closure of the process only it is deallocating the complete memory of 3776

Re: Vitual memory increasing

Dennis Handly — Fri, 09 Apr 2010 15:37:56 GMT

>invoked the scripting and allocated memory to the particular process and deallocated but virtual memory is increasing but not decreasing or returning

That's correct. Once you malloc something the space is kept around until you need it again.

>is this memory leak?

No. The space is available for your next record/file processing.
There is a mallopt(3) option, M_REL_LAST_FBLK, that attempts to free the last region in the heap but that may not be practical in most cases.