Vitual memory increasing

[Satish Kumar Malisetti]

All,

in my application i used start a process it occupies virtual memory and while processing the files with the same PID it is increasing the virtual memroy size

ex:

i have started process x occupied memory 1 GB
under that i have processed multiple files it is increasing the virtual memory 1.04 GB , i have verifed the open handles before and after processing files is same and the oracle session is same

can any body suggest me to how to know the appended memory ?

[James R. Ferguson · Accepted solution]

Hi:

First, a process that 'malloc()'s memory and then subsequently 'free()'s that memory doesn't see the freed memory returned to the operating system until the process terminates. Instead, the freed memory is returned to the process's "heap" for reuse.

This is by design to reduce the overhead of memory allocation and deallocation and to reduce memory fragmentation.

I'm not sure the 4% increase you see is significant. Memory leaks and heap fragmentation will lead to growth in a process's size.

See this thread, with particular regard to the 'mallinfo(3)' and 'wdb':

http://forums.itrc.hp.com/service/forums/questionanswer.do?threadId=1351858

Regards!

...JRF...

[Michael Steele_2]

Hi

To find a memory leak you need a script executed in cron every 20 minutes or so. This command is the meat of the script.

UNIX95=1 ps -ef -o vsz,pid,ppid,state,wchan,args | sort -rn | head 10 >> outfile

You are finding biggest 'vsz' consumers and by taking a report every 20 minutes you are observing any growth.

[Bill Hassell]

You can't fix program x problems with HP-UX changes. The process is growing because of how it was designed. Whether that growth is correct or not requires information from the programmer. As mentioned, a 4% growth is insignificant. When the program grows to 1.5 to 2 GB, you definitely need to contact the programmer and get the code fixed (if necessary). After all, the programmer may tell you that the program is correctly designed to grow to 50 GB under some circumstances.

[Dennis Handly]

>can anybody suggest me to how to know the appended memory?

As JRF suggested, you can look at my replies in that thread.
Ideally, for every file you process, you should come back to some constant state. But if your files are larger and need more heap, you would keep growing.
But if you processed that "large" file first, it should grow to that value right away. (I'm not sure if that experiment is easy to do for you?)
Except if you have heap fragmentation, then the small requests split up the big blocks so nobody can use them.

>Bill: You can't fix program x problems with HP-UX changes.

Some times you can with something like MallocNextGen. Especially with heap fragmentation, it really isn't the program's fault.

>the programmer may tell you that the program is correctly designed to grow

In this case, the program must be remembering some state for each file.

[Satish Kumar Malisetti]

I have tried the with malloc env varibles as posted in above link

export _M_ARENA_OPTS=1:2024
export _M_SBA_OPTS=65536:50:256

but still my virtula memory is increasing after processing file 7MB per file and

i can't test it based on huge file first and small file later coz i can't expect the upstream file sizes

Any suggessions please

[Satish Kumar Malisetti]

Bill ,

I am from the development team itself ,
as per the design virual memory should not increase , if it reaches around 8GB then component/particular process automatically go down

[Dennis Handly]

>but still my virtual memory is increasing after processing file 7MB per file

Then you need to first assume you have a memory leak and use gdb to find them.

>as per the design virtual memory should not increase

Then if you don't have leaks, you have heap fragmentation. Have you tried MallocNextGen?

[Satish Kumar Malisetti]

can i have any links how to use
mallocnextgen and gdb please

[Dennis Handly]

>can I have any links how to use mallocnextgen and gdb

Did you look at all the URLs in thread JRF listed above?

[Satish Kumar Malisetti]

Dennis,

I am from development background , i am unable to get clearly how to use these

please could you expalin with a example

[Dennis Handly]

>I am unable to get clearly how to use these

Have you read the WDB document: Debugging dynamic memory usage errors using HP WDB
http://www.hp.com/go/wdb

>please could you explain with a example

Should be some examples above.

You could call a function like this, for every file you process:
#include
#include
void printMallocInfo(const char* title) {
struct mallinfo info;
info = mallinfo();
printf("%s\n",title);
printf("Memory allocation info:\n");
printf(" total space in arena = %d\n", info.arena);
#ifdef DETAILS
printf(" number of ordinary blocks = %d\n", info.ordblks);
printf(" number of small blocks = %d\n", info.smblks);
printf(" space in holding block headers = %d\n", info.hblkhd);
printf(" number of holding blocks = %d\n", info.hblks);
printf(" space in small blocks in use = %d\n", info.usmblks);
printf(" space in free small blocks = %d\n", info.fsmblks);
printf(" space in ordinary blocks in use = %d\n", info.uordblks);
printf(" space in free ordinary blocks = %d\n", info.fordblks);
printf(" keep option space penalty = %d\n", info.keepcost);
#else
printf(" space in use = %d\n",
info.usmblks + info.uordblks);
printf(" space free = %d\n",
info.fsmblks + info.fordblks);
#endif
}

[Satish Kumar Malisetti]

Dennis,

i can't this function from my application/while file processing it it not supporting c functions or calls , it is seperate scripting language dsd

how to proceed on this ?

please help me

[Satish Kumar Malisetti]

dennis,

(const char* title)

colud you please let me know what is this ?

[Satish Kumar Malisetti]

Dennis,

I have set the env variables as given in thread http://forums13.itrc.hp.com/service/forums/questionanswer.do?admit=109447627+1270727289258+28353475&threadId=1360331


set unwindonsignal on
set heap-check leaks on
set heap-check frame-count 16

and ran the application after i checked with gdb

it is showing no processes are running

cbh10702 $ gdb
HP gdb 6.0 for HP Itanium (32 or 64 bit) and target HP-UX 11iv2 and 11iv3.
Copyright 1986 - 2009 Free Software Foundation, Inc.
Hewlett-Packard Wildebeest 6.0 (based on GDB) is covered by the
GNU General Public License. Type "show copying" to see the conditions to
change it and/or distribute copies. Type "show warranty" for warranty/support.

(gdb) info leak
There is no running process.

do i need to call this gdb from the application ?

please suggest me

[Satish Kumar Malisetti]

Dennis,

When i am ran the gdb with particular pid

gdb -pid 4395

i got the messages like

Attaching to process 4395
Reading symbols from /fwdev/home/fwprod1/data/mediation/product/bin/cor...
warning: Load module /fwdev/home/fwprod1/data/mediation/product/bin/cor has been stripped.
Debugging information is not available.

warning: Load module /fwdev/home/fwprod1/data/mediation/product/lib/libfworks.sl has been stripped.
Debugging information is not available.

attached the log for the same

please suggest me to proceed on this?

[Dennis Handly]

>When I ran the gdb with particular pid
>please suggest me to proceed on this?

Did you read the wdb document?
You won't be able to attach to a process instead of starting off in gdb directly:
(gdb) set heap-check on
librtc is not loaded: Either use -leaks command line option, set heap-check before starting the program, or link librtc explicitly

[Satish Kumar Malisetti]

Dennis,

I have tried in different way of testing my application

invoked the scripting and allocated memory to the paricular process and deallocated but virtual memory is incresing but not decreasing or returning

ex:
cbh10702 $ dsd
DSD> getpid
14581
DSD> getppid
10481
DSD> allocate 1200000
DSD> allocated
1200000
DSD> deallocate 1200000
DSD> allocated
0

cbh10702 $ UNIX95= ps -ex -o sz=Kbytes -o vsz=Kbytes -o ruser,pid,ppid | grep 10481
651 2624 fwsup01a 14581 10481
128 548 fwsup01a 10481 10480
cbh10702 $ UNIX95= ps -ex -o sz=Kbytes -o vsz=Kbytes -o ruser,pid,ppid | grep 10481
939 3776 fwsup01a 14581 10481
128 548 fwsup01a 10481 10480
cbh10702 $ UNIX95= ps -ex -o sz=Kbytes -o vsz=Kbytes -o ruser,pid,ppid | grep 10481
939 3776 fwsup01a 14581 10481
128 548 fwsup01a 10481 10480

after deallocation also its is not decreased the size of virtual memory

is this memory leak ?

please suggest

[Satish Kumar Malisetti]

after complete closure of the process only it is deallocating the complete memory of 3776

[Dennis Handly]

>invoked the scripting and allocated memory to the particular process and deallocated but virtual memory is increasing but not decreasing or returning

That's correct. Once you malloc something the space is kept around until you need it again.

>is this memory leak?

No. The space is available for your next record/file processing.
There is a mallopt(3) option, M_REL_LAST_FBLK, that attempts to free the last region in the heap but that may not be practical in most cases.

[James R. Ferguson]

Hi (again):


I have tried in different way of testing my application...invoked the scripting and allocated memory to the paricular process and deallocated but virtual memory is incresing but not decreasing or returning...after deallocation also its is not decreased the size of virtual memory...is this memory leak ?

Go back to the beginning of this thread and my first response:

> me: a process that 'malloc()'s memory and then subsequently 'free()'s that memory doesn't see the freed memory returned to the operating system until the process terminates. Instead, the freed memory is returned to the process's "heap" for reuse...

Regards!

...JRF...

[Satish Kumar Malisetti]

Thanks for the response

i have one more query on the same memory
we have a process which refresh the oracle table data from a file

if i am running the process with 12400 records in a file , every time it it increasing the virtual memory in different sizes of kb

if the same procedure is using and the same number of files are loading why the virual memory is increasing

please suggest me ?

[Michael Steele_2]

First of all, it would make me feel better if you assigned points to all of those above me who have spent time with you working out this problem.

As for your problem, virtual memory is increasing only when you achieve the 12400 record in a file mark.

Question: Are you saying that when this procedure is run elsewhere and 12,400 records are put into a file, you don't see virtual memory increasing?

[Satish Kumar Malisetti]

Michael,

I will assign the points now , thanks for reminder

If there is no records then only virtual memory is not increasing

if there is 1 record also it is increasing the virual memory

[James R. Ferguson]

Hi (again) Satish:

> we have a process which refresh the oracle table data from a file...if i am running the process with 12400 records in a file , every time it it increasing the virtual memory in different sizes of kb ... if the same procedure is using and the same number of files are loading why the virual memory is increasing

Without knowing the internals of the process this is hard to say. Re-read everything said thus far. Why are you concerned, or is this just a matter of academic interest?

Regards!

...JRF...