Growing process memory

[Benedetto Mangiapane]

I've two system for a test.
In the first, the memory of a process growing (at least 400Mb and after the kernel kill it). In the second with the same process, the memory is normal (24 Mb).

(uname -a)
HP-UX first B.11.11 U 9000/800 1106414681 unlimited-user license

HP-UX second B.11.11 U 9000/800 3678999180 unlimited-user license

The process use the STL routine (map, list, ...), and TCSI Solution Core OSP framework.

Question:
1 - There are particular cargo conditions, that they produce results of this type?

2 - It is possible that the memory in a system comes freed automatically, and in the other not?

3 - How I can control and verify these anomalies?

Thanks.
Benedetto.

[Peter Godron]

Benedetto,
if you are running the same executable on two machines and the behaviour is different, you must have a different environment/libraries.

Have you tried re-compiling the code into a standalone module (no calls to external libs) and re-running it?

[Benedetto Mangiapane]

I cannot compile the process with the method stand-alone because part of a package larger that contains other processes that link the shared libraries.

I would want to try to understand what happens...

I have need of some suggestions in order to analyze the problem and to try to find the cause.

[Dennis Handly]

You should use gdb's leak detection commands to see where it is leaking/growing. You can download gdb for free.

[Benedetto Mangiapane]

Hi,
I have tried to find through wdb the memory leaks, but I have not found any. There is virtual memory busy. I use the STL very much, like map very, vector, list... etc.

I think that the problem comes from the the free or delete operation when it comes freed the memory used from the STL object.

(In the wdb memory use window, there are many new [] operations...)

I go for the just road?
Thanks.

[Dennis Handly]

>I have tried to find through wdb the memory leaks, but I have not found any.

If no leaks, can you compare the systems by using "info heap" on both.

There should be a big difference if one is 400 Mb and the other 24 Mb. Do both processes have the same inputs?

[Benedetto Mangiapane]

The same input for the processes.

In the thread http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=1076848, Don Morris talk about similar problem.

Insert the swapinfo of two nodes:

FIRST > swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 2640 1456 64% 0 - 1 /dev/vg00/lvol2
dev 1600 0 1600 0% 0 - 2 /dev/vg00/lvol_swap
dev 8000 2635 5365 33% 0 - 1 /dev/vg_swap/lvol_swap2
reserve - 8205 -8205
memory 6286 4217 2069 67%
total 19982 17697 2285 89% - 0 -

SECOND > swapinfo -atm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 4096 2751 1345 67% 0 - 1 /dev/vg00/lvol2
dev 8192 2752 5440 34% 0 - 1 /dev/vg00/lvol_swap
reserve - 6785 -6785
memory 6282 5585 697 89%
total 18570 17873 697 96% - 0 -

[Dennis Handly]

>The same input for the processes.

Do you have the same patches on each?
And what does gdb's heap commands say about each?

>Don Morris talk about similar problem.

Yes, but there never was a resolution.

>Insert the swapinfo of two nodes:

This just shows more VM total on FIRST.

[Benedetto Mangiapane]

To the verification through the gdb (wdb), info heap it gives back to me "Heap analysis is not enabled now."

To enable?

[Benedetto Mangiapane]

Extract of info heap:

No. Total bytes Blocks Address Function
0 4055040 30 0x62867000 __alloc_stack()
1 2547160 28945 0x40f49700 __libc_mutex_alloc()
2 911168 1165 0x40ba2cf0 operator new(unsigned long)()
3 370494 1 0x407050f0 sktsfMalloc()
4 340105 12940 0x4112d5c8 operator new(unsigned long)()
5 281764 6 0x4076afa8 lxldalc()
6 262496 1 0x408094a0 nsgbliuc()
7 257294 9891 0x40f4f828 operator new(unsigned long)()
8 230296 2617 0x406891b0 __libc_mutex_alloc()
9 157144 4936 0x40b39ad8 operator new(unsigned long)()
10 145768 6 0x40f60b60 operator new(unsigned long)()
11 139264 1 0x4087d2c8 slwmmgetmem()
12 94080 84 0x41332130 operator new(unsigned long)()
13 69632 1 0x6297d000 __alloc_stack()
14 65620 1 0x407f9438 nsgbliuc()
15 65620 1 0x407e93d0 nsgbliuc()
16 65620 1 0x407d9368 nsgbliuc()
17 65620 1 0x407c9300 nsgbliuc()
18 65608 11 0x409349b0 operator new [](unsigned long)()
19 65536 1 0x40af6c10 operator new [](unsigned long)()
20 65288 9 0x4077f818 sktsfMalloc()
21 65000 1 0x40558520 operator new [](unsigned long)()
22 57344 1 0x40849610 nsgbliuc()
23 49468 9 0x40c1c128 operator new(unsigned long)()
24 30561 1165 0x412bf550 operator new(unsigned long)()
25 28160 55 0x4054e7d8 operator new [](unsigned long)()
26 24064 47 0x40933dd8 operator new [](unsigned long)()
27 20340 1 0x4075f840 lxldalc()
28 20000 1 0x40548d48 operator new [](unsigned long)()
29 18944 8 0x409c8e60 __nss_XbyY_buf_alloc()
30 17896 1 0x40765030 sktsfMalloc()
31 16844 5 0x40c271f0 operator new(unsigned long)()
32 16416 2 0x406fcef8 _findbuf()
33 16384 8 0x4097c300 memset()
34 16280 10 0x40e14a70 operator new(unsigned long)()
35 9320 1165 0x40962d80 operator new(unsigned long)()
36 9272 61 0x40782a90 operator new(unsigned long)()
37 9036 753 0x406e7018 operator new(unsigned long)()
38 8772 731 0x406db368 operator new(unsigned long)()
39 8704 17 0x40592f68 operator new [](unsigned long)()
40 8240 2 0x4076bb58 sktsfMalloc()
41 8060 31 0x405536a0 operator new(unsigned long)()
42 7728 1 0x408608b8 nsgbliuc()
43 7080 30 0x40575b30 __pthread_id_lookup()
44 6720 5 0x40942d38 operator new(unsigned long)()
45 6144 12 0x4054f458 operator new [](unsigned long)()
46 5760 30 0x406f7a50 __private_data_setup()
47 5632 1 0x4053c640 __libCsup_mutex_init()
48 4480 1 0x406fef18 localtime_r()
49 4448 1 0x4085b368 nsgbliuc()
50 4264 1 0x408741d8 slwmmgetmem()
51 4200 1 0x407bf8b0 nngsini_init_streams()
52 4187 295 0x40580470 operator new [](unsigned long)()
53 4136 1 0x4077c740 sktsfMalloc()
54 4096 2 0x408626f8 nlhtnsl()
55 4096 2 0x40862f08 nlhtnsl()
56 3920 35 0x4085dd30 operator new(unsigned long)()
57 3584 32 0x4085e530 operator new(unsigned long)()
58 3540 295 0x4057dea8 operator new(unsigned long)()
59 3468 1 0x40875290 lpmrist()
60 3360 35 0x406f4ac0 operator new(unsigned long)()
61 3300 55 0x40546928 operator new(unsigned long)()
62 3072 1 0x40c1b0b0 operator new(unsigned long)()
63 2664 14 0x40555bd0 operator new [](unsigned long)()
64 2536 159 0x406e8c08 operator new [](unsigned long)()
65 2440 2 0x40c23ed0 operator new(unsigned long)()
66 2368 1 0x408646c8 __nss_XbyY_buf_alloc()
67 2268 1 0x40865018 nsmal()
68 2208 1 0x4076b2a8 kouodalc()
69 2184 31 0x406d9058 operator new [](unsigned long)()
70 2160 90 0x40864328 operator new [](unsigned long)()
71 2136 1 0x407647c8 lxldalc()
72 2104 1 0x408ab6e0 __nss_XbyY_buf_alloc()
73 2070 1 0x408aa690 nsbGetBFS()
74 2070 1 0x408a8c08 nsbGetBFS()
75 2070 1 0x408aaeb8 nsbGetBFS()
76 2070 1 0x408a9e68 nsbGetBFS()
77 2070 1 0x408a9430 nsbGetBFS()
78 2068 11 0x409087a0 operator new(unsigned long)()
79 2048 1 0x40b08c60 operator new [](unsigned long)()
80 2048 1 0x40b09470 operator new [](unsigned long)()
81 2048 1 0x40b08450 operator new [](unsigned long)()
82 2048 1 0x40b07c40 operator new [](unsigned long)()
83 2048 1 0x40b07430 operator new [](unsigned long)()
84 2048 1 0x40ad09d0 operator new [](unsigned long)()

....

[Bill Hassell]

The simple answer is that your RAM is far too small (in both machines) for the processes that you are running. Once all the processes use up available RAM, processes will be deactivated, then paged out to swap. When those processes are needed, other processes must be deactivated and paged out. By doubling your current RAM, you can avoid this massive impact on performance. vmstat will help in showing the memory pressure -- look at the po column. 2 digit (or larger) numbers indicate significant lack of RAM. Use this command to sort the top 10 processes by local (heap) memory usage:

UNIX95=1 ps -e -o vsz,ruser,pid,args | sort -rn | head -10

(the UNIX95 variable is require to activate the -o options -- see man ps). The ideal performance for a busy system is virtually no swap space used to pageouts. Some swap space is used for memory mapped files which is normal.

[Benedetto Mangiapane]

vmstat of the first node:

vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
6 1 0 2628838 69990 471 91 17 1 0 0 17 835 45843 11734 33 13 53

po = 1

and second node:

vmstat
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
2 0 0 2017996 15424 401 150 11 0 0 0 12 1162 28140 3665 9 7 84

po = 0.

Therefore, adding physical memory I would have to resolve the problem?

[Bill Hassell]

Based on the vmstat numbers, po is too small to be a concern. But this is a single measurement and the system appears not to be paging at all. So the applications may be highly interactive which means that the page outs took place at some other time. It is quite possible to run a lot of applications that won't all fit into memory at the same time, as long as they aren't computing at the same time. Once all the apps start computing, the po rate will skyrocket and performance will suffer. If the application is primarily compute-bound (very little disk I/O) then not much can be done to improve performance except a rewrite or faster CPUs.

[Dennis Handly]

>"Heap analysis is not enabled now." To enable?

I assume you figured this out because you produced some output in your next message?

Where is the results for the other system?

The total from what you provide is only 10.9 Mb. Where is the 400 Mb example?

If you want more details on each, you can use "info heap #". This will give a call stack for each. You may need to increase the depth if that isn't enough.

1 2547160 28945 0x40f49700 __libc_mutex_alloc
8 230296 2617 0x406891b0 __libc_mutex_alloc
47 5632 1 0x4053c640 __libCsup_mutex_init

The first two if these can be greatly reduced if the all come from C++ strings. Just compile with -D__HPACC_FIXED_REFCNT_MUTEX:
http://www.docs.hp.com/en/5991-4872/ch01s03.html#chdgeeid

[Benedetto Mangiapane]

In the info heap list, the biggest allocated memory is #0. The command info heap 0, returns:

(gdb) info heap 0
9357656 bytes in 106337 blocks (39.57% of all bytes allocated)
These range in size from 88 to 88 bytes and are allocated
#0 __libc_mutex_alloc() from /lib/libc.2
#1 HPMutexWrapper::init(void)() from /lib/libstd.2
#2 basic_string,allocator>::getRep(unsigned long, unsigned long)() at /sw_common/aCC-3.31/aCC/include/string.cc:112
#3 basic_string,allocator>::replace(unsigned long, unsigned long, char const *, unsigned long, unsigned long, unsigned long)() at /sw_common/aCC-3.31/aCC/include/string.cc:459


(gdb) info heap 1
4055040 bytes in 30 blocks (17.15% of all bytes allocated)
These range in size from 135168 to 135168 bytes and are allocated
#0 __alloc_stack() from /lib/libpthread.1
#1 __pthread_id_lookup() from /lib/libpthread.1
#2 __pthread_create_system() from /lib/libpthread.1
#3 thrThreadServerPOSIX::StartThread(void)() from /opt/tcsi/osp/5.4/hpux1111/lib/libOspThreads.sl

The allocated memory is not a lot, but it grows little for time.

I have compiled the processes with the -D__HPACC_FIXED_REFCNT_MUTEX directive and I'm looking the behaviour.

[John Guster]

How about share memory area?
ipcs -mob to check if there are any segment with zero process attached.

[Dennis Handly]

>I have compiled the processes with the -D__HPACC_FIXED_REFCNT_MUTEX directive and I'm looking the behaviour.

This should remove all of #0. The previous #47 is the fixed pool of 64 mutexes. You might look at #8.

__alloc_stack is used to allocate thread stacks.

[Benedetto Mangiapane]

For John Guster.
The output of command:

IPAHU028 > ipcs -mob
IPC status from /dev/kmem as of Thu Feb 1 08:20:23 2007
T ID KEY MODE OWNER GROUP NATTCH SEGSZ
Shared Memory:
m 0 0x411865f3 --rw-rw-rw- root root 0 348
m 1 0x4e0c0002 --rw-rw-rw- root root 1 61760
m 2 0x411c06df --rw-rw-rw- root root 1 8192
m 6735 0x0c6629c9 --rw-r----- root sys 1 33657808
m 4 0x06347849 --rw-rw-rw- root root 0 77384
m 617 0x4910c3e3 --rw-r--r-- root root 0 22908
m 50190 0x5e14003f --rw------- root root 1 512
m 29995 0x00000000 D-rw------- root root 8 1052672
m 11636 0x00000000 D-rw------- www other 8 184324
m 7353 0x0654b034 --rw-rw---- root sys 108 337088512
m 2458 0xe8ba27c4 --rw-rw---- root sys 104 722964480
m 4907 0x353be1d0 --rw-rw---- root sys 63 337088512
m 10416 0x32e46938 --rw-rw---- root sys 73 190812160
m 9193 0xf5f50cc8 --rw-rw---- root sys 37 72323072


For Dennis Handly.
You can be explained?
Excuse me but I not use these tools often.

I must change the environment variable aCC_MUTEX_ARRAY_SIZE?

[Dennis Handly]

>I must change the environment variable aCC_MUTEX_ARRAY_SIZE?

How many CPUs do you have? The default is 64 to make sure there isn't too much contention when the runtime randomly shares mutexes.

[Benedetto Mangiapane]

The number of CPU is 2 or 4.


The output of info heap today is:

No. Total bytes Blocks Address Function
0 290869526 3870584 0x4eb0ba90 ???()
1 9357656 106337 0x411e3440 __libc_mutex_alloc()
2 4055040 30 0x626bb000 __alloc_stack()
3 3355456 4269 0x415d42d0 operator new(unsigned long)()
4 1253305 47676 0x41ac9998 operator new(unsigned long)()
5 947422 36419 0x42561ac8 operator new(unsigned long)()
6 569672 17928 0x40f7ab58 operator new(unsigned long)()
7 372448 48 0x41015c90 operator new(unsigned long)()
8 370494 1 0x407050f0 sktsfMalloc()
9 344960 308 0x4100e120 operator new(unsigned long)()
10 281764 6 0x40769910 lxldalc()
....

With "info heap 0":

(gdb) info heap 0
290869526 bytes in 3870584 blocks (92.49% of all bytes allocated)
These range in size from 4 to 34812 bytes and are allocated

How I verify this memory?

[Dennis Handly]

>1 9357656 106337 0x411e3440 __libc_mutex_alloc()

These means you still need to recompile some libs with -D__HPACC_FIXED_REFCNT_MUTEX. There are ~106 K strings.


(gdb) info heap 0
290869526 bytes in 3870584 blocks (92.49% of all bytes allocated)
These range in size from 4 to 34812 bytes and are allocated

>How I verify this memory?

Well without a function or stack trace, I'm not sure what it is trying to tell you??

That size is maybe that 400 Mb you initially mentioned.

What version of wdb do you have?

[Benedetto Mangiapane]

Ok I must recompile all libs.

The version of wdb is:
HP WDB-GUI - HP Debugger Graphical User Interface - Version 5.4
Rev: 20060320.192245

Infact, the function is unknown "???()", I ask to you if it is possible to understand in other way?

[Dennis Handly]

>Ok I must recompile all libs.

No, you don't have to do that. But if you want to get the reduced memory and 5 to 10% speed up, you should do it.

> HP WDB-GUI - HP Debugger Graphical User Interface - Version 5.4

You may want to download a newer gdb.

>the function is unknown "???()", I ask to you if it is possible to understand in other way?

I didn't see any stack trace when you did:
(gdb) info heap 0

Did you snip it, or is this a symptom of why things are messed up?

I hope you haven't stripped your executable??
But I would have thought there would be more of them if that was true.

[Benedetto Mangiapane]

Hi,
I call chatr command to the process, and it show me:

...
global hash table disabled
plabel caching disabled
global hash array size:1103
global hash array nbuckets:3
shared vtable support disabled
static branch prediction disabled
executable from stack: D (default)
kernel assisted branch prediction enabled
lazy swap allocation disabled
text segment locking disabled
data segment locking disabled
third quadrant private data space disabled
fourth quadrant private data space disabled
third quadrant global data space disabled
data page size: D (default)
instruction page size: D (default)
nulptr references disabled
shared library private mapping disabled
shared library text merging disabled

Someone of these its not correct?
Not still I resolved the problem...

[Dennis Handly]

>I call chatr command to the process, and it show me:

Why did you use chatr(1)?

>Someone of these its not correct?

Nothing obviously wrong.