not enough memory for compilation

[T G Manikandan]

We have a UCM environment where we used to do build process which involves compilation.

We have almost 2Gb RAM on all the servers

Frequently we get

aCC :not enough memory to finish the compilation

i have set maxdsiz to high value.
There are no processes running on the server.
2GB RAM +4GB swap
Is there errors are for shortage of memory?
Is there something else I should look at.


Thanks

[kish_1]

looks like memory, check following URL you will have some clue
http://lists.trolltech.com/qt-interest/2001-05/thread00546-0.html

[T G Manikandan]

At present I am running a compilation process
which goes fine on a L-2000
with memory 2GB

but it gives
aCC not enough memory to finish the compilation on a A-class 580 model machine also with 2GB memory.



How shall I approach the problem?

Thanks

[Hai Nguyen_1]

Do you have glance?
If so, you can run "glance -m" from command line and post the current memory utilization on the two boxes.
Then we go from there.

If you do not have glance,
you can run "swap -tam" and post the output here from the two boxes for us to analyze?

Hai

[A. Clay Stephenson]

I would first make sure that the patch levels for the compiler are the same on both compilers. It appears that maxdsiz is big enough but you may need to increase maxssiz. I would do a kmtune (or sydef) of both boxes and compare the tunables.

[Hai Nguyen_1]

Sorry.

# swap -tam
should actually have read as follows:
# swapinfo -tam

Hai

[Wodisch_1]

Hi,

you may have to set the "maxssiz" to a higher value, too, as local variables are allocated on the stack, and the default of 8MB / process isn't that much. Since I did not see your HP-UX version, you may use up to 79MB without problems, even on HP-UX10.20...

HTH,
Wodisch

[Steve Labar]

I'm not sure how large your program is. We have an application that has 500MB of source code. It required 4x RAM of swap before it would compile. For instance we have 512 MB RAM and use 2GB swap.

Good Luck.
Steve

[Sandip Ghosh]

By the way, what is the size of dbc_max_pct of your new A580 Server? If it is default 50% reduce it to 15%.

Sandip

[T G Manikandan]

Could not reach the site from the morning.
This is not yet resolved.


I have attached some documents and reports.Please go through it and let me know the way I should proceed

The document has the the kernel parameter details and patch details.

Everything is similar on both the servers.

I just checked using glance and found that there is no swapping happening but it gives a "not enough memory" error.

I checked the kernel parameters.
maxdsiz 2GB
maxdsiz_64bit 2GB
maxssiz 401MB
maxssiz_64bit 159MB
zmaxtsiz 133MB
maxtsiz_64bit 1GB


How about making the value of
swapmem_on to 0.
I have modified that to 0.


I run glance and found the following information.

This was on a A-class server

Private RSS 640 MB
Data RSS 640 MB
Data VSS 655 MB
Text VSS 10MB

At this point it throws out the error.

This was on the L2000 server
private RSS 520MB
Data RSS 516MB
Data VSS 700MB
Text VSS 9MB

The only difference I can see now is the number of processors on L-2000 is 4 and A-class is 2.

The Private RSS does not increase beyond 520MB on L-2000.

On the A-class the Private RSS and the Data RSS rapidly increase with the Data VSS.

I am attaching the document.

The document has some less values of maxdsiz and maxssiz but those are increase to the above mentioned value.


I am waiting for your replies.
It is late night here.

Thanks

[Hai Nguyen_1]

I looked into your doc and did not spot any problem except for one thing that your maxdsize* have been set very high (about 2GB). If a process has a memory leak, this will allow it to eat up RAM over time. mib2agt is an example.

You can check if mib2agt is running on your box.
#ps -ef | grep mib2agt

If it is then do:
# top -d 1 -f /tmp/top.txt
# grep mib2agt /tmp/top.txt | awk '{print $6}'

which outputs the total memory allocated to this process in KB (ending in K) or
MB (ending in M)... The normal number is around 600K. Any number much bigger than this is a suspect of memory leak. Then you can download the patch PHSS_26138 to fix it. If you do not need it, you can turn it off from its system startup file.

Hai

[T G Manikandan]

THanks Hai,

THe Mibagent process did not utilize much memory.It was around 20K.


but I have stopped it.

Thanks

[Hai Nguyen_1]

Can you run the following commands and post the output file /tmp/output.txt

# swapinfo -tam > /tmp/output.txt
# glance -iterations 1 -m >> /tmp/output.txt

The glance command will take about 15 seconds to complete simply because it needs to run at least 3 times at 5-second interval.

Thanks.

Hai

[T G Manikandan]

The output file is attached


Thanks

[A. Clay Stephenson]

This is pretty strange now. Because you have not explicitly stated this, I'm going to ask. Are both of these systems 11.0 or is one running 11.11?

You have got to compare the patch levels on both of these boxes and not just aCC. It could be a difference in libc. Are the boxes at the same General Release (or Quality Pack or whatever?) patch levels?

The only other thing that is tickling my brain is that you have large amounts of shared memory that are locked in memory and malloc is failing during the compilation.

Look in the patch database for 'aCC' there are a number of patches that relate to header files, libraries, and the compiler itself. You might also try the latest aCC version as 3.30 is a little old.

[T G Manikandan]

Thanks a lot!
I am been waiting all night here to get inputs from you people and get a reply from clay is very special.


Both the systems are running 11.0
Both the machines are with XSWGR1100 Mar2001 and QPK Dec 2000 Patch Bundles.

Both have aCC3.30.
The kernel parameters on both the machines are same.

Clay,
Is there any other kernel parameter I should look at.

I will search the compiler downloads and the patch downloads.

I will keep things updated


Thanks again

[Sridhar Bhaskarla]

Hi,

Though you have set your maxdsiz to a higher value, the actual memory that can be allocated is dependent on what your users set the "ulimit" to.

Let the users execute ulimit and see if it returns 4194303 or unlimited based on their shell.


-Sri

[T G Manikandan]

Great point Sridhar

These are the values

ulimit -aS

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 392192
memory(kbytes) unlimited
coredump(blocks) 4194303
nofiles(descriptors) 2048

ulimit -aH

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 392192
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) 2048

Thanks

[T G Manikandan]

I used tusc to verify the process

I increased the max_thread_proc to see if i can catch things bit it did not work.
I thought to increase maxssiz about its limit in SAM.
Manually increased and re-build the kernel.

nothing works.

After examining the compilation process using tusc
I checked the log file to catch the err#12 ENOMEM error.

There is nothing swapping outside.
I have added another 3GB of swap space.
All my
maxdsiz
maxssiz
maxtsiz hve been in GB's

My shared memory is 2GB.
max_thread_proc
2048
nkthread
2048
maxfiles and maxfiles_lim
3000
maxuprc
600

how I shall I work on the problem.
I checked the make and ld versions on both the systems .They are same.


1.Will a re-installation solve the problem?
2.What else I should look at?

When I probed using tusc I saw the lseek() call giving the not enough memory,
I cleaned up the disk space.

Any more answers.
Some big guys are working today,
Bill and Clay


Thanks

[Bill Hassell]

Just a thought...a standard 32bit executable cannot access any memory beyond about 960 megs. Check the executable on both systems with chatr. Look specifically for the MAGIC options (multi-quadrant addressing for the data area). That may be the difference. The ENOMEM error means:

- All available virtual memory has been exhausted (RAM and extra swap space) ** not likely in this case

- A 32bit executable needs more than 960 megs (or more than 1750 megs for an EXEC_MAGIC executable). ** this is fairly likely

The second scenario is fundamental to 32bit programs...large RAM and swap cannot benefit a 32 bit program once it hits addressing limits. Check the proc_mgt and mem_mgt.txt files in /usr/share/doc.

[T G Manikandan]

That a great point to make Bill.

1.Is that the version of ld,ar and make should also be resposible for the increase in the process data segment size.
2.Can a re-installation solve the problem?


I use the same version of aCC on both the machines.

This is what chatr is to say about the executable


# chatr ./ctcom
./ctcom:
shared executable
shared library dynamic path search:
SHLIB_PATH disabled second
embedded path disabled first Not Defined
shared library list:
dynamic /usr/lib/libm.2
dynamic /usr/lib/libcl.2
dynamic /usr/lib/libc.2
static /usr/lib/libdld.2
shared library binding:
deferred
global hash table disabled
plabel caching disabled
global hash array size:1103
global hash array nbuckets:3
shared vtable support disabled
static branch prediction disabled
executable from stack: D (default)



on both the machines.
THis is a very good you have given me Bill.
Thanks for that.
Everything is simiar on the other machine.
I am trying some other ways to find out a solution.
I don't know whether a re-installation would solve the problem.