topic Re: Catastrophic error #21015: Out of memory in Operating System - HP-UX

Catastrophic error #21015: Out of memory

Nandisha — Mon, 21 Dec 2009 12:10:30 GMT

Build of one of oracle (11g) source file gives following error:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Catastrophic error #21015: Out of memory

1 catastrophic error detected in the compilation of "kdli.c".
Compilation terminated.
gmake: *** [kdli.o] Error 4
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

I am using following compiler version:
$ ... ansic/bin/cc -V
cc: HP C/aC++ B3910B A.06.14 [Feb 22 2007]

Any idea why this error? Is there any known is reported on that compiler?

Re: Catastrophic error #21015: Out of memory

Steven E. Protter — Mon, 21 Dec 2009 14:08:14 GMT

Shalom,

I'd take the error on its merits.

as root, check swapinto -tam
Lets see what the memory situation actually is.

I'd also check the system is patched appropriately, the error could come from not meeting Oracle patch requirements.

SEP

Re: Catastrophic error #21015: Out of memory

Bill Hassell — Mon, 21 Dec 2009 15:00:37 GMT

Building Oracle normally requires enormous amounts of memory (dozens of GB), much more than the running Oracle. Your installation notes from Oracle should indicate how much memory you will need. Note that this does not have to be real memory, it can be mostly swap space. This is temporary for the build process -- use swapon to add an unused lvol while building. After reboot, the lvol will be returned to unused state.

Re: Catastrophic error #21015: Out of memory

Dennis Handly — Mon, 21 Dec 2009 16:37:40 GMT

If you are building (and working for?) Oracle, you should be contacting HP directly, not asking here. (Unless you know that most of HP is closed for the end of the year. :-)

>Catastrophic error #21015: Out of memory

Since you aren't on A.06.15 or later where you you need to configure maxdsiz_64bit to 4 GB or larger, you need to do what SEP and Bill suggests and look at your swap and maxdsiz.
The latest version of aC++ is A.06.24.

>Bill: Building Oracle normally requires enormous amounts of memory (dozens of GB)

This version of the compiler can only use 1 GB at a time.

>Your installation notes from Oracle should indicate how much memory you will need.

To install Oracle shouldn't take large amounts of swap, only building it.

Re: Catastrophic error #21015: Out of memory

Nandisha — Tue, 22 Dec 2009 13:21:48 GMT

Thanks for the reply.

I know it's shutdown period for hp folks and not available till 3rd of next month :-)

Looks like we have a good memory. See detail here:

$ swapinfo
Kb Kb Kb PCT START/ Kb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8388608 9900 8378708 0% 0 - 1 /dev/vg00/lvol2
dev 37109760 0 37109760 0% 0 - 5 /dev/vg00/lvol10
dev 8388608 9756 8378852 0% 0 - 1 /dev/vg01/lvol1
dev 32563200 0 32563200 0% 0 - 5 /dev/vg01/lvol3

Re: Catastrophic error #21015: Out of memory

Dennis Handly — Tue, 22 Dec 2009 14:28:16 GMT

>$ swapinfo

Please provide "swapinfo -tam" as SEP asked.
Also provide the values of maxdsiz, maxdsiz_64bit, maxssiz and maxssiz_64bit.

What optimization options are you using?

Re: Catastrophic error #21015: Out of memory

Nandisha — Wed, 23 Dec 2009 09:56:23 GMT

> Please provide "swapinfo -tam" as SEP asked.

Kindly see the same below:
$ swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol2
dev 8192 0 8192 0% 0 - 1 /dev/vg01/lvol1
reserve - 885 -885
memory 16364 4012 12352 25%
total 32748 4897 27851 15% - 0 -

> Also provide the values of maxdsiz, maxdsiz_64bit, maxssiz and maxssiz_64bit.

$ kctune |grep -i maxdsiz
maxdsiz 2952790016 2952790016 Immed
maxdsiz_64bit 17179869184 17179869184 Immed

$ kctune |grep -i maxssiz
maxssiz 134217728 134217728 Immed
maxssiz_64bit 1073741824 1073741824 Immed
pa_maxssiz_32bit 83648512 Default
pa_maxssiz_64bit 536870912 Default

> What optimization options are you using?

+Olibmerrno +Ovolatile=__unordered +Ofast +Olit=all +DSblended +Onopreserved_fpregs +Onorotating_fpregs +Onofailsafe +Odatalayout +FPd +Ofltacc +O2 +Oprofile=use:.../flow.data

env variable 'PROFILE_ON=use' is set.

Should we reduce/disable optimization? If so, is this will not create any unexpected results/probolems?

Note:-
Actually, this issue (Catastrophic error #21015: Out of memory) is start coming after applying one of the oracle patch which is adding fix code changes to xyz.c source file. And then the issue is coming during compilation of xyz.c.

Re: Catastrophic error #21015: Out of memory

Dennis Handly — Wed, 23 Dec 2009 10:34:09 GMT

>total 32748 4897 27851 15%

Now, you have plenty of swap.

>maxdsiz 2952790016

This is more than ecom can use.

>maxssiz 134217728

This is way too much. You could recompile by first using:
ulimit -s $(( 40 * 1024 ))

>Should we reduce/disable optimization? If so, is this will not create any unexpected results/problems?

You'll have to do that. Either that or go to the latest, A.06.24, where there is no limit.

Otherwise dropping the optimization level may slow down the application.
If you aren't using the very latest compilers, you should remove +Onofailsafe.

>after applying one of the oracle patch which is adding fix code changes to xyz.c source file. And then the issue is coming during compilation of xyz.c.

If you know the function(s) that changed, then you can either split them up or use:
+O1=function1,function2

Re: Catastrophic error #21015: Out of memory

Nandisha — Wed, 23 Dec 2009 12:28:05 GMT

> ulimit -s $(( 40 * 1024 ))

$ ulimit -s
131072

$ ulimit -s 40960
$
$ ulimit -s
131072

The value 40960 is not taken effect for stack size change. Any idea why is it so?

Oracle 11gR1 is recommended A.06.14. So, I don't think we can upgrade to latest which is A.06.24.

Trying out one of other suggested options i.e., removing 'Onofailsafe' and compiling the source. Will update the results.

Re: Catastrophic error #21015: Out of memory

Nandisha — Wed, 23 Dec 2009 12:37:59 GMT

I am able to build my source without +Onofailsafe option.

Is this okay to use as a workaround? OR is it worth while to try other option? i.e.,
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you know the function(s) that changed, then you can either split them up or use:
+O1=function1,function2
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Re: Catastrophic error #21015: Out of memory

Dennis Handly — Wed, 23 Dec 2009 14:21:44 GMT

>I am able to build my source without +Onofailsafe option.

Do you get a warning saying the compile is restarting at a lower opt level?
(In future compilers, we will no longer do that if we run out of memory.)

>Is this okay to use as a workaround? OR is it worth while to try other option?

If it drops the opt level of the whole file then you'll probably get better performance if you limit the opt level drop with:
+O1=function1,function2

Re: Catastrophic error #21015: Out of memory

Nandisha — Thu, 24 Dec 2009 10:07:35 GMT

1.
After removing option +Onofailsafe, it builds with folloiwng warning:
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Warning #21015-D: Out of memory

Warning #20013-D: Optimization aborted in function unknown. Attempting
compilation unit again but dropping the optimization level to +O1.
Please report this warning and error 21015 to HP. (20013)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

2.
I directly lowering the optimization from +O2 to +O1. With this I am able to build required object file.

3.
Then I tried to include almost all changed procedure to +O1 (+O1=func1,func2, func3, etc) in the compiler line. But, still I got same Catastrophic error #21015. So, I don't think it's too effective.

Re: Catastrophic error #21015: Out of memory

Dennis Handly — Thu, 24 Dec 2009 12:04:02 GMT

>After removing option +Onofailsafe, it builds with following warning:

Ok, the whole file is dropped to +O1 and unfortunately it doesn't list a particular function name. Do you have lots of data initialization?

>I directly lowering the optimization from +O2 to +O1.

That's the same as +Ofailsafe.

>3. Then I tried to include almost all changed procedure to +O1 (+O1=func1,func2, func3, etc) in the compiler line.

You may have to include ALL functions and then binary search though them to remove them.

Or break up any large functions.

Re: Catastrophic error #21015: Out of memory

Javed Khan_1 — Thu, 24 Dec 2009 12:58:26 GMT

also look at the space available in /tmp

Re: Catastrophic error #21015: Out of memory

Dennis Handly — Thu, 24 Dec 2009 13:18:04 GMT

>Javed: also look at the space available in /tmp

Why? It would be illegal for the compiler to use space in /tmp/. It uses /var/tmp/ or $TMPDIR and then very very rarely. Use "cc -v".
error #21015: Out of memory is for swap space or maxdsiz and not disk space.

Re: Catastrophic error #21015: Out of memory

Nandisha — Mon, 28 Dec 2009 12:31:21 GMT

Thanks..Will check this.