topic Re: Out of Memory Error in Operating System - HP-UX

Out of Memory Error

Kavita Poonia — Sun, 15 Jun 2008 19:25:16 GMT

Hello there,

On one of our HP-UX server, we have got following alerts :

Critical 07:51:22 06/15/08 cntrepa1.na.jnj.com HP VantagePoint Operations OpC Scheduled action NCS_OS_HPUX_CUS_Service_Outage(service_outage_check.sh) for user root failed.

Critical 11:36:12 06/15/08 cntrepa1.na.jnj.com HP OpenView Operations OpC opcle (Logfile Encapsulator) Can't run command /sbin/dmesg - > /var/opt/OV/log/OpC/dmesg.out. (OpC30-127) fork(2) failed; no more memory Not enough space (OpC20-411)

On checking server, I have found that both memory and swap utilization are 100%. There are lots of SAP processes running on the server.

ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit
B3692A GlancePlus C.03.86.00 15:52:29 cntrepa1 9000/800 Current Avg High
----------------------------------------------------------------------------------------------------------------------------------------------
CPU Util SU U | 25% 26% 29%
Disk Util FVV | 3% 5% 12%
Mem Util S SU UB B |100% 100% 100%
Swap Util U UR R |100% 100% 100%
----------------------------------------------------------------------------------------------------------------------------------------------
PROCESS LIST Users= 2
User CPU Util Cum Disk Thd
Process Name PID PPID Pri Name ( 400% max) CPU IO Rate RSS Cnt
--------------------------------------------------------------------------------
dw.sapRP1_D 7184 21507 154 rp1adm 95.3/77.0 41451.5 3.9/ 1.9 1.18gb 1
dw.sapRP1_D 22651 21507 155 rp1adm 0.7/ 2.6 52499.0 0.1/ 0.7 140.4mb 1
krsd 3267 1 168 root 0.3/ 0.4 57515.0 3.3/ 4.0 120kb 1
vxfsd 40 0 138 root 0.2/ 0.1 19848.2 2.9/ 8.4 1.8mb 23
midaemon 3241 1 -16 root 0.2/ 0.3 39558.2 0.0/ 0.0 22.0mb 3
sh 11735 21741 179 root 0.0/ 0.0 0.0 1.1/ 1.1 212kb 1
dw.sapRP1_D 1847 21507 155 rp1adm 0.0/ 2.0 22876.9 0.0/ 0.5 71.4mb 1
sh 11766 21741 179 root 0.0/ 0.0 0.0 1.1/ 1.1 212kb 1
dw.sapRP1_D 21507 21488 154 rp1adm 0.0/ 0.1 6423.5 0.0/ 0.0 24.8mb 1
dw.sapRP1_D 21685 21507 155 rp1adm 0.0/ 0.1 3377.5 0.0/ 0.0 38.5mb 1
dw.sapRP1_D 21661 21507 155 rp1adm 0.0/ 0.0 141.8 0.0/ 0.0 31.8mb 1
dw.sapRP1_D 21660 21507 155 rp1adm 0.0/ 0.0 325.5 0.0/ 0.0 31.8mb 1
dw.sapRP1_D 15997 21507 155 rp1adm 0.0/ 0.1 303.1 0.0/ 0.0 45.5mb 1
dw.sapRP1_D 8455 21507 155 rp1adm 0.0/ 0.2 2406.5 0.0/ 0.0 65.6mb 1
dw.sapRP1_D 21679 21507 155 rp1adm 0.0/ 0.0 199.5 0.0/ 0.0 34.7mb 1
dw.sapRP1_D 7020 21507 155 rp1adm 0.0/ 0.1 258.1 0.0/ 0.0 63.0mb 1
dw.sapRP1_D 21675 21507 155 rp1adm 0.0/ 0.0 2.1 0.0/ 0.0 31.8mb 1
dw.sapRP1_D 4865 21507 155 rp1adm 0.0/ 0.5 1846.9 0.0/ 0.1 74.8mb 1
dw.sapRP1_D 15161 21507 155 rp1adm 0.0/ 1.0 3612.5 0.0/ 0.1 103.7mb 1
dw.sapRP1_D 21680 21507 155 rp1adm 0.0/ 0.0 9.6 0.0/ 0.0 31.8mb 1
dw.sapRP1_D 21655 21507 155 rp1adm 0.0/ 0.1 6329.9 0.0/ 0.0 31.8mb 1
dw.sapRP1_D 21653 21507 155 rp1adm 0.0/ 0.4 22546.0 0.0/ 0.1 31.9mb 1

Because of this I am not able to execute some of the commands as out of memory error is coming.

Please suggest what need to be done in this case.

Thanks a lot for your help !

Re: Out of Memory Error

Kapil Jha — Sun, 15 Jun 2008 21:39:29 GMT

Hello Kavita,
As of now what i can suggest is ask SAP team to shutdown the application and see if you have some free memory.
If this does not help , reboot seems to be the only option.

ALl the error your pasted is just because there is no memory for anything.
BR,
Kapil

Re: Out of Memory Error

Rajeev Shukla — Sun, 15 Jun 2008 22:11:58 GMT

Yes when the memory and swap are 100% utilized the system wont let you fork any new process and this usually happens if there is some memory leak in applications or application not properly configured which might have resulted in too many process eating up all the memory.
Best as suggested it to shutdown the SAP application and see if the system is stable.

Cheers
Rajeev

Re: Out of Memory Error

Dennis Handly — Mon, 16 Jun 2008 02:58:18 GMT

What does "swapinfo -tam" show?
You may have to add more device swap.

Re: Out of Memory Error

Anka — Mon, 16 Jun 2008 05:07:21 GMT

first determine which processes consume much memory
If it is normal for the processes/system consider adding more memory to the system
also add additional swap space, there should be swap space requirements for SAP, SAP should give such recommendations.
As already mentioned no more processes can be created because there is a lack of memory and at the process creation the reservation of memory/swap fails.
Try first with adding more swap space.
Check the buffer cache, may be the kernel tunable is with a high value and it can be lowered in order to free some spaceWhat is the OS version?

Re: Out of Memory Error

Rasheed Tamton — Mon, 16 Jun 2008 08:47:55 GMT

Hello Kavitha,

>dw.sapRP1_D 7184 21507 154 rp1adm 95.3/77.0 41451.5 3.9/ 1.9 1.18gb 1

What is this process 7184 doing. It is consumong 1.18gb - looks not normal comparing to your other SAP processes. If you can identify the process and if it is a normal user process, perhaps you can ask that user to stop the process. Continues the same method with your other big processes, if you still have problems. In that way, you can save a shutdown of the application.

Adding a device swap might help to solve the issue temporarily but then you will get more pageouts because your mem and swap usage is 100%.

As told just give the output of the swapinfo -tam

Regards.

Re: Out of Memory Error

Kavita Poonia — Mon, 16 Jun 2008 19:25:51 GMT

Thanks a lot Kapil, Rajeev, Dennis, Anka and Rasheed.

I have contacted SAP team and they recycled the process which was using 1.8 Gb. Now the memory is still 100 % utilized ans swap is 83%.

B3692A GlancePlus C.03.86.00 16:15:51 cntrepa1 9000/800 Current Avg High
----------------------------------------------------------------------------------------------------------------------------------------------
CPU Util SU | 3% 3% 3%
Disk Util F FV | 5% 5% 5%
Mem Util S SU UB B |100% 100% 100%
Swap Util U UR R | 83% 83% 83%
----------------------------------------------------------------------------------------------------------------------------------------------
GlancePlus Commands Menu

Output of swapinfo -tam is:

# swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 6035 2157 74% 0 - 1 /dev/vg00/lvol2
reserve - 2121 -2121
memory 2962 1066 1896 36%
total 11154 9222 1932 83% - 0 -

Please suggest what needs to be done now as memory is still 100% utilized.

Re: Out of Memory Error

Patrick Wallek — Mon, 16 Jun 2008 19:29:35 GMT

It appears that you need more memory.

Run vmstat and check the 'po' column. If the value is double digits or greater then you have a fairly serious memory shortage.

# vmstat 5 15

Re: Out of Memory Error

Kavita Poonia — Mon, 16 Jun 2008 19:49:51 GMT

Hi Patrick,

Thanks alot for this. Here is the output of vmstat command :

# vmstat 5 15
procs memory page faults cpu
r b w avm free re at pi po fr de sr in sy cs us sy id
1 1 0 447584 7083 98 43 14 8 3 0 44 1212 3981 241 4 1 94
1 1 0 447584 6656 804 432 20 0 0 0 0 1136 28552 346 5 7 88
1 0 0 447737 7048 599 241 6 0 0 0 0 1065 23262 246 4 5 92
1 0 0 447737 7048 206 86 2 0 0 0 0 1057 9674 147 0 1 99
1 0 0 209908 7032 67 27 2 0 0 0 0 1046 3372 122 0 0 100
1 0 0 209908 7032 33 15 1 0 0 0 0 1076 3200 123 0 0 99
1 0 0 401410 7032 11 4 0 0 0 0 0 1049 1213 105 0 0 100
1 0 0 401410 6869 21 8 11 0 0 0 0 1064 2886 145 1 2 97
1 0 0 401009 7032 90 12 6 0 0 0 0 1058 2434 142 0 0 100
1 0 0 401009 7032 29 3 2 0 0 0 0 1052 941 105 0 0 99
1 0 0 304623 7032 13 3 1 0 0 0 0 1060 1470 110 0 0 100
1 0 0 304623 7032 4 0 0 0 0 0 0 1036 749 98 1 0 99
1 0 0 492770 7032 5 2 0 0 0 0 0 1042 1169 94 0 0 100
1 0 0 492770 7032 1 0 0 0 0 0 0 1072 836 108 2 0 98
2 0 0 493447 7026 5 2 1 0 0 0 0 1158 1506 145 2 0 98

Re: Out of Memory Error

Bill Hassell — Mon, 16 Jun 2008 23:32:28 GMT

vmstat does not show a lot of page outs (po). However, running your swap space between 80-100% means you will have more problems. Start by doubling your swap space. Now you should not get an out-of-memory error.

BUT:

As you add more processes and/or SAP or Oracle are configured to use more RAM and they get busy, performance will be awful as pages have to be moved out to the swap area. If you are not concerned about performance, more swap will be fine. Otherwise, double or triple your RAM size and performance will significantly improve.

Re: Out of Memory Error

Rajeev Shukla — Tue, 17 Jun 2008 01:10:58 GMT

Hi Kavita,

If memory is utilized 100% and there aren't any swaping of process happening then its the best use of memory. But from capacity planning/management point of view and so that you dont run down into the same problem its suggested you increase the memory on the system and for atleast for now till more memory is installed increase the swap space by adding a new swap device.

Cheers
Rajeev

Re: Out of Memory Error

Rasheed Tamton — Tue, 17 Jun 2008 07:43:47 GMT

If SAP is really FINE TUNED WELL ENOUGH and your system is patched to latest, then adding more device swap is your immediate solution.

But as Bill and Rajeev already said above, it will really effect your system performance as SAP needs more resources. In that case, adding more physical memory is the only solution on the long run for you to avoid future performance issues on the system.

rgds.