Out of memory Oracle

[Gerry Tully]

I'm running into a problem with 3 instances of Oracle using up memory recources. The total SGA of the three is approx. 1.37gb. The system is running 32 bit processing as opposed to 64 due to home grown applications restrictions. The system is an N-Class 440 with 6gb ram running HPUX11.0. The problem occurs late in the afternoon after most of the users are off. I've attached the output from ipcs -am and my kernel parameters. I was hoping someone could take a look and see if anything obvious jumps out. Thanks in advance.

[Ted Ellis_2]

don't see anything here that inspires a target solution.. can you post the error?

[Gerry Tully]

The error message they are getting while running jobs in Oracle is:
/usr/lib/dld.sl: Call to mmap() failed -TEXT /apps/mmcprodora/8.0.6/lib/librw60.sl
/usr/lib/dld.sl: Not enough space

[John Palmer]

This sounds as though you are running out of swap space. What does swapinfo -mt tell you?

Regards,
John

[Gerry Tully]

swapinfo -mt looks like this:

Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
dev 8000 0 8000 0% 0 - 1 /dev/vg00/lvol9
reserve - 2682 -2682
memory 4933 1542 3391 31%
total 13957 4224 9733 30% - 0 -

[John Palmer]

Well you've got plenty of swap space!

Another possibility is that the processes are hitting the maxdsiz kernel limit (maximum data segment size). Check to see what you've got it set to (the default is 64Mb) and check your processes with glance.

In order to increase maxdsiz, you'll have to generate a new kernel and reboot.

Regards,
John

[Gerry Tully]

John, I think maxdsize is set high enough, I don't see any apparent problems in Glance.

[Ted Ellis_2]

your maxdsize is only about 100 MB... we use about 320 MB...

Oracle may be able to give you some clean recomendations for your environment

[Gerry Tully]

Ted, I think my maxdsize is around 1 gig unless I calculated wrong.

[Ted Ellis_2]

oops

[Brian Crabtree]

Your maxssiz and shmmax appear to be rather small. maxssiz should be around 120m and shmmax should be around 1g or higher.

Also, you might want to look at your settings for sort_area_size and sort_area_retained_size. This could possibly cause problems of allocating memory to oracle that is not released by sessions closing. I dont think that this would cause what you are seeing, as it would most likely happen at a peak time, rather than a slower period.

Brian

[Sridhar Bhaskarla]

Hi Gerry,

Add up all the SEGSZs in ipcs command for non-root 32bit processes and see if it is under around 1.75GB. If the delta of 1.75GB and the sum of the used SEGSZs is less than the SGA of the other instance, you can't bring up the database. You will need to use memory windows.

-Sri

[Yogeeraj_1]

hi,

this should be a kernel parameter problem. I had a similar problem recently when i increased the number of processes in my init.ora.

The attached document helped me review these parameters.

Hope this helps!

Regards
yogeeraj

[Bill Hassell · Accepted solution]

Your problem is the most common problem there is with Oracle running 32 bits (get those home grown apps FIXED). The problem is that you have one and only one location for Oracle SGA's, and that is a chunk of memory roughly 1.75Gb in addressable area (using the executables you have).

The problem is that that same area is populated with shared libraries, memory mapped files and other items (including shared memory segments from other programs). This poor area is VERY easy to fragment...just stop a couple of instances of Oracle, run a small fbackup and then use kill -9 on fbackup while it is running, and now you'll never get both instances running again (out of memory). You could put 32Gb or RAM in the machine and it would be ignored. 32bits cripples large apps like Oracle. ipcs will not show you the fragmentation.

The fix is easy:

1. Tell your DBA's that they must reduce their SGA usage by about 50%. Users will take an enormous hit on performance but at least it will run.

2. Fix the apps so they can interface with a 64bit version of Oracle and you'll never have a memory addressing problem again (well, you will when you need more than 8,000 Gb of RAM).

3. Setup memory windows for the 32 bit apps. Essentially, you will be giving each Oracle instance a private shared memory window. NOTE: every app that will access the Oracle SGA must be started in this window.

Now item #3 is very attractive since each instance now has a contiguous area of up 1.75 Gb of SGA space (DBA's will love this). But you need all of the memory window patches and read the white paper in /usr/share/doc on how to setup these windows.

ipcs -bmop is not a very complete tool to see shared memory. Instead, get a copy of shminfo so you can see the entire 32 bit shared memory area: ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

[Sridhar Bhaskarla]

While no one could beat Bill's explanation, you seemed to care less about the answers that were given to you on the same day as of your post.

We would appreciate if you do not ignore the value of the time spent by others for you.

Thanks,
-Sri

[Prasanna Venkateswaran]

Hi,

I'm getting the same error..

$ sftp sampleserver@10.21.164.36
/usr/lib/dld.sl: Call to mmap() failed - TEXT /usr/lib/libkrb5.sl
/usr/lib/dld.sl: Not enough space
Abort(coredump)
$

but in my case I'm not using Oracle but i'm trying to connect to a server using sftp.

Thanks
Prasanna.

[Bill Hassell]

Most likely, you are out of swap space. TRun the command: swapinfo -tm

To see what is using all your virtual memory:

$ UNIX95= ps -efo vsz,ruser,pid,ppid,args | sort -rn | more