Shared memory problem

[Mark S Meadows]

Hi Forum,

We are encountering the following problem with an Oracle Application, which we are in need of assistance with and would appreciate any assistance/pointers with regard to what is actually causing the restriction.

Problem : Whenever the number of concurrent Oracle jobs generated by an instance requires shared memory of more than 1400 MB in total at the same time we are encountering the error "unable to shmget gather refdata, errno = 12" , the concerned system call is shmget and the concerned error code to errno 12 is :- #define ENOMEM 12 /* Not enough core */

Server details :

Model rp4440
HP-UX B.11.11 64 bit
8 processors
16Gb Physical Memory
18Gb swap space allocated

swapinfo -tam
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 8192 0 8192 0% 0 - 1 /dev/vg00/lvol2
dev 10240 0 10240 0% 0 - 2 /dev/vg00/swap2
reserve - 3994 -3994
memory 11480 1826 9654 16%
total 29912 5820 24092 19% - 0 -
Kernel parameters :
STRMSGSZ 65535
dbc_max_pct 10
dbc_min_pct 10
dnlc_hash_locks 512
max_thread_proc 256
maxdsiz 3221224472
maxdsiz_64bit 21474883648
maxfiles 16384
maxfiles_lim 16384
maxssiz 401603608
maxssiz_64bit 1073741824
maxswapchunks 16384
maxtsiz 0x10000000
maxtsiz_64bit 0x40000000
maxuprc 10098
maxusers 1400
maxvgs 99
msgmax 65535
msgmnb 65535
msgmni (NPROC)
msgseg 32767
msgssz 8192
msgtql (NPROC)
ncallout (NKTHREAD+2)
ncsize ((8*NPROC+2048)+VX_NCSIZE)
nfile (15*NPROC+2048)
nflocks 8192
ninode (NFILE+50)
nstrpty 60
semmni 4096
semmns (SEMMNI*2)
semmnu (NPROC-4)
semume 1000
semvmx 32768
shmmax 8589934592
shmmni 512
shmseg 256
st_san_safe 1
vps_ceiling 64

The system recently had 8Gb of additional Ram and a 10Gb swap device (swap2)added.

The problem is no doubt to do with hitting a kernel limit, maxdsiz (or maxdsiz_64bit) or more swap space required, but would like to know where the 1400MB restriction is if possible.

Please let me know if further information is required.

Thanking you in anticipation

[Mark S Meadows]

Additional information :

Oracle version 9.2.0

[Hein van den Heuvel]

At first glance your params look fine.
Should be no issue to create 1400mb, uless you need many many (250+) small sections.

The OS is 64 bit enabled, but is your Oracle intall?
Is it using $ORACLE_HOME/lib64 or $ORACLE_HOME/lib ?

Anyway... please help us(me!) understand the context slightly better, as there are some confusing issues.

Specifically, the 'normal' Oracle shared memory area (SGA) is NOT dependend on the number of number of jobs and really only shows issues during Oracle startup.

So...

From what program/logfile do you get that message?
Anything in bdump/alert.log to help?
Any udump/*.trc file to give more detail?

How did you count/see that 1400 mb limit?
What tool? ipcs -m ?

Can you do an ipcs well before the failure point, and maybe one more just before? Stick both outputs in a text file and diff?
If needed, attach output as a text file to a future reply?

What does the oracle maxmem tool report?
Or even better, if you have SAP:
/usr/sap//SYS/exe/run/memlimits | more

Hope this helps some,
Hein van den Heuvel (at gmail dot com)
HvdH Performance Consulting

[Michael Steele_2]

This is going to be a problem related to how your Oracle SGA is configured. And looking for answers in the O/S or the HW will be very limited.

Shared memory is loaded once and then shared from parent to children on a as needed basis. It doesn't created 1400 MB every time there's are request. Its loaded once and then reread over and over.

Paste in the report from this command when you have the problem:

ipcs -moba

And note the NATTACH column for non-root owners and a zero value.

[Mark S Meadows]

Many thanks for your replies so far.

The Oracle install is 32 bit :

ipcscs1:I1 > pwd
/u01/app/oracle/product/9.2.0/lib

I have passed the information that you have requested onto our DBA's and will get back to you asap.

Thanks

[Mark S Meadows]

Hein,

Hopefully some answers to your questions from our DBA's, which might provide a pointer to the problem.

**The Operating system is 64 bit enabled, but the Oracle install being used is 32 bit ? **

Oracle install 64-bit

sqlplus /nolog

SQL*Plus: Release 9.2.0.7.0 - Production on Mon Jan 28 15:11:29 2008

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

SQL> connect /as sysdba
Connected.
SQL> exit
Disconnected from Oracle9i Enterprise Edition Release 9.2.0.7.0 - 64bit Production
With the Partitioning option
JServer Release 9.2.0.7.0 - Production


** From what program/logfile do you get that message ? **

When we try to allocate the shared memory using shmget library call, If we try to go beyond limits(which is 1GB as the max stack size is defined) or if we try to up our two applications simultaneously, shmget call throws error no 12 documented in /usr/bin/include/sys/errno.h with below listed details:

#define ENOMEM 12 /* Not enough core */

** Anything in any log files to help or give more detail ? **

This is our application define log file.
How did you count/see the 1400 mb limit?

We have following three applications on this box which are using following amount of shared memory:

PSTN - 800 mb
BMOB - 650 mb
AVALON - 700 mb

We can not up any two of the above applications simultaneously.
We have also tested that we are able to load the chunks of 500mb, 500mb, 200mb and 100mb successfully, after that application throws error no 12 describe above.

** What tool did you use ? ipcs -m ? **

Yes for showing a loaded shared memory of any particular user we use ipcs utility.

** What does the oracle maxmem tool report? **

No idea what this is

[Michael Steele_2 · Accepted solution]

This is one way that is very reliable. You want to note the VSS parameter and any other PS command parameter like PID and PPID to track this down. But this won't work unless you put it into a 15 minute cron script and cat the reports out to an output file.

What you seem to be describing is not a shared memory problem but a memory leak with your application. And by building this 15 min cron script you'll be able to track the growth of the culprit upto the point that it kills your box.

Good luck Mon Cher Amis.


UNIX95=1 ps -e -o vsz,rss,pid,ppid,comm | sort -rn | head
UNIX95=1 ps -e -o pcpu,vsz,rss,pid,ppid,comm | sort -rn | head
UNIX95=1 ps -e -o time,pcpu,vsz,rss,pid,ppid,comm | sort -rn | head
UNIX95=1 ps -e -o etime,time,pcpu,vsz,rss,pid,ppid,comm | sort -rn | head

vsz - total size in kb of the process in virtual memory - Memory leak indicator
pcpu - cpu time used recently / cpu time available during same period
etime - elapsed time since process start
rss - resident set size in kb - not as accurate as pmap
time - cumulative cpu time of process

[Hein van den Heuvel]

Hein,

>> Hopefully some answers to your questions from our DBA's
Good.

>> Oracle install 64-bit
Edition Release 9.2.0.7.0 - 64bit Production
Good. That's the right place to find it.
[Sorry for suggesting to look as the libraries.]


Hein>** From what program/logfile do you get that message ? **
> When we try to allocate the shared memory using shmget library call,

Ah! We suspected as much. So this does not really have anything to do with Oracle. Sure the application also uses Oracle, but it is an appliaction issue. If there is any Oracle component then it could be a bas (32-bit) client library, or excessive system resource consumption by the Oracle server, but that there is currently no indication for that.
You DBA may want to do SQL> SHOW SGA to confirm, or the SA can look at the ipcs -m details for the Oracle shm in use.

>> if we try to up our two applications simultaneously, shmget call throws error no 12 documented

If 2 independent programs can do this, where supposedly each can run cussesfully alon, but not at the same time, then the problem area shifts froOf course it would be preferrable/better to use fewer but larger shm sections.
mt the more common per-process Virtual address space usage into thte realm of physical memory, and swapspace.

Now in the opening words you mentioned 11.11. What patches applied?

Is that memory allocated in many smaller chunks or in one go? It woudl be preferrable to allocate fewer btu larger sections. For many shm chunks you may need ShmemExtensions and a patch PHKL_30196.
See http://docs.hp.com/en/B2355-60127/shmmni.5.html
And http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=968655

Hein>> ** What does the oracle maxmem tool report? **
> No idea what this is
That's an undocumented tool in the $ORACLE_HOME/bin directory.
Just run it! :-)

You may prefer to check out:
ftp://hprc.external.hp.com/sysadmin/programs/shminfo/

Hope this helps some more,
Hein.

[Don Morris_1]

It isn't shmmni (running out of segments gives ENOSPC from shmget()), or shmmax [EINVAL] or shmseg [doesn't affect shmget(), only shmat()].

It isn't any of the max{t,d,s}siz tunables either (Sys V shmem isn't affected by them).

It doesn't sound like physical memory or swap exhaustion (or lockable) since as you describe it, there should be a lot more than 1.4Gb available, and you'd be less consistent with failing on shmget() [and I assume you'd check the obvious anyway].

So I have to conclude there are 32-bit applications talking with 64-bit Oracle and you're simply exhausting the default Memory Window's shared address space (all shared objects by default come from there, and on 11.11 there's only 1.75Gb in there total... the other 350Mb is likely in shared libraries or shared memory mapped files, etc.). Or the apps are 64-bit but the call to shmget() uses IPC_SHARE32 (in which case, don't unless you really need to share with 32-bit apps as well -- then you want to limit the sharing to segments that fit).

Either run the applications in different Memory Windows (if possible and if they don't need each other's segments) or [better if possible] go fully 64-bit -- the VA limits there are terabytes and you would run out of swap/memory swap first.

[Mark S Meadows]

Many thanks for all the feed back and advice, which I will pass onto our DBA's for progressing.

Will come back with an update asap and allocate points.

Regards,