HP-UX 12 out of memory error.

[PRAMOD KUMAR_1]

Hello,
I want to know whether any OS parameters may produce the HP-UX 12 out of memory error or not?
Please refer the attachment to understand the problem we are facing. Expecting an early reply either here or to my Email-id :
pramod_penguin@yahoo.com

Regards
Pramod Kumar.

[Michael Tully]

Hi,

You are either out of memory or swap. The best way to check your current usage is by using 'glance' (if you have it installed. /opt/perf/bin/glance)

You can install it from your application CD if it is not installed. The CD contains a trial copy.

A simple work around would be to configure some additional swap space.

Regards
Michael

[Bill Hassell]

Out of memory has 4 possibilities (none of which has to do with physical RAM). HP-UX, like most flavors of Unix, are virtual memory systems so RAM is only a special case of memory. Swap space plus a portion of RAM constitutes the virtual memory area. For HP-UX, RAM and swap are handled in 2 ways with the kernel parameter swapmem_on controlling the behavior.

When set =0 the total virtual memory space is defined solely by swap space and if total swap space is less than RAM, you cannot use the extra RAM at all. In this case, swap must at least be the size of RAM, perhaps 2x or 3x if more processes are going to be run than will fit in RAM.

When set =1, then about 75% of RAM is added to the swap space and the total is now virtual memory. This is default (and recommended) setting.

Since your swapinfo values show only 9% of actual swap space used, virtual memory (swap space) is not your problem.

One kernel parameter that must be cheked is the runaway program fence: maxdsiz and maxdsiz_64. maxdsiz should be 1750 megs in size and I would set maxdsiz_64 to several gigabytes.

The next problem is identifying whether the error message refers to local program data memory or shared memory. The (cryptic) messages from the application are not clear. If you are out of program memory (local data) then most likely you are running a 32bit application that is asking for more than 940 megs. You'll need to relink or chatr the program to address up to 1750 megs of RAM. (the term EXEC_MAGIC defines the new addressability) That should clear up the problem although if the program needs more, it must be recompiled as a 64bit executable to eliminate all these addressability issues.

Another possibility is that the error 12 refers to shared memory and for a 32bit program, things get a lot more complicated. 32bit programs must all share a common memory space which can easily become fragmented by share libraries, memory mapped files and other shared memory programs. By carefully starting programs in a specific order, the shared memory area fragmentation can be minimized. 64bit programs do not have this limitation.

There are 2 solutions for shared memory fragmentation: convert to 64bit in the applications, or if you must stay with a 32bit app, install the memory windows patches and run the apps in their own memory window.

You need to look over the memory management and process management white papers in /usr/share/doc and if you have the memory window patches, you'll have a memory window white paper too.

[T G Manikandan]

What kernel bits you are running?
how much of shared_memory you have allocated?

If you have hpux 32bit then there is a limitation of 1.75GB(64 bit no limitation) for shared memory.

Also check your kernel parameters like

maxdsiz
maxdsiz_64
max_thread_proc
nk_thread
nflocks
maxfiles

if these parameters are very low with the application you are running you can face the problem.

Revert

[PRAMOD KUMAR_1]

Hello Michael,
Are you not checked the file I attached with this mail message? From the command 'swapinfo -mt' itself we can understand current usage of swap memory. Also I mentioned in the attached file that most of the time the memory used value is greater than 97%, while rising this error.

Thanks Mr. Bill Hassell. I checked your reply note and different possibilities for rising this type of error, you are mentioned in this note. In our kernal parameter, swapmem_on=1 . SO I don't think this is because of first possibility. Also If I checked with kmtune command, the swap parameter values is like :

Parameter value
SHMMAX 0X40000000
SHMSEG 120
SHMMNI 512
SHMEM 1
MAXDIZ 0X040000000
MAXDSIZ_64BIT 0X0000000080000000

And this error mostly rising when the people accessing Oracle9i database table from Java front end. So How we can tell this is a 32bit program error?

Expecting an early solution for the predefined error.

With Thanks
Pramod Kumar.

[T G Manikandan]

java_max_sessionspace_size=
java_soft_sessionspace_limit=

Also increase these values.
These values are low on the server.
you have just 50 and 30 MB respectively to these parameters.Try increasing them.

JAVA_MAX_SESSIONSPACE_SPACE by default is 4GB.

Try commenting this parameter and checking.
Also shared memory on the system is 1GB.
you can increase it further.


Thanks & Revert


Thanks

[PRAMOD KUMAR_1]

Hello Manikandan,
Our max_thread_proc value is 256 ie., default maximum value (64 -256)
nk_thread also 7184
nflocks is 4096. The default value is 200+sum of all database file for all instance on the server. We have only one instance and no. of files also less than current value.
Value of maxfiles also 60 (Default). So can you check any other possibility to rise this error?

With thanks
Pramod Kumar.

[T G Manikandan]

I am not sure about the design of the Oracle database.

your shared_pool_size and java_pool_size are very low.

you should even increase them.

So summary you should increase these parameters.

shmmax
shared_pool_size
java_pool_size
JAVA_SOFT_SESSIONSPACE_LIMIT
JAVA_MAX_SESSIONSPACE_SIZE

http://gwynne.cs.ualberta.ca/~oracle/817doc/java.817/a83728/perf2.htm


Thanks

[PRAMOD KUMAR_1]

Mr.Manikandan,
Without knowing our database settings how you confirmed our settings are very low. Please check our init.ora settings below :

*.aq_tm_processes=1
*.background_dump_dest='/u01/app/oracle/admin/PRD/bdump'
*.compatible='9.2.0.0.0'
*.control_files='/u02/oradata/PRD/control01.ctl','/u02/oradata/PRD/control02.ctl','/u02/oradata/PRD/control03.ctl'
*.core_dump_dest='/u01/app/oracle/admin/PRD/cdump'
*.db_block_size=8192
*.db_cache_size=157286400
*.db_domain=''
*.db_file_multiblock_read_count=16
*.db_name='PRD'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=PRDXDB)'
*.fast_start_mttr_target=300
*.hash_join_enabled=TRUE
*.instance_name='PRD'
*.java_pool_size=52428800
*.java_max_sessionspace_size=52428800
*.java_soft_sessionspace_limit=30720000
*.job_queue_processes=10
*.large_pool_size=16777216
*.open_cursors=600
*.pga_aggregate_target=25165824
*.processes=150
*.query_rewrite_enabled='FALSE'
*.shared_pool_size=117440512
*.shared_pool_reserved_size=10485760
*.sort_area_size=524288
*.star_transformation_enabled='FALSE'
*.timed_statistics=TRUE
*.undo_management='AUTO'
*.undo_retention=10000
*.undo_tablespace='UNDO1'
*.user_dump_dest='/u01/app/oracle/admin/PRD/udump'
*.log_archive_start=true
*.log_archive_dest=/u02/arch
*.log_archive_max_processes=10
*.log_archive_format="%t_%s.dbf"
*.remote_login_passwordfile=exclusive


db_cache_size is 157286400 and java_pool_size=52428800,java_max_sessionspace_size=52428800,java_soft_sessionspace_limit=30720000.

This is not enough ? Please give a reply. Our DB_Cache hit ratio is >90%. So I think it is enough for our current process.

Please give a reply.


Regards
Pramod Kumar.

[T G Manikandan]

I am not confirming things,
I am just pointing that the errors come from here and you should filter the cause.

The out of memory errors come up for several reasons


1.First check your memory and swap usage on the system.
if it is very high check for the memory usage by each process and try to bring them to a minimum.
2.If step 1 is ok then check your shared memory on your kernel.Also make sure that the buffer cache paramters are ok.
Also check whether your application does not touch the limits of
max_files
max number of open files

max_thread_proc
max number of thread per each process.

The reason I was telling you was the reasons for which these errors pop up.

I could see that swapinfo command could give you a full utilization of the memory.

Are you running a server with 1GB memory.
Also your shared memory is set to 1GB.
What are the values of dbc_max_pct and dbc_min_pct?
These are the buffer cache parameters.


Also run

UNIX95= ps -e -o ruser,pid,vsz=Kbytes|more

during the time when you see a full memory usage by the server.


Thanks

[T G Manikandan]

There is more usage of memory.

If your memory size is 1GB
you can reduce the dbc_max_pct to 20 and min_pct to 10.
These paramters define the buffer cache on the system.
By default they are half the memory on the system.
Bring them down to the specified values.

Use glance to find out which all processes are using more memory.
Also you can use

UNIX95= ps -e -o ruser,pid,vsz=Kbytes|more

to find the process and their memory usage.

[Wodisch]

Hi Pramod,

just having a short glance at your details, I would like to mention a few points:
- error 12 on HPUX is "not enough core", which means not enough virtual memory (permitted to a single process)
- Oracle9i is available in 64bit versions only, AFAIK, so you won't have 32bit problems on the server side, but on the Java-Clients (I don't believe there exists a Java runtime with 64bit), and the error message in your attachments shows that it's a Java runtime message
- maxfiles=60 is way too small on an Oracle-server
- max_thread_proc=256 is too small
- you haven't mentioned your maxssiz (stack size), so is it on the default of 8MB? Increase it to something like 32MB (depending on your HW 79MB might be your maximum)
- have you checked the output of "ulimit" fcor the user starteing the Java clients? Any restrictions there?

additional notes (Oracle9i is intended for huge databases, though your setup seems not to be one - why would you use 9i then?):
- nkthread=7184 is too small
- 117M of shared_pool is pretty scarce (even for Oracle8i)
- 1GB of RAM is totally unadequate for Oracle9i
servers
- 0.5MB of a sort_area_size is way too small
- 157MB of db_cache_size is sooo small that your instance will crawl

FWIW,
Wodisch

[PRAMOD KUMAR_1]

Mr.Wodisch,
Thank you for your suggestions. As per your confirmation, you are mentioned in this forum that the 'HP-UX 12 out of memory error' is not because of Oracle9i on server instead due to 32bit Java-clients. Some of the parameters like maxfiles=60 is very small. But this is the default one. Also you are telling that max_thread_proc=256 also very small, So How we can varry the default range 64-256?
Our 'maxssiz' parameter value is 134217728bytes and 'maxssiz_64bit'also 1073741824 bytes, this is the maximum value permitted.
nkthread value of ours is 7184. How I can increase this value because this value I got from the equation '(((NPROC*7)/4)+16).
If I execute 'ulimit' command the output is 'unlimited'. So can you clarify your results for getting me better solution?

Regards
Pramod Kumar

[PRAMOD KUMAR_1]

Thanks Mr.Manikandan.
After I executing the command,UNIX95= ps -e -o ruser,pid,vsz=Kbytes|more , I am getting the RUSER,PID and Kbytes. All Oracle process have minimum 7000Kbytes. So Is there any problem? If it is a problem, how can I solve this?

Regards
Pramod Kumar.

[T G Manikandan]

Do you have a dedicated server model of the database.
If that is the case then each Oracle process spawns up a seperate process which will utilize some specified MB of memory.Even each client connection will spawn a new process.

With the new versions of Oracle each process takes around 30MB of memory.

If you are in shortage of memory I would like you to think of a multithreaded type of database (MTS) instead of dedicated model.

Did you bring down your buffer cache.
Any improvement you could find out?

Revert


Thanks

[PRAMOD KUMAR_1]

Hello Mr.Manikandan,
We have only one database in our HP/UX Server. Just now I change the dbc_max_pct parameter value as 20 and pct_min_pct as 10. I will check the difference and get back to you.

Regards
Pramod Kumar.