URGENT: Informix out of virtual shared memory SYSTEM DOWN

[Craig A. Sharp]

We are having an issue with Informix 7.31 running out of virtual shared memory and crashing.

I have shmmax at 1 Gb and Informix is set to use 1 Gb share mem and increase as needed by 100 Kb chunks.

The system has 6 Gb of mem and 9 Gb of swap.

Here is the swapinfo:
# swapinfo -tm
Mb Mb Mb PCT START/ Mb
TYPE AVAIL USED FREE USED LIMIT RESERVE PRI NAME
dev 1024 0 1024 0% 0 - 1 /dev/vg00/lvol2
dev 8192 0 8192 0% 0 - 0 /dev/vg01/swap1
reserve - 2043 -2043
memory 4773 2534 2239 53%
total 13989 4577 9412 33% - 0 -


I am attaching my current kernel settings.

Any ideas?

Thanks,

Craig

[Mark Grant]

Shouldn't you have a higher value of shmmax if you are going to start off at 1GB and then increase?

[Craig A. Sharp]

Informix is 32 bit so shmmax can not be larger than 0x40000000 or 1Gb.

[Pete Randall]

Craig,

The "Machine Specific Notes for Informix-OnLine 7.31.FC4" recommend shmmax be set to 4294967296!


Pete

[A. Clay Stephenson]

If shmmax is set to 1GB and that is what Informix initially allocates then it can't increase above that value. You need to icrease the shmmax setting. Now for the big question: Is your Informix 32-bit or 64-bit? If 64 then increase shmmax and you are fine. If 32-bit then you need to look into memory windows and SHMEM_EXEC so that shared memory for a group of related processes can exceed 1GB.

[Michael Schulte zur Sur]

If Informix is 32bit should it not have 2^31-1=2gb-1 shmmax?

Michael

[Craig A. Sharp]

Informix is running 32 Bit. 7.31 FC4. Is that 64 bit?

[Craig A. Sharp]

Oh ya, we are at 7.31 UC5

[Mark Grant]

So it would seem. You are not using swap and you can not, in fact, create shared segments greater that the amount of available swap you have at the time, regardless of shmmax. I don't think the OS is going to be able to help you here.

[Pete Randall]

Craig,

Check the release notes in /opt/informix/release/en_us/0333/IDS_7.3 (or something similar to that) for the shmmax recommendation.


Pete

[Mark Grant]

Is the OS running 32 bit as well? If it is, 1GB is definately the max.

[Michael Schulte zur Sur]

Hi Mark,

he has two swap devices, totalling 9gb.

Michael

[Craig A. Sharp]

Correct, two swap devices. In monitoring the system, we are not swapping.

[Craig A. Sharp]

We are running on an N-Class. I can't remember how to check if the OS is running 32 or 64.

[Pete Randall]

Craig,

"getconf KERNEL_BITS"


Pete

[Craig A. Sharp]

64 Bit

[Steve Lewis]

Your SHMADD is too small, if it has to keep adding extra segments in 100kb chunks your system will get very slow. Add up all the V class segment sizes, as listed in
onstat -g seg
then set SHMVIRTSIZE to be the total of that, then set SHMADD to be at least 200Mb for your system. You may also run out of shmseg.
You could try setting shmtotal in your onconfig parameters. This will stop the instance but not the machine, if you run out.

I always thought that shmmax was the maximum size of a segment, not the total size of allocated shared memory for the system. According to the docs.hp.com Configurable Kernel Parameter Guide, it is a system-wide parameter for maximum shared memory segment size and a single call to shmget() cannot exceed this, but it doesn't say that you cannot have several items of this size. I presume that the total limit is the size of your swap space, or shmmni*shmmax.

If you have run out of RAM, you may have to reduce BUFFERS in order to increase SHMVIRTSIZE. If onstat -g seg shows just 1 block used in a segment then run onmode -F to free them up in the short term, then schedule a re-boot.

A large index build can eat up memory.
Also check for runaway processes that have eaten up memory without giving it back. It may be a patching issue, for instance mib2agt was an old chestnut, now fixed.

[A. Clay Stephenson]

Shmmax does indeed limit the maximum size of any one shared memory segment. It is not a global maximum. However, a 1GB limit instantly comes into play in the 32-bit world. If your application is 32-bit you are still bound by the 32-bit rules even in a 64-bit OS. The problem is that 32-bit address space is divided into 32-bit quadrants and all 32-bit applications use the same quadrant for shared memory; you can spread shared memory over 2 quadrants (upto 1.75 GB
by changing the executables to SHMEM_MAGIC). If you use memory windows then a process (or group of related processes) get their own 4GB address space which is not shared globally.

[Pete Randall]

Craig,

Let's clear up this 32/64bit confusion. Do a file on oninit:

file /opt/informix/bin/oninit
/opt/informix/bin/oninit: ELF-64 executable object file - PA-RISC 2.0 (LP64)



Pete

[Craig A. Sharp]

We got it! It turns out that the DBA has loaded two instances within the same 32 Bit Informix engine. This is causing both instances to try to use over 1.75 Gb of shared memory and as such, the engine is crashing when the memory is gone.

The fix is to run two distinct engines and each will have its own 1 Gb chunk of shared memory.

Thanks to all who replied. It is great to be involved in a community of people with such a level of concern and experience!

I will go back through and assign points.

Thanks,

Craig

[Michael Schulte zur Sur]

Hi Craig,

just a reminder. ;-)

greetings,

Michael