Shared memory voodoo

[John Poff]

Hi everybody,

Here is a good one that I just had to share.

We've just been through a long weekend [17 hours on Saturday]
moving a bunch of data from our IBM Shark to our new
HP XP1024 array. Part of this move included essentially
renaming one server as another one and moving a large
application from the first server to the new one. Well,
this morning we try bringing up this application [a test instance]
and it won't go. After beating on it for a while and comparing
it to the similar production instance that is running fine, I figure
out that this application should open five shared memory segments but
is only opening three when it croaks. Our guy gets on the phone with
the vendor and he tells us we need to reboot the system. He says we
don't have enough shared memory for all the applications we are running.
Funny thing is, this is an rp8400 with 24Gb of RAM, and we are only using
about a third of the memory. No matter, he says we should still reboot.
We trade e-mails, and then he figures out that our problem is because we
have copied the application from another system. He believes that when
we copied the files over from the other system, we "copied the shared
memory files over" from the other system as well and that is why the
shared memory segments can't be opened. "None of the process IDs would
match up" which would cause "much confusion" on the new server.

Right. Sounds like shared memory fertilizer to me.

JP

[Jeff Schussele]

Hey JP,

Boy that's a ripe one.
I can smell it over here...

Cheers,
Jeff

P.S. Is that a 32-bit app?
If so, you may need to start using memory windows.....

[A. Clay Stephenson]

Well, I can think of one thing that would explain this. This application might actually have 'shared memory files' but they are not what you think. They MIGHT be used to identify the shmed Keys and ID to a group of processes. I admit it's a dumb way to write code but ...

The other (and more likely) explanation is that this is 32-bit code (possibly in its own window) and the memory was too fragmented to allow another large chunk to be allocated. I suspect that after you shutdown the application, you could have done some careful ipcrm's and cleaned up the mess. A better question is why didn't shutting down the application clean up the shmem? I would ask the vendor that question.

[Bill Hassell]

There have been many questions like this here in the forums. The issues are always: old-as-dirt 32bit applications that have never been ported to 64bit code. SImply put, 32bit programs are limited to an addressing space of 960megs (or 1750 megs with chatr or compile options). Additionally, *ALL* shared memory segments for *ALL 32bit* applications must use a single memory map, about 1750 megs in size.

Not only must all 32bit applications share the same memory map, but shared libraries and memory mapped files (and other IPC items) which fragment this limited resource (4gb or 24Gb of RAM makes no difference...that's why 64bit opsystem and apps was created for HP-UX way back in 1997). To see just how bad the shared memory area has become, get a copy of shminfo from: ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

Then have your vendor read a copy of the memory-management and process-management whites papers in /usr/share/doc (they do have an HP-UX box don't they?) If the vendor can't recompile and fix their code, then you'll have to implement memory windows where each instance of the application (and related programs) are run in private windows. In that case, each app will no longer be sharing their limited memory area). If you are running 11.0, make sure all memory-window patches are installed.

[Steven E. Protter]

JRF probaby hit it with the key files.

I've gotten similar results by copying over shared memory key files with a data move.

In essence I had one database with shared memory key 0x133333, and a second database that won't come up with the same shared memory key.

It was madenning because we'd done everything right except run change the key.

You could run the ipcs command and then scan the configuration files with a

find /conf -exec grep -l 'keyvalue' {} \;

Find two configuration files maybe they have a point.

Then again, maybe not.

SEP