question on shared memory and IPC

[Todd McDaniel_1]

My client is asking about shared memory and IPCs in a 32bit environment... which I know very little about...

any help to answer his question is greatly appreciated.

--------------------------------------------------
QUOTE:

Do you know much about IPC (Inter-Process Communication) on UNIX? Specifically about the shared memory aspects? I've added some shared memory capability to UPSE's application and, though we are now 64-bit, I am still seeing some of the 32-bit limitations springing up. For instance, the maximum size I can get for a shared memory segment is still 1GB. I believe this is set by the SHMMAX setting in the /etc/conf/master.d/sysv-shm file. Additionally, I think there is still a constraint on the total amount of shared memory on the system that appears to be in the 4GB range.

With 64-bit, I'm under the assumption that all of these memory constraints should be gone. Do you know if any further updates would need to be made in order to expand the shared memory pool for our processes? In my research (there seems to be very little about IPC out there), it sounds like making SHMMAX too big can confuse 32-bit Oracle instances. Do you think we could try setting SHMMAX up to, say, 2 GB (0x80000000 instead of 0x40000000) and see what happens? Would that require a reboot of the system or is that a dynamic change?


UNQuote...
--------------------------------------------------

[A. Clay Stephenson]

If the client is a 32-bit application then the 32-bit restrictions apply. If the shared memory segment is created with a 64-bit application but accessed via a 32-bit application, that can be made to work but you must assert IPC_SHARE32 when the segment is created. Note hoever, that even in this scenario the 32-bit restrictions apply. Man shmget for details.
shmmax is dynamic so you can increase it "on the fly". One thing to lok at is memory windows so that each group of related 32-bit processes gets its own 4GB VAS rather than sharing one with all 32-bit applications.

[Jeff Schussele · Accepted solution]

Hi Todd,

As I'm sure you're aware that even if the OS is 64-bit, if the app is 32-bit you'll still be subject to the 32-bit rule of 4 quadrants - 1 GB each.
I assume the app is SHARED_MAGIC linked which would limit you to 1GB in the 3rd quadrant.
If you can relink it to SHMEM_MAGIC you could add .75 GB from the 2nd quad & have 1.75 GB now available and if linked properly the absolute max would be 2.75 GB with SHMEM_MAGIC and memory windows.
Increasing SHMMAX will have no effect unless the binary is linked properly to take advantage of that space.
If you have an 11.0 system look in /usr/share/doc for a memory windows whitepaper named mem_wndws.txt

Here's a few memory management whitepapers:

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000075724948

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000062686726

http://www2.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000009254450

NOTE: You may need a system handle tied to your ITRC account to get to these.

HTH,
Jeff

[Raj D.]

Ho Todd ,

1. shmmax is a dynamic parameter and can be change on the fly ,

To check current shared memory defined
# kmtune -q shmmax

To see usage:
# ipcs -m

2. Max size ,
hpux 10 and 11 suport max 1GB ( 0x40000000)
Default is 64MB (0x04000000)
Not sure for 11i

to set 2GB should be( 0x80000000)


Check this link for further:
http://docs.hp.com/en/J2716-90017/ch06s05.html

http://asktom.oracle.com/pls/ask/f?p=4950:8:::::F4950_P8_DISPLAYID:454420165038
------------------------------------------

Cheers ,
Raj.

[Todd McDaniel_1]

okay thanks for the quick responses...

The app was recently upgraded to 64bit but I am not sure if any aspect of it is still running at 32bit.

My question, I guess, is if they are still accessing at 32bit level... will it make it worse if I increase shmmax above the 32bit limit?

he mentions something about it could harm oracle dbs...

[Jeff Schussele]

No - I don't think it will.
I've never heard of anything like that.

Run the file command against all the executables for this app. If they *all* return ELF-64 then the *are* 64-bit. If they don't - they aren't.

Rgds,
Jeff

[Todd McDaniel_1]

They won't be using shared memory for older 32bit app... only on the 64bit version.

Also, I think I found out that 32bit won't have issues till shmmax is around 4gb.

We are only going to 2gb for now which is fine for what they need at the moment.

[Bill Hassell]

Shared memory and local data in 32 bit apps are subject to similar limitations. The difference is that you can have a bunch of 32 bit programs use the max available (960Mb for std executables, 1.75Gb for EXEC_MAGIC, 2.75Gb for +q3p programs and 3.75Gb for +q4p programs). Shared memory for 32bits is much more limited because there is only one map while local data in each program has it's own map. So if your developers are hitting a memory limit, simply ask them to run the "file" command on the program that is having a problem. If it says "ELF-64" then it is 64bits and the shared memory limit is several TERABYTES.

But if the file cfommand says "PA-RISC..." then the program is 32bits. Note also that 64bit and 32bit programs require a *LOT* of care (read: rewrite) in order to properly share data. The reason is that 64bit programs default to 64bit data elements while 32bit programs default to 32bit elements. Mixing 32 and 64 using a common shared memory segment will require extensive testing.

The shared memory pool does not expand or contract. It's always at the maximum value, whether 32 or 64. But 32bit programs CONSTANTLY run out of memory because there is just one map, shared by all 32bit programs. More importantly, this area can become fragmented and when a program needs 100megs, but there are a dozen 75meg areas, the program will fail because each segment must be contiguous. ipcs -bmop is useless to tell you about fragmentation and true availability of segment space. Get a copy of shminfo from ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

shminfo will answer all the questions about shared memory problems, and also demonstrate the need for 32bit memory windows so that shared memory can be mapped exclusively.