HPUX11.0 Shared Memory Fragmentation

[Drew Mayo]

Hi there all, I've scanned most of the posts on ITRC and can't see an answer to my question, so I'm resorting to asking it directly :)

I'm work for an app vendor supporting an Oracle/Tuxedo app on HPUX 11.0 (800 series box etc. etc.)

Our system requires a large-ish (200-700Mb) single chunk of shared memory to use as an application data cache. Right now everything's running as 32-bit, meaning that we're kind of stuck using a maximum of 1.75Gb of SHM...

However, that's not the issue, the issue is that our shared memory can get pretty badly fragmented. we've got two Oracle instances with 1/2 Gb SGA's, plus our application and tux using SHM, and sometimes we just can't allocate our cache! HPUX reports us an error 12 (ENOMEM) on the shmget call, even when there's significant amounts of SHM actually free.

How can I identify which process or processes are responsible for fragmenting the SHM? how can I prevent it on an ongoing basis?

Our current proposed solutions is to move Oracle to 64-bit so that it doesn't use the 32-bit SHM segment, but in the shorter term (and from a general question point of view) I'd like to be able to identify SHM fragmentation, and ideally identify which processed do it, are doing it and how to STOP them doing it :)

I've 'proved' fragmentation using a simple C program where I attempt to get a block of X Mb, then if that fails, I try two blocks of X/2, then 4 blocks of X/4 etc up to the point where it hits the shmseg limit. According to my hazy interpretation, on a static system where an allocation of 512Mb fails, but 4 allocations of 128Mb succeed, the memory is fragmented.

I know HP Omniback II is a SHM user, and unfortunately our downtime on things like Oracle and our app tend to happen while backups are running, so that's my gut feeling/suspicion, but if glance, or any undocumented API call can give me an answer then I'll be a happy happy camper.

Thanks in Advance.

[Don Morris_1]

Hope this helps:

ipcs -mpt

should give you the owner, group, cpid, and lpid for every segment as well as the time of creation.

Hopefully that will be enough.

[Don Morris_1]

Ah... missed a flag.

Make that ipcs -bmpt

so you get the segment size as well.

Unfortunately, I don't know anyway other than using q4 on the kernel to find out the virtual addresses things are mapped to (if you *can* use q4 on the kernel, let me know) - but the size information and PID should give you a start, I'd hope.

[Bill Hassell · Accepted solution]

As mentioned, ipcs does help (I use ipcs -bmop which shows creator+lastaccessPID). However, for everyone using Oracle, Sybase, Informix, or any other big shared-memory users, here is the tool:

shminfo

You can get a copy at:

ftp://contrib:9unsupp8@hprc.external.hp.com/sysadmin/programs/shminfo/

Here's selected portions:

Global 32-bit shared quadrants:
===============================
Space Start End Kbytes Usage
Q4 0x00f33c00.0xc0000000-0xc0008fff 36 OTHER
Q4 0x00f33c00.0xc0009000-0xc0009fff 4 SHMEM id=0
Q4 0x00f33c00.0xc000a000-0xc000bfff 8 SHMEM id=2
Q4 0x00f33c00.0xc000c000-0xc0056fff 300 OTHER
Q4 0x00f33c00.0xc0057000-0xc005efff 32 SHMEM id=1 locked
Q4 0x00f33c00.0xc005f000-0xc00cefff 448 OTHER
Q4 0x00f33c00.0xc00cf000-0xc00cffff 4 FREE
Q4 0x00f33c00.0xc00d0000-0xc00fbfff 176 OTHER



Private 32-bit shared quadrants:
================================

Shared space from Window id 0 (global):
Space Start End Kbytes Usage
Q2 0x0213e800.0x40000000-0x7fffffff 1048576 FREE
Q3 0x00f33c00.0x80000000-0xbfffffff 1048576 FREE

Shared space from Window id 1 (q3private):
Space Start End Kbytes Usage
Q2 0xffffffff.0x40000000-0x7fffffff 1048576 OTHER
Q3 0xffffffff.0x80000000-0xbfffffff 1048576 OTHER

Limits for 32-bit SHMEM allocation:
===================================

Maximum shmem segment: 65536 Kbytes (shmmax)
Largest free segment: 1048576 Kbytes (Window id 0 (global) quadrant 2)
Available swap : 852432 Kbytes (swapspc_cnt)
Available pseudo-swap: 248408 Kbytes (swapmem_cnt)

----
Along with an actual map of shared memory areas, this tool will also map memory windows when used. Along with shminfo, sourcecode for a test program called shmalloc.c which can be used to test shared memory calls.

It is a contributed program with all the requisite disclaimers about goodness and fitness and all that...

[Drew Mayo]

Thanks _heaps_ Bill, that tool is exactly what I wanted...

(strangely enough after browsing all questions related to SHM on the ITRC, I thought that you might be the one to answer it - heh)