Shared memory/library space exhaustion on HP-UX 11.0

[Neil Armstrong]

Hi,

We're running the following system:
- HP V2500, HP-UX 11.0, 8GB RAM, Sept2000 patches, lotsa disk
- Informix 7.31.UC5, 32-bit
- SHMEM_MAGIC fudge on the Informix executables to gain 2.75GB of ShM. Currently we're using 2,152,243,200 bytes of ShM for Informix (466MB res segment, 1GB virt segment, 600MB virt segment).

Lately we've been having problems running some 32-bit applications, when loading in shared libraries. It seems to only be affecting those executables that utilise Orbix (CORBA s/w) shared libs.

Here's the error:
/usr/lib/dld.sl: Call to mmap() failed - TEXT /opt/iona/lib/liborbix.3.0.1.aCC.A.03.13.1
/usr/lib/dld.sl: Not enough space
Abort(coredump)

Now, i'm pretty sure this is because of the SHMEM_MAGIC fudge, that we've run out of contiguous shared memory space to load the lib into. A reboot will fix the problem, however it does come back after a while.

Because of the lack of info about quadrants (and the SHMEM_MAGIC fudge), i'm forced to theorise about how it all works. This is a bit scary with management asking for explanations! ;o)

Looking at the 'memwin_stats' command, it shows 718 free in quad3. If this is in 4K blocks (which i'm assuming here), that would equal 2.9MB free - the Orbix libraries are ~4MB each. I was initially under the impression that shared libs were loaded into quad3, but nothing seems to be touching it (apart from Informix when we allocate ShM to it).

Am I right in assuming that quad4 is used first, before a library is loaded into quad3? If that is so, does quad4 fragment over time thereby dis-allowing large libraries to load (or in the case of these Orbix libs, not enough contiguous space in quad3 to "flow-over")?
Is there any way of viewing the usage of quad4? memwin_stats doesn't show anything, and I couldn't get anything meaningful from adb.

Also, the "white paper" that accompanies memwin_stats, etc states:
- "Only if the globally visible address range is completely filled are shared libraries mapped privately."

If this is true, I should never EVER see this mmap() error?

Sorry if this is long-winded, but any help would be most appreciated.

cheers,
Neil Armstrong
Vodafone New Zealand

[Stefan Farrelly]

Hi Neil,

Not often we get fellow Kiwis on the forum, nice the see them! (Im a Kiwi stuck in London for too long....)

A few things to check first;

1. You have 8 GB of RAM, how much device swap do you have ?
2. Is pseudo_swap on ? (swapinfo -mt returns a memory line)
3. The binaries your running, from chatr do they have q3 and q4 enabled or just one or the other ?

Cheers,

Stefan
HP OSD
HP UK

[Neil Armstrong]

Hi,

We're running with 8GB of dev swap (4 x 1GB + 1 x 4GB lvols), with pseudo-swap switched on.

Typical usage of swap shows on average 50-55% USED (~65% of pseudo USED), no dev swap in use. Glance shows just over 2GB of free memory.

Running a chatr across the Informix executables shows them as SHMEM_MAGIC execs, but with both Q3 and Q4 Private Data Spaces disabled (is this normal?). The client processes that error are shown as normal shared execs, also with Q3 and Q4 disabled.

I haven't had too much experience with chatr flags and like - the SHMEM_MAGIC 'fudge' was recommended by HP here as a medium-term workaround until we migrated to 64-bit Informix (which is underway, just a little way off). Unfortunately the HP consultants who worked with us on this have since left, and it's proving difficult to find information regarding this.

thanks!
neil armstrong
vodafone new zealand

[Stefan Farrelly]

Ni Neil,

Theres a really good question/reply from Bill Hassel (HP Guru) on this, have a read first;

http://www.dutchworks.nl/htbin/hpsysadmin?h=3&dn=58729&q=SHMEM%5fMAGIC&fh

It doesnt specifically address your problem but it certainly helps explain the situation. Now, in your case its possible maxssiz is set to high which is not leaving enough space in q2 for these libs to load, causing your malloc error. Try reducing it (description of this was in the link above). Libs by default should go to q1/q2, not q3, unless these libs have q3 enabled ? (check with chatr on them, if so turn it off). You could try doing a chatr +q3p enable on these libs but as you said theres not enough free mem in q3 for them anyway!. You could also try chatr +q4p enable, if there is sufficient memory available in q4. But I would try reducing maxssiz first.

Cheers,

Stefan

[Neil Armstrong]

Hi,

Thanks for the link...but it's opened up a can of sorts...

Bill (and the HP Memwin white paper) both state that on a system with only the Global window, shared libs are loaded into Q4, then Q3 before failing, not into Q1/Q2...what is the reality here? Is there any documentation that explains how and where processes portions load into each quadrant?

Also, since SHARED_MAGIC execs use Q3 for memory windows, and SHMEM_MAGIC execs use both Q2 and Q3, could I chatr my offending libraries as SHMEM_MAGIC in order to utilise Q2 as well?

And finally, is there a command that can show the utilisation of Q1 and Q4 (Q4 especially)?

cheers,
neil armstrong
vodafone new zealand

[Stefan Farrelly]

Hi Neil,

Ive emailed Bill Hassell direct to ask if he can help. Lets hope he can.

Im off for a weeks holiday.

Cheers,

Stefan

[Rita C Workman]

Well this is out of my realm, but I figured I'd do some detective work at least...and I came up with the following:
First:
Some information about the quadrants, (you mentioned you were in ?? on this)..also mentions some necessary patches:
http://www.sybase.com/detail/1,3693,1010341,00.html
Second:
Found this reference where someone had similar error, possibly what is mentioned about the libraries not being privately mapped...:
http://devresource.hp.com/CXX/Forums/cxx-dev/cxx-dev.0104/0073.html

Anyway, hope it helps,
Rita

[Stefan Farrelly]

Hi Neil,

back from holiday and I have had a response from Bill Hassell;

Sorry, the 32 bit quadrant tricks are buiried in a lof C code and
chatr or compiler/maike options. Couple that with shared memory fragmentation and trying to get big RAM chunks in a 32 bit program needs better expertise than mine. THe person who can clear this up quickly is Dennis Handly in Cupertino.

--
Bill Hassell
Remote Engineering Services
HP Atlanta Response Center


If you are still looking to resolve this I can chase up Dennis for some help ?

Cheers,

Stefan

[Tim D Fulford]

I'm in two minds about replying to this!! I used to work for another telco who had the same problem as you (same OS, & DBase). When I was there we went through the exactly same thing! I may end up working for them again!

I will not reply explicity, so feel free to not to give points

1/ Speak to Informix UK. The've done it before.

2/ Think about at when informix starts & how best to fill up Q1,Q2... using tayloerd informix params < Only a hint here I'm afraid :-( >.

I'm also assuming that you will be doing this for a short while before moving onto Informix 9.?.FC? Once you do this then the shared memory fudges go away!!

Tim

[Neil Armstrong]

Tim, you've hit the nail on the head with regards to massaging the Informix segments.

I've spent a truckload of time looking into this, and with some help from HP i've figured the whole mess out.

It seems that the problem (while obviously a side-effect from SHMEM_MAGIC) is caused directly by fragmentation in Quadrant 4 - shared libraries need contiguous space to load, regardless of how much "free" space is actually in the Quadrant.

From some other initiatives to reduce the shared memory usage by Informix, we've been able to reduce the size of the 2nd Virtual segment that Informix allocates, and as such i've been able to load it completely into Quadrant 2, thereby leaving Quadrant 4 completely free.

Problem solved! :o) As long as our demands on the database don't necessitate more shared memory to cope, we should be plain sailing until our upgrade to Informix 9.21 64-bit.

Thanks for everyone's help!