Shared Memory Issue

[Ramil Castillo]

My Bank customer had an issue on their HPUX system where the BANCS application aborted abnormally. It was then shutdown . The next time/day when the application was trying to startup , one of the modules was complaining about a particular shared memory (semaphore ) segment which it was trying to access and failing. They tried to check the segment via 'ipcs' but nothing showed up. They even tried shutting down the application and then running 'ipcs' to see if there was any stray SHEM segment, but no matching segment ( the Bancs application would always try and access the same segment when trying to start ). They figured it could be because of the fact the the previous night the application had aborted abnormally and hence that particular SHEM segment might be stuck somewhere.

The only option seen was to reboot the host so that all shared memory was release and refreshed. The application started fine after the reboot. Is there any other way to do this other than rebooting the host?? Especially when the segment in question does not show up via 'ipcs' ??

[Steven E. Protter]

Shalom,

ipcs -m
ipcs -s

It is possible that the shared memory was taken off the table listing but not actually released.

There may be something in the /var/adm/syslog/syslog file on this problem.

SEP

[Peter Godron]

Ramil,
I assume the application did not report what segment is was trying to access when it failed. Otherwise you may have been able to use ipcrm -m / -M to clear. If the application is now running, may be worthwhile to log the segment use.

[Kent Ostby]

Ramil --

I have seen a similar thing with Oracle from time to time where we had to reboot because the shmem segment was not released.

What type of applicatin is accessing the shmem segment?

[Bill Hassell]

Very common problem, especially with old 32bit applications. There is a single memory map for all normal 32bit programs and they all share this map. But shared memory segments must be allocated as contiguous segments (no holes) so although there may be hundreds of megs available, no contiguous section of this map is large enough to accomodate the restarted program.

What probably happened is that the aborted program did not delloacate all the segments, or another program asked for a small segment which was assigned a location that reduced the largest contiguous memory area. In other words, the memory map is fragmented but because the map is shared, it cannot be reorganized. To see how the shared memory map is used, get a copy of shminfo:
ftp://hprc.external.hp.com/sysadmin/programs/shminfo

The ipcs utility shows the assigned segments but does not show the empty spaces. And the shared memory map has a number of other components that can significantly contribute to the fragmentation.

There are several solutions:

1. Using ipcs -bmop, find all applications that are still running and shut them down normally. Then start the problem app followed by any other programs that are needed.

2. Using shminfo, look for the largest areas of unused space and see if elements that border the segment can be released by the applications.

3. Move the applications to 64bit versions where addressing limits are essentially removed.


4. Reboot the server.

Another solution is to create memory windows for the 32bit applications. This requires patches for old 11.00 systems, and some startup scripting to ensure all related programs start in the same window. See:
http://docs.hp.com/en/943/memwn1_4.pdf