Semaphore Kernel parameter

[vinay_26]

I have recieved an error on the HP Box which says " OUT OF RANGE FOR ALARM OF GLOBAL PARAMETERS KERSemOps TRIGGERED ON KERNEL KERNEL. 163.27 > 100"

I understand this parameter is for out-of-range for number of semaphore operations per second. How to find the current parameter.

[Kwan Fong, Emile]

Hello Vinay,

Please specify the platform and the OS release.

[vinay_26]

Its HP UX 11.0

[Neil Wilson_2]

Vinay

I am guessing you might need to set shmmax to 25% of memory. Memory is defined as ram plus swap.

Make sure your swap equals a miniumum of 1.5x memory.

Increase shmseg incrementally during maintenance windows until the problem stops happening.

N.

[Chris Wilshaw · Accepted solution]

I'm guessing from the message format that this alert was raised via BMC Patrol.

This uses sar to obtain the stats for the KERSemOps parameter.

eg: from sar -m

00:00:00 msg/s sema/s
00:20:00 0.00 128.60
00:40:00 0.00 130.02
01:00:00 0.00 126.03
01:20:00 0.00 128.69
01:40:00 0.00 130.42
02:00:00 0.00 128.41
02:20:00 0.00 128.51
02:40:00 0.00 130.06

As such, it may not be an indication of a problem on your system (unless users are also reporting issues), it could just be that the Patrol threshold has been set too low (100).

Is this a one-off alert that has been raised, or is it something that's happening all the time?

[vinod_25]

hi vinay

The first 17-19 semaphores are always used by the system. To check which user
is taking up most of the semaphores, run:

# ipcs -s -a ---> check then NSEMS column.


There is a WTEC/GSE tool called seminfo that will display good information
about the semaphore resources on the system. It will also display the
PID and name of the processes using the semaphores. Contact the Response
Center for a copy of the shminfo tool.

You can also use the script below. It has been written and provided by a user
on the ITRC Forums to map semaphores to process IDs.

NOTE: Use this script with caution.


Script that will find the process consuming semaphores
************** Cut here *******************************************
#!/sbin/sh
#
# NOTE: This is an Unsupported script.
#
if [ `uname -r|grep 11 >/dev/null;echo $?` -eq 0 ]
then
/usr/contrib/bin/q4pxdb /stand/vmunix 2>&1|grep "already" >/dev/null
2>&1
if [ $? -ne 0 ]
then
echo "\n/usr/contrib/bin/q4pxdb says /stand/vmunix is not ready for
degbugging!"
echo "you need to run q4pxdb /stand/vmunix which will create a new vmunix "
echo "with debug mode on. Backup your current /stand/vmunix file first!!"
echo "Once you have done this you can rerun this script, dont forget to"
echo "replace your /stand/vmunix file after running this script.\n"
exit 1
fi
else
/usr/contrib/bin/q4pxdb -s status /stand/vmunix|grep "ready for debug"
>/dev/null 2>&1
if [ $? -ne 0 ]
then
echo "\n/usr/contrib/bin/q4pxdb says /stand/vmunix is not ready for
degbugging!"
echo "you need to run q4pxdb /stand/vmunix which will create a new vmunix "
echo "with debug mode on. Backup your current /stand/vmunix file first!!"
echo "Once you have done this you can rerun this script, dont forget to"
echo "replace your /stand/vmunix file after running this script.\n"
exit 1
fi
fi
let V=0
let LOOP=`ipcs -bs|wc -l`
let LOOP=$LOOP-3
ipcs -bs|head -1
echo "T ID KEY MODE OWNER GROUP NSEMS
PID\nSemaphores:"
ipcs -bs|tail -$LOOP|while read A B C D E F G
do
echo "$A $B $C $D $E $F $G \c"
ied q4 /stand/vmunix /dev/mem </tmp/tmp.$$ 2>/dev/null
load struct seminfo from &seminfo
load struct semid_ds from &sema count $LOOP
keep indexof == $V
load struct __sem from sem_base
print sempid
EOF
cat /tmp/tmp.$$|grep -v sem
let V=$V+1
done

Also,


Use q4 to troubleshoot this in order to discover which process
is consuming semaphores. Refer to the following example:

1. ipcs -bs
IPC status from /dev/kmem as of Tue Mar 3 15:59:05 1998
T ID KEY MODE OWNER GROUP NSEMS
Semaphores:
s 0 0x2f180002 --ra-ra-ra- root sys 6

(There are others in use but only one using six.)

2. ied q4 /stand/vmunix /dev/mem

(Find out how the kernel semaphore tables were built).
q4> load struct seminfo from &seminfo
loaded 1 struct seminfo as an array (stopped by max count)
q4> print -t
indexof 0
mapped 1
spaceof 0
addrof 5197536
physaddrof 5197536
realmode 0
semmap 2002
semmni 2000
semmns 30000
semmnu 400
semmsl 2048
semopm 500
semume 512
semusz 4112
semvmx 32767
semaem 16384

q4> load struct semid_ds from &sema count semmni
loaded 2000 struct semid_ds's as an array (stopped by maximum count).

(Only one should have a count of 6. You could use "> 1900" instead of
"== 6" )
q4> keep sem_nsems == 6
kept 1 of 2000 struct semid_ds's, discarded 1999

q4> load struct __sem from sem_base
loaded 1 struct __sem as an array (stopped by maximum count).
q4> print sempid
sempid
371

The sempid is "pid of last operation" so it will vary, but it will
give you a starting point for working out the set of processes
responsible for consuming the semaphores.

By following the suggested action, you uncovered the process which
consumes the semaphores in this case (samba application daemon).


Hope this sprayed some light to ur query...

Regards

Vinod K

[vinay_26]

Thanks Friends.

As Chris said it was an BMC Patrol Alert. I was keen to know more about semaphores.

Vinod Suggestions were really useful for getting to base of the problem and resolving it. Many thanks for your valuable suggestions.