Operating System - HP-UX
1748106 Members
4817 Online
108758 Solutions
New Discussion юеВ

sema/s is over 2000 and pefromance issue with application

 
SOLVED
Go to solution
skt_skt
Honored Contributor

sema/s is over 2000 and pefromance issue with application

HP-UX 11.00
N4000

# sar -m 5 5

HP-UX eurleg1p B.11.00 A 9000/800 11/11/09

10:37:37 msg/s sema/s
10:37:42 1.00 1556.18
10:37:47 1.20 1433.80
10:37:52 0.80 1378.60
10:37:57 1.00 1641.20
10:38:02 1.20 2084.17

Average 1.04 1618.55

I have an which is ofetn going trhough performancer issue and my ssystem is below 50% used interms of CPU/MEM/Swap also the disk service time is good. I see a trend sema/a is very high (normal below 100) during problem window.

have you been though a similar sirtuation. is there a work around or a fix to increase performance to application
20 REPLIES 20
TTr
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

Semaphore activity is usually related to application activity. On the OS side you can control the number and size of semaphores. But if the application does not get what it needs it will error out.

Check the application recommendations and verify the semaphore settings in the kernel. If the semaphore size is too small the application may be opening too many of them. Adjusting the semaphore parameters may force the application to open fewer semaphores of larger size. There are other semaphore parameters in the kernel as well. Maybe the app has a bug maybe it is normal activity.

Use ipcs -a (ipcs -s) to monitor the semaphore usage and user/app ownership
skt_skt
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

I already have higher value semmns which controls the limit.(The number of System V semaphores configured for use is controlled by the kernel parameter SEMMNS.)
Under below list , not sure which one controls the "size"

# kmtune|grep -i sem
sema 1
semaem 16384
semmap (SEMMNI+2)
semmni 4096
semmns 16384
semmnu 1024
semmsl_override 2048
semume 64
semvmx 32767
TTr
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

Unless this secret application have any unusual recommendations for semaphore settings, the semaphore settings look ok.You need to verify with the application support if this is normal. Or check if there are any application settings or fixes/patches that may be addressing this issue? Even the 100sema/s under normal activity is kind of high. Unless this application is a large environment with amny different independent components.

Did you verify the semaphores and the owner with the "ipcs -s" command?
skt_skt
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

normal case is below , even though i said below 100.

08:23:34 msg/s sema/s
08:23:39 1.20 41.60
08:23:44 1.00 36.80
08:23:49 1.00 40.00
08:23:54 1.20 43.00
08:23:59 1.00 36.80

Average 1.08 39.64

I dont see a big list under "ipcs -s" even though almost 2000 sema/s. I see only around 50 lines in this o/p around problemwindow also.Not sure how t map them(is there a mapping?)


Here is the current one.

# ipcs -s
IPC status from /dev/kmem as of Thu Nov 12 08:31:05 2009
T ID KEY MODE OWNER GROUP
Semaphores:
s 0 0x411c0e87 --ra-ra-ra- root root
s 1 0x4e0c0002 --ra-ra-ra- root root
s 2 0x412000a3 --ra-ra-ra- root root
s 3 0x01090522 --ra-r--r-- root root
s 4 0x61200615 --ra-ra-ra- root root
s 5 0x73200615 --ra-ra-ra- root root
s 6 0x70200615 --ra-ra-ra- root root
s 7 0x69200615 --ra-ra-ra- root root
s 8 0x75200615 --ra-ra-ra- root root
s 9 0x63200615 --ra-ra-ra- root root
s 10 0x64200615 --ra-ra-ra- root root
s 11 0x66200615 --ra-ra-ra- root root
s 12 0x6c200615 --ra-ra-ra- root root
s 13 0x6d200615 --ra-ra-ra- root root
s 14 0x6f200615 --ra-ra-ra- root root
s 4014095 0x4453314d --ra------- root sys
s 860176 0x55535253 --ra-ra-ra- root sys
s 106513 0x00005653 --ra-ra-ra- root sys
s 18 0x410c073b --ra-ra-ra- root root
s 19 0xffffffff --ra-r--r-- root root
s 2076692 0xf7a3cbf8 --ra-r----- oracle dba
s 118805 0x0c2b04d6 --ra-r----- oracle dba
s 827414 0x368ba5f0 --ra-r----- oracle dba
s 16407 0xb4894a70 --ra-r----- oracle dba
s 2580504 0x00000000 --ra-ra-ra- camprd camprd
s 16409 0xfa88a96c --ra-r----- oracle dba
s 27 0x52200615 --ra-ra-ra- root root
s 12317 0x299dfc44 --ra-r----- oracle dba
s 1802270 0xa00bf0bc --ra-r----- oracle dba
s 16415 0x276f9ef4 --ra-r----- oracle dba
s 1327137 0xd9fd40e0 --ra-r----- oracle dba
s 704547 0x40d5868c --ra-r----- oracle dba
s 4164 0x00000000 --ra-ra-ra- proteus proteus
TTr
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

I looked at the man page for sar (-m option) and the "sema/s" column is not number of semaphores, it is rather "semaphore calls per second", that is system calls made to the OS to access the semaphores.

So the ipcs output is what really shows you the semaphores that exist.

Now, yes you have a busy application, and if it is distributed, with many different processes working on the same task, the samaphore call activity will be high.

If the server response is good, then there is probably nothing wrong. Otherwise you either consider adding more server power or see if the application can take any fine tuning and configuration changes.
skt_skt
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

see this one. So it match with what you said?( i did not catch "number of semaphore primitives called per second")
http://osr507doc.sco.com/en/PERFORM/view_IPC_act.html

If the sema/s column in the output from sar -m shows that the number of semaphore primitives called per second is high (for example, greater than 100), the application may not be using IPC efficiently. It is not possible to recommend a value here. What constitutes a high number of semaphore calls depends on the use to which the application puts them and the processing power of the system running the application
TTr
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

> ...is high (for example, greater than 100)

"High" is a relative term here. that document is from 2003. And it should be your application specialist, or your application vendor support to determine if the sema activity is normal or not.

If you have too many app processes running they could be continuously checking the semaphores to determine if they will take a specific action or not.

You need to look at this from the application side.
Michael Steele_2
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

Hi

I've review your settings and I don't see anything out of norm., but I don't see a total memory size which is needed for calculations.

I've reread the attachment and there's a possiblity that you have frequent small file calls that are using up all your semaphore metadata space.

Review both the JFS and semaphore doc I've attached, and also provide your memory, dbc_max_pct, and vx_bc_bufhwm values.

Support Fatherhood - Stop Family Law
Rita C Workman
Honored Contributor

Re: sema/s is over 2000 and pefromance issue with application

You ask about performance, so I'm going to ask:

How much server memory on this box?

What are the setting for the following parms:

dbc_max%
dbc_min%
ninode
vx_ninode


Thanks,
Rita