topic Re: Kernel Semaphore table utilization high in Operating System - HP-UX

Kernel Semaphore table utilization high

G V R Shankar — Sat, 14 Jul 2007 00:55:19 GMT

Hello All,

I have a system running on HP Ux 11.11. I found that semaphore table utilization is at 99%. I wish to find and remove the unused semaphores. Is the following process correct.

Check for defunct semaphore using ipcs –sb. If the third column is like 0x0000000, then we can remove the semaphore.

Use ipcrm –s to remove the semaphore. ID is the second column from ipcs –sb output.

If the above process is in correct, could you please advice how to bring the semaphore usage, without rebooting the box.

Thanks,

Ravi.

Re: Kernel Semaphore table utilization high

skt_skt — Sat, 14 Jul 2007 04:34:30 GMT

Before removing anything you should check with your database team too. I have seen most of the semaphores are being used by either root or oracle.Killig the wrong one could bring down the database.

I am not sure the current step you have is right.
Do you see any line with 0x0000000 entry currently in "ipcs -sb" output.

where are you getting the semaphore table utilization is at 99% messages?

Could you send the output of "kmtune|grep -i sem"

Re: Kernel Semaphore table utilization high

Bill Hassell — Sat, 14 Jul 2007 09:42:37 GMT

As mentioned, removing the wrong semaphores can destroy communication between modules and cause SERIOUS database corruption. Semaphores are an infinitesimally small amount of RAM. Rather than risk massive downtime to repair the databases, I would just triple the number of semaphores and forget about them. NOTE: semaphores and shared memory will be orphaned by using kill -9...NEVER use kill -9 for database applications. All commercial-quality applications are designed to be terminated in a graceful manner, but kill -9 cannot be graqcefully handled by any process.

Re: Kernel Semaphore table utilization high

G V R Shankar — Sat, 14 Jul 2007 18:23:41 GMT

Hello Santosh,
Here is a part of o/p of ipcs -sb

s 15325223 0x00000000 --ra------- ftphr ftpgroup 1
s 5718056 0x00000000 ----------- ftphr ftpgroup 1
s 798761 0x00000000 ----------- ftphr ftpgroup 1
s 95274 0x00000000 --ra------- ftphr ftpgroup 1
s 55339 0x00000000 ----------- ftphr ftpgroup 1
s 53292 0x00000000 ----------- ftphr ftpgroup 1
s 53293 0x00000000 --ra------- ftphr ftpgroup 1
s 39982 0x00000000 --ra------- hrpadm sapsys 1
s 12335 0x00000000 --ra------- hrpadm sapsys 1

sema 1 - 1
semaem 16384 - 16384
semmap 1026 - (SEMMNI+2)
semmni 1024 - 1024
semmns 2048 - 2048
semmnu 1024 - 1024
semmsl 2048 Y 2048
semume 1024 - 1024
semvmx 32767 - 32767

Ravi.

Re: Kernel Semaphore table utilization high

Dennis Handly — Sat, 14 Jul 2007 20:58:44 GMT

>Here is a part of o/p of ipcs -sb

You might want to use -sa so you can see all of the info.

Re: Kernel Semaphore table utilization high

G V R Shankar — Sat, 14 Jul 2007 21:08:24 GMT

ipcs -sa or sb, what does 0x00000000 mean. How can I know, which process is using it.

Re: Kernel Semaphore table utilization high

Bill Hassell — Sun, 15 Jul 2007 07:18:43 GMT

There is no process owner for a semaphore. A program can create the semaphore but from then on, it is simply a flag that is known to specific programs. No data is transferred, it is just go/no-go flag. For shared memory segments, the -c option will show you PIDs of the processes that created or last attached to the resource. Those processes may no longer exist but the resource may still be meaningful to other processes -- only the author of the program (and occasionally, the documentation) knows for sure. Use ipcs -a to all the options (you'll need to expand your terminal window to 150 columns to read it).

Re: Kernel Semaphore table utilization high

G V R Shankar — Sun, 15 Jul 2007 09:38:01 GMT

Hi Bill,

Your explanation is interesting. Could you please let me know, is there any procedure to know the unused semaphores.

Ravi.

Re: Kernel Semaphore table utilization high

G V R Shankar — Sun, 15 Jul 2007 12:22:00 GMT

Hello All,

I found a document that supports my statement at the beginning of the thread.

http://developer.softwareag.com/AdaNat/knowhow//adabas_tech/110205_AdabasUNIXMemory.pdf

I wish to know your comments.

Ravi.

Re: Kernel Semaphore table utilization high

skt_skt — Sun, 15 Jul 2007 14:36:12 GMT

here is the value i have

sema 1 - 1
semaem 16384 - 16384
semmap 32767 - 32767
semmni 32765 - 32765
semmns 32767 - 32767
semmnu 29984 - 29984
semmsl 2048 Y 2048
semume 64 - 64
semvmx 32767 - 32767

Re: Kernel Semaphore table utilization high

skt_skt — Sun, 15 Jul 2007 15:22:09 GMT

Even if the third column looks 0x0000000 (key)and tracing it with id (in my case 34938) shows there are processes still assocaited.Not sure what is the impact if we we kill.

Still need more analysis.

#ipcs|grep 0x0000000
m 34938 0x00000000 D-rw------- applmgr dba

root [/opt/local/unix/bin] ./shminfo -s 34938
libp4 (7.0): Opening /stand/vmunix /dev/kmem

Loading symbols from /stand/vmunix
shminfo (3.6)

Shmid 34938:
struct shmid_ds at 0xdfb5b8
Pseudo vas at 0x9d3da800
Pseudo pregion at 0x94f55100
Shared region at 0xa9017c00
Segment at 0x74db400.0xc8126000
Segment allocated out of "Global 32-bit quadrant 4"
Processes using this segment:
proc=0x519eb500 (pid 11168 "httpd"): vas=0x9d0d4200, SHMEM preg=0x9f892f00
proc=0x519b6580 (pid 20116 "httpd"): vas=0x9e0c9d00, SHMEM preg=0xace54e00
proc=0x519d8900 (pid 28977 "httpd"): vas=0x78fcbb00, SHMEM preg=0xa9059000
proc=0x51a2b4c0 (pid 28775 "httpd"): vas=0x9f89ab00, SHMEM preg=0xa4669300
proc=0x519da340 (pid 28771 "httpd"): vas=0x9edccc00, SHMEM preg=0x95654600
proc=0x519e9e80 (pid 28503 "httpd"): vas=0x9f3e0b00, SHMEM preg=0xa5577500
proc=0x519e9700 (pid 28502 "httpd"): vas=0x9c6fbd00, SHMEM preg=0xaceb8100
proc=0x51a1ee00 (pid 28501 "httpd"): vas=0xa076b500, SHMEM preg=0xa48d0900
proc=0x519ef880 (pid 28500 "httpd"): vas=0xa4555200, SHMEM preg=0x9fe69300
proc=0x51a2fc00 (pid 28498 "httpd"): vas=0x94445c00, SHMEM preg=0xa9e62000
proc=0x519e22c0 (pid 28468 "httpd"): vas=0xacda2e00, SHMEM preg=0x94a46900

acadv05:root [/opt/local/unix/bin] ps -ef|grep 28468
applmgr 20116 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28977 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28771 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28468 1 0 Jul 10 ? 0:36 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28502 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28503 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 11168 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28500 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28488 28468 0 Jul 10 ? 2:03 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
root 2860 2052 1 16:15:33 pts/5 0:00 grep 28468
applmgr 28501 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28775 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s
applmgr 28498 28468 0 Jul 10 ? 0:00 /sa_lgd3_q/applmgr/q303ora/iAS/Apache/Apache/bin/httpd -d /s

acadv05:root [/opt/local/unix/bin] ./shminfo|grep 34938
Q4 0x074db400.0xc8126000-0xc83affff 2600 SHMEM id=34938

Re: Kernel Semaphore table utilization high

G V R Shankar — Mon, 16 Jul 2007 08:41:45 GMT

Hi Santhosh,

Thanks for the values.

Second, I was speaking about semaphores, not shared memory segments.
Following one is a shared memory segment.
m 34938 0x00000000 D-rw------- applmgr dba

Thanks Ravi.

Re: Kernel Semaphore table utilization high

Dennis Handly — Tue, 17 Jul 2007 00:14:48 GMT

Ok, I've figured out why 0x0000000 is special. You are only imagining you are seeing it. ;-)
It is really IPC_PRIVATE. With this info, it is pretty simple. If there are no processes using it, that semaphore is dead.

Hmm, it seems an application can do ipcs -s and just pick up existing private semaphores and just use them without going through semget(2).

I guess your Adabas documentation is impling they don't do that, so as long as it is not running, IPC_PRIVATE semaphores are dead.

Attached is a program that uses pstat(2) to get the time in days, not just in hours and seconds, perhaps that maybe helpful to tell if they are still used?
ID KEY NSEMS OTIME CTIME
0 0x4f0c2576 1 06-11|10:43 06-11|10:43
...
40983 0x00000000 1 NO-ENTRY 07-16|21:59
24 0x00000000 1 07-14|11:44 07-14|11:44

So 40983 was created on July 16.

Re: Kernel Semaphore table utilization high

G V R Shankar — Tue, 17 Jul 2007 02:44:25 GMT

Hi Dennis,

Your explanation is wonderfull.

I tried to compile the C program on HP UX 11.00 OS, I got the following error.

/usr/bin/cc 291509.c
(Bundled) cc: "291509.c", line 13: error 1000: Unexpected symbol: "t".
(Bundled) cc: "291509.c", line 14: error 1588: "t" undefined.
(Bundled) cc: "291509.c", line 15: error 1563: Expression in if must be scalar.
(Bundled) cc: "291509.c", line 29: error 1588: "pse_semid" undefined.
(Bundled) cc: "291509.c", line 29: error 1531: Invalid member of struct or union.

I am not good at C programming. Could you please help me.

Ravi.

Re: Kernel Semaphore table utilization high

Dennis Handly — Tue, 17 Jul 2007 05:08:59 GMT

>I tried to compile the C program on HP UX 11.00 OS

You must compile the program with an ANSI C compiler. Or you must port it to K&R.

Also, it seems that source only compiles on 11.23 and higher.

Here is a new version. You'll need to compile with -DHP11_11 for 11.00 and 11.11.

Re: Kernel Semaphore table utilization high

G V R Shankar — Wed, 18 Jul 2007 00:20:11 GMT

I shall try and let you know.

Thanks
Ravi

Re: Kernel Semaphore table utilization high

G V R Shankar — Wed, 18 Jul 2007 05:48:15 GMT

Hello Dennis,

Could you please provide me a URL, where I can download DHP11_11 compiler

Regards,

Ravi.

Re: Kernel Semaphore table utilization high

Dennis Handly — Wed, 18 Jul 2007 14:29:57 GMT

>Could you please provide me a URL, where I can download DHP11_11 compiler

-DHP11_11 is a compiler option that should work with your bundled compiled.

Re: Kernel Semaphore table utilization high

G V R Shankar — Thu, 19 Jul 2007 01:06:04 GMT

Hi Dennis,

It works fine. Thank You Very Much.

Regards,
Ravi.

Re: Kernel Semaphore table utilization high

G V R Shankar — Thu, 19 Jul 2007 01:13:45 GMT

Thanks for all YOUR Contributions.

Regards,
Ravi.