topic Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28) in Operating System - HP-UX

HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Peter Renner (rie) — Wed, 09 Jul 2008 10:06:17 GMT

We´ve got a suspicious problem since we did a maintenance reboot of our rp5400 server running HP-UX B.11.11:

As soon as we start some specific non-HP tools (DB/C 8 dbcsql and other) they´re quitting an this error message appears:

"Keyboard initialization error: unable to decrement semaphore (errno = 28)"

We´ve looked onto the free semaphores and the connection of directly-attached console & keyboard, all ok. We had the keyboard issue twice in the last year: it was removed by someone and the reconnection had helped in the the past, but now it doesn´t anymore.

Until now nothing was found in documentations of HP-UX or anywhere in the internet, it seems a little bit unique, this nasty issue.

Suggestions are welcome, thank you in advance.

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Ivan Krastev — Wed, 09 Jul 2008 10:17:56 GMT

What is your value of semmns ?

Check for patch PHKL_28703 as well.

regards,
ivan

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Peter Renner (rie) — Wed, 09 Jul 2008 11:48:39 GMT

The semmns parameter is set to "Current Value" = 256 and "Pending Value" = 256, it is static.

PHKL_28703 is not installed.

The server is not under the highest duty, max. 20% of this semaphore limit is reached.

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Rita C Workman — Wed, 09 Jul 2008 11:57:10 GMT

Never saw anything like this...

errno=28 on hpux is that some device is 100% full. It is saying it is unable to decrement (i.e. release) semaphore, not that it is short and can not gain a lock on another semaphore. It is possible that since it can not release the lock on the semaphore, possibly due to some application error, it then can no proceed to get the required input from the keyboard. Or because it could not get the input from the keyboard, because a "?? device ??" hit 100%, that it then could not release the semaphore lock. ..... dizzying....

Can you check your logs and see what is running when this hits? Can you see where or what it is trying to do when this happens ? Maybe you can at least narrow down what is potentially causing this,...cause it looks to be in some code somewhere.

Just my ramblings,
Rita

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Rita C Workman — Wed, 09 Jul 2008 12:05:07 GMT

Here is one thought. You mention your that you have 256 set for a parm value. Which is quite small.

By chance...could it be possible that your semaphore un-do parm (semmnu)is set too low???

It's a wild shot in the dark?

/rcw

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Peter Renner (rie) — Wed, 09 Jul 2008 12:22:55 GMT

Thanks, every shot in the dark is welcome, because only a few hours are left until highly important cron jobs are going to run (they will surely fail as it actually seems).

WeÂ´ve got nothing (for our knowldedge) to analyze regarding the faulting utilities. For example, youÂ´re in the shell, invoking the DB/C tool:

$ dbcsql
Keyboard initialization error: unable to decrement semaphore (errno = 28)

It is crashing immediately, youÂ´ll find no further information whatÂ´s going in syslog or in viewing "top" in example, it seems nothing is logged regarding this failure.

As you wrote, the process is stalling, the semaphore could not be decreased and the err msg is displayed.

semmnu is set to 30.

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Ivan Krastev — Wed, 09 Jul 2008 12:38:09 GMT

Can you try to run the same command under tusc and watch system calls?

regards,
ivan

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Rita C Workman — Wed, 09 Jul 2008 13:17:01 GMT

tusc output might be helpful as Ivan mentioned.

Default for semmnu is 30...default for semmni is 64. You said you increased the one; did you increase the other?

Make sure your semmnu is about maybe half the size of your semmni value.

/rcw

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Peter Renner (rie) — Wed, 09 Jul 2008 13:27:56 GMT

Thank you.

Attached is the output from tusc, worked for the first time with it. Interesting.

IÂ´ve never changed semmnu and semmni from default to the current values, must have been someone a decade ago - well, IÂ´ll do as suggested, semmnu half the size of semni (<-which IÂ´ll change to a much higher value, why not.)

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Rita C Workman — Wed, 09 Jul 2008 13:53:21 GMT

Well I'm no export at reading tusc output, so hopefully Ivan can see more.

But I did notice that the error you report is actually not the first error. I noticed:
open("/usr/informix/lib/esql/libsql.0", O_RDONLY, 0) ERR#2 ENOENT
[19261] open("/usr/informix/lib/esql/libsql.sl", O_RDONLY, 0) = 4

Hard to read the output, but it seems to want to open some Informix lib file and can't find it (HPUX error=2 is file not found).

So your jobs kicks off, starts gathering a few things, allocates some semaphore, does query and goes starts looking for "stuff" (lseek) and then there's some write and mapping of shared memory. Then it works to keep going....until it hits the 100% full.

So...how's about tell us about your hardware [hpux rp5400, 11.11] how much memory?
What is value of following:
dbc_max_%
dbc_min_%
nproc
ninode
nfile
nflocks
vx_ninode
semm* all semaphore parms
nbuf
swapmem_on

Anything else running on this box?

I think that's a start...

/rcw

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Ivan Krastev — Thu, 10 Jul 2008 07:39:58 GMT

You can see the error:
[19261] semget(0x31142c39, 0, 0666) ...................... = 25
[19261] semctl(25, 0, IPC_STAT, 2013202648) .............. = 0
[19261] semop(25, 0x40004480, 1) ......................... ERR#28 ENOSPC
sem[0].sem_num: 0
sem[0].sem_op: -1
sem[0].sem_flg: SEM_UNDO

Take a look at this kernel parameter - http://docs.hp.com/en/939/KCParms/KCparam.SemUME.html

and adjust it.

regards,
ivan

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Dennis Handly — Thu, 10 Jul 2008 09:04:19 GMT

>Rita: I noticed:
open("/usr/informix/lib/esql/libsql.0", O_RDONLY, 0) ERR#2 ENOENT
open("/usr/informix/lib/esql/libsql.sl", O_RDONLY, 0) = 4

>but it seems to want to open some Informix lib file and can't find it (error=2 is file not found).

This is dld mapping in the shlibs and looking at several paths before it finds them.

Re: HP-UX B.11.11: unable to decrement semaphore (errno = 28)

Rita C Workman — Thu, 10 Jul 2008 17:01:54 GMT

Hi Dennis,

I noticed the error, but didn't look closely at the rest. That's why it's so great to have everyones eyes looking.

From what Ivan is saying, he is leaning where I did. Parm settings. Kind of hard to say too much (other than some basic parms I mentioned for us to check), without knowing more about the database (SGA, etc) and number of users, box physmem. So if Peter comes back with some info maybe we can see something else there.

Rgrds,
Rita