HP-UX B.11.11: unable to decrement semaphore (errno = 28)

[Peter Renner (rie)]

We´ve got a suspicious problem since we did a maintenance reboot of our rp5400 server running HP-UX B.11.11:

As soon as we start some specific non-HP tools (DB/C 8 dbcsql and other) they´re quitting an this error message appears:

"Keyboard initialization error: unable to decrement semaphore (errno = 28)"

We´ve looked onto the free semaphores and the connection of directly-attached console & keyboard, all ok. We had the keyboard issue twice in the last year: it was removed by someone and the reconnection had helped in the the past, but now it doesn´t anymore.

Until now nothing was found in documentations of HP-UX or anywhere in the internet, it seems a little bit unique, this nasty issue.

Suggestions are welcome, thank you in advance.

[Ivan Krastev]

What is your value of semmns ?

Check for patch PHKL_28703 as well.

regards,
ivan

[Peter Renner (rie)]

The semmns parameter is set to "Current Value" = 256 and "Pending Value" = 256, it is static.

PHKL_28703 is not installed.

The server is not under the highest duty, max. 20% of this semaphore limit is reached.

[Rita C Workman]

Never saw anything like this...

errno=28 on hpux is that some device is 100% full. It is saying it is unable to decrement (i.e. release) semaphore, not that it is short and can not gain a lock on another semaphore. It is possible that since it can not release the lock on the semaphore, possibly due to some application error, it then can no proceed to get the required input from the keyboard. Or because it could not get the input from the keyboard, because a "?? device ??" hit 100%, that it then could not release the semaphore lock. ..... dizzying....

Can you check your logs and see what is running when this hits? Can you see where or what it is trying to do when this happens ? Maybe you can at least narrow down what is potentially causing this,...cause it looks to be in some code somewhere.

Just my ramblings,
Rita

[Rita C Workman]

Here is one thought. You mention your that you have 256 set for a parm value. Which is quite small.

By chance...could it be possible that your semaphore un-do parm (semmnu)is set too low???

It's a wild shot in the dark?

/rcw

[Peter Renner (rie)]

Thanks, every shot in the dark is welcome, because only a few hours are left until highly important cron jobs are going to run (they will surely fail as it actually seems).

We´ve got nothing (for our knowldedge) to analyze regarding the faulting utilities. For example, you´re in the shell, invoking the DB/C tool:

$ dbcsql
Keyboard initialization error: unable to decrement semaphore (errno = 28)

It is crashing immediately, you´ll find no further information what´s going in syslog or in viewing "top" in example, it seems nothing is logged regarding this failure.

As you wrote, the process is stalling, the semaphore could not be decreased and the err msg is displayed.


semmnu is set to 30.

[Ivan Krastev]

Can you try to run the same command under tusc and watch system calls?

regards,
ivan

[Rita C Workman]

tusc output might be helpful as Ivan mentioned.

Default for semmnu is 30...default for semmni is 64. You said you increased the one; did you increase the other?

Make sure your semmnu is about maybe half the size of your semmni value.

/rcw

[Peter Renner (rie)]

Thank you.

Attached is the output from tusc, worked for the first time with it. Interesting.

I´ve never changed semmnu and semmni from default to the current values, must have been someone a decade ago - well, I´ll do as suggested, semmnu half the size of semni (<-which I´ll change to a much higher value, why not.)

[Rita C Workman]

Well I'm no export at reading tusc output, so hopefully Ivan can see more.

But I did notice that the error you report is actually not the first error. I noticed:
open("/usr/informix/lib/esql/libsql.0", O_RDONLY, 0) ERR#2 ENOENT
[19261] open("/usr/informix/lib/esql/libsql.sl", O_RDONLY, 0) = 4

Hard to read the output, but it seems to want to open some Informix lib file and can't find it (HPUX error=2 is file not found).

So your jobs kicks off, starts gathering a few things, allocates some semaphore, does query and goes starts looking for "stuff" (lseek) and then there's some write and mapping of shared memory. Then it works to keep going....until it hits the 100% full.

So...how's about tell us about your hardware [hpux rp5400, 11.11] how much memory?
What is value of following:
dbc_max_%
dbc_min_%
nproc
ninode
nfile
nflocks
vx_ninode
semm* all semaphore parms
nbuf
swapmem_on

Anything else running on this box?

I think that's a start...

/rcw