topic Re: Apache Error in Operating System - HP-UX

Apache Error

Ricardo_14 — Mon, 22 Jul 2002 17:30:12 GMT

Hi, All.
Recently i have some problems with my Apache 2.0.39, on an Unix HP-UX 11.00.
The error_log file show the following error messages, and stop my HTTPD service:

Can anyone help me, please?

[Mon Jul 22 14:16:05 2002] [emerg] (28)No space left on device: couldn't grab th
e accept mutex
[Mon Jul 22 14:16:06 2002] [alert] Child 24511 returned a Fatal error...
Apache is exiting!
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:07 2002] [emerg] (22)Invalid argument: couldn't release the ac
cept mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:10 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:10 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:19 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:19 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:19 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:24 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:25 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:37 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:47 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:17:20 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex

Re: Apache Error

Hai Nguyen_1 — Mon, 22 Jul 2002 17:34:42 GMT

Ricardo,

Apparently, the filesystem on which this apache resides or writes to has been filled up.

You may want to stop the apache server, save the apache log for investigation,
start up the apache server again.

Hai

Re: Apache Error

Martin Johnson — Mon, 22 Jul 2002 17:37:31 GMT

Do "bdf" or "df -k" to determine which file system is full. You should bring down apache, clear room in the full file system, then restart apache.

HTH
Marty

Re: Apache Error

Ricardo_14 — Mon, 22 Jul 2002 17:56:35 GMT

Hai, and Martim

Thanks for your reply.

Hai,
I??m already examine my apache logs. The only error messages is that.
I??m not found any other error message.

Martin,
This is the result os the "bdf". The filesystem aparently is good.

The apache software is installed in directory /opt/apache2
The documents is in the directory /webcoe

Best Regards,

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1536000 23761 1418604 2% /
/dev/vg00/lvol1 99669 35682 54020 40% /stand
/dev/vg00/lvol9 4096000 1241133 2676441 32% /webcoe
/dev/vg00/lvol8 1536000 639819 840913 43% /var
/dev/vg00/lvol7 4608000 499433 3851790 11% /usr
/dev/vg00/lvol6 204800 1762 190542 1% /tmp
/dev/vg00/lvol5 4096000 292878 3565437 8% /opt
/dev/vg00/lvol4 53248 2266 47802 5% /home
/dev/vg00/lvol10 4096 213 3648 6% /bkp_erro

Re: Apache Error

Martin Johnson — Mon, 22 Jul 2002 18:12:05 GMT

Have you checked the system tables? (The glance t option).

HTH
Marty

Re: Apache Error

Ricardo_14 — Mon, 22 Jul 2002 18:44:31 GMT

Martin,

My Glance reports table is this:

(Aparently is this ok, ins??t?)

Regards,

SYSTEM TABLES REPORT Users= 1

System Table Available Requested Used High
--------------------------------------------------------------------------------
Inode Cache (ninode) 476 na 476 476
Shared Memory 12.5gb 6.0mb
Message Buffers 16kb na 0kb 0kb
Buffer Cache 256.0mb na 256.0mb 256.0mb
Buffer Cache Min 25.6mb
Buffer Cache Max 256.0mb
DNLC Cache 1500

Model : 9000/800/K360 Phys Memory :512.0mb Network Interfaces : 4
OS Name : HP-UX Number CPUs : 1 Number Swap Areas : 2
OS Release: B.11.00 Number Disks: 2 Avail Volume Groups: 1
OS Kernel Type: 64 bits Mem Region Max Page Size: 64.0mb

Re: Apache Error

Hai Nguyen_1 — Mon, 22 Jul 2002 18:56:16 GMT

Ricardo,

You
I see some unrelated issue here which is worth looking into.

From your glance -t report,
it says that you maximum buffer can take up to 50% of your total RAM. You should trim this down to 10-15% by modifying the value of dbc_max_pct using SAM which requires a reboot.

Hai

Re: Apache Error

Wodisch_1 — Mon, 22 Jul 2002 18:57:17 GMT

Hi Ricardo,

this is only a guess, but the "device" running out of space is your "mutex device" (the same message appears when you are running out of semaphores). So I would recommend to increase "nkthreads" and "maxthreads" as this is all thread-related...

HTH,
Wodisch

Re: Apache Error

Hai Nguyen_1 — Mon, 22 Jul 2002 19:03:45 GMT

Ricardo,

In addition to dbc_max_pct, you should modify your inode cache as well. On your system, it has reached the maximum. You can modify ninode's value to 2488 which should be good enough using SAM -> Kernel Configurations as in my previous message.

Hai

Re: Apache Error

Ricardo_14 — Wed, 24 Jul 2002 14:26:19 GMT

Thanks for your help!!!
I??ve made the changes on the system and i??ll be waiting the results, because the apache server rarelly crash.
If the problem returns, ?? will send new messages in this forum.

Regards,

Re: Apache Error

Cahya — Fri, 18 Oct 2002 07:06:53 GMT

Hi,
I have the same problem, and then I
made all the kernel configuration changes suggested to do, but it doesn't work, apache will exit with the same
error message after one hour or more.
I use apache 2.0.43.
Any help is very appreciated, thanks.
Cahya

Re: Apache Error

Jonathan R Hughes — Fri, 18 Oct 2002 15:24:38 GMT

You may be running out of semaphore undo. try increasing kernel parameter semmnu to the sum of maxclients and the number of semaphores in use when apache is not running.

Re: Apache Error

Harald Coeleveld — Wed, 11 Dec 2002 10:54:54 GMT

Re: Apache Error

Bill McNAMARA_1 — Wed, 11 Dec 2002 11:00:17 GMT

doesn't look like the lasted apache version in any case, you should include an swlist output.
I was in Nijmegen (philips site for a call once.. loved the place, worked with some nice guys there, Karl Rambach, don't know if you know him..) Anyway, I haven't looked much at your log or kernel parameter although am sure you suspicions are right to go there, but the error message starts with file system full warnings, this is possibly causing it all, run a "bdf" during the problem and find out which filessyetm is full..

What else is the box doing?.

Re: Apache Error

Harald Coeleveld — Wed, 11 Dec 2002 11:09:10 GMT

Some more additional info:

# bdf -l
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 4800512 2075002 2555203 45% /
/dev/vg00/lvol1 99669 25827 63875 29% /stand
/dev/vg00/lvol4 1777664 9219 1657961 1% /mnt/spool

Apache web server pages + config stored on account on separate file server:
bdf .
Filesystem kbytes used avail %used :/mnt/.../.../
8388608 724468 7468652 9%

Re: Apache Error

Harald Coeleveld — Wed, 11 Dec 2002 11:14:43 GMT

Bill,

Karl's a direct (well-experienced) collegue of mine (Consumer Business dept.). Have forwarded the URL; perhaps he'll drop a line or two (he's quite busy at the time). I'll say hi for you next time we speak (this afternoon probably). Thanks for replying.

Harald Coeleveld
UNIX sysadmin / Local UNIX support business Line Cellular.

Re: Apache Error

Harald Coeleveld — Wed, 11 Dec 2002 11:19:16 GMT

Re: Apache Error

Stefan Stechemesser — Wed, 11 Dec 2002 11:45:38 GMT

Hi,

I had the same problem on a 10.20 E55 Box running apache 2.0.43.

I increased the parameters semmns => 150
semmnu => 50
semume => 20
shmmax => 0X6000000

Since then, I had no more problems with my webserver.

Re: Apache Error

Harald Coeleveld — Wed, 11 Dec 2002 12:25:03 GMT

Stefan, thanks for your reply.

As we are using about 300 HP-UX servers, patch management / kernel parameter tuning is strictly managed. Consulted with UNIX dept to tweak params on web server node as a test case.

Note: We are using HP-UX 11.11 (not 11.0)

Current values:
sema 1 1 Static N/A Enable Sys V Semaphores
semaem 16384 16384 Static N/A Max Value for Adjust on Exit Semaphores
semmap 130 130 Static N/A Max Number of Semaphore Map Entries
semmni 128 128 Static N/A Number of Semaphore Identifiers
semmns 512 512 Static N/A Max Number of Semaphores
semmnu 30 30 Static N/A Number of Semaphore Undo Structures
semmsl 2048 2048 Dynamic N/A Maximum Number of Semaphores Per ID
semume 10 10 Static N/A Semaphore Undo Entries Per Process
semvmx 32767 32767 Static N/A Semaphore Maximum Value
shmmax 0x4000000 0x4000000 Dynamic N/A Max Shared Mem Segment (Hex)

Changes to be made:
semmns: 512 => 150 WILL **NOT** BE CHANGED
semmnu: 30 => 50 WILL BE CHANGED
semume: 10 => 20 WILL BE CHANGED
shmmax: 0x4000000 => 0X6000000 WILL BE CHANGED

I'll post later to see if things are stable.
Click on my name for e-mail address details if you want to me, if required.

Thanks to all for contributing !

Re: Apache Error

Harald Coeleveld — Wed, 11 Dec 2002 12:43:54 GMT

Hmmm.. one last comment though.

mutex = mutual exclusion = semaphores = probably multi-threading, right ?

How to disable multi-threading in the Apache config as the web server load is not that high ?

I prefer this over kernel param changes for now...