Apache Error

[Ricardo_14]

Hi, All.
Recently i have some problems with my Apache 2.0.39, on an Unix HP-UX 11.00.
The error_log file show the following error messages, and stop my HTTPD service:

Can anyone help me, please?

[Mon Jul 22 14:16:05 2002] [emerg] (28)No space left on device: couldn't grab th
e accept mutex
[Mon Jul 22 14:16:06 2002] [alert] Child 24511 returned a Fatal error...
Apache is exiting!
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:06 2002] [emerg] (36)Identifier removed: couldn't grab the acc
ept mutex
[Mon Jul 22 14:16:07 2002] [emerg] (22)Invalid argument: couldn't release the ac
cept mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:08 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:10 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:10 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:19 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:19 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:19 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:24 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:25 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:37 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:16:47 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex
[Mon Jul 22 14:17:20 2002] [emerg] (22)Invalid argument: couldn't grab the accep
t mutex

[Hai Nguyen_1 · Accepted solution]

Ricardo,

Apparently, the filesystem on which this apache resides or writes to has been filled up.

You may want to stop the apache server, save the apache log for investigation,
start up the apache server again.

Hai

[Martin Johnson]

Do "bdf" or "df -k" to determine which file system is full. You should bring down apache, clear room in the full file system, then restart apache.

HTH
Marty

[Ricardo_14]

Hai, and Martim

Thanks for your reply.

Hai,
I??m already examine my apache logs. The only error messages is that.
I??m not found any other error message.


Martin,
This is the result os the "bdf". The filesystem aparently is good.

The apache software is installed in directory /opt/apache2
The documents is in the directory /webcoe

Best Regards,

Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 1536000 23761 1418604 2% /
/dev/vg00/lvol1 99669 35682 54020 40% /stand
/dev/vg00/lvol9 4096000 1241133 2676441 32% /webcoe
/dev/vg00/lvol8 1536000 639819 840913 43% /var
/dev/vg00/lvol7 4608000 499433 3851790 11% /usr
/dev/vg00/lvol6 204800 1762 190542 1% /tmp
/dev/vg00/lvol5 4096000 292878 3565437 8% /opt
/dev/vg00/lvol4 53248 2266 47802 5% /home
/dev/vg00/lvol10 4096 213 3648 6% /bkp_erro

[Martin Johnson]

Have you checked the system tables? (The glance t option).

HTH
Marty

[Ricardo_14]

Martin,

My Glance reports table is this:

(Aparently is this ok, ins??t?)

Regards,

SYSTEM TABLES REPORT Users= 1

System Table Available Requested Used High
--------------------------------------------------------------------------------
Inode Cache (ninode) 476 na 476 476
Shared Memory 12.5gb 6.0mb
Message Buffers 16kb na 0kb 0kb
Buffer Cache 256.0mb na 256.0mb 256.0mb
Buffer Cache Min 25.6mb
Buffer Cache Max 256.0mb
DNLC Cache 1500

Model : 9000/800/K360 Phys Memory :512.0mb Network Interfaces : 4
OS Name : HP-UX Number CPUs : 1 Number Swap Areas : 2
OS Release: B.11.00 Number Disks: 2 Avail Volume Groups: 1
OS Kernel Type: 64 bits Mem Region Max Page Size: 64.0mb

[Hai Nguyen_1]

Ricardo,

You
I see some unrelated issue here which is worth looking into.

From your glance -t report,
it says that you maximum buffer can take up to 50% of your total RAM. You should trim this down to 10-15% by modifying the value of dbc_max_pct using SAM which requires a reboot.

Hai

[Wodisch_1]

Hi Ricardo,

this is only a guess, but the "device" running out of space is your "mutex device" (the same message appears when you are running out of semaphores). So I would recommend to increase "nkthreads" and "maxthreads" as this is all thread-related...

HTH,
Wodisch

[Hai Nguyen_1]

Ricardo,

In addition to dbc_max_pct, you should modify your inode cache as well. On your system, it has reached the maximum. You can modify ninode's value to 2488 which should be good enough using SAM -> Kernel Configurations as in my previous message.

Hai

[Ricardo_14]

Thanks for your help!!!
I??ve made the changes on the system and i??ll be waiting the results, because the apache server rarelly crash.
If the problem returns, ?? will send new messages in this forum.

Regards,

[Cahya]

Hi,
I have the same problem, and then I
made all the kernel configuration changes suggested to do, but it doesn't work, apache will exit with the same
error message after one hour or more.
I use apache 2.0.43.
Any help is very appreciated, thanks.
Cahya

[Jonathan R Hughes]

You may be running out of semaphore undo. try increasing kernel parameter semmnu to the sum of maxclients and the number of semaphores in use when apache is not running.

[Bill McNAMARA_1]

doesn't look like the lasted apache version in any case, you should include an swlist output.
I was in Nijmegen (philips site for a call once.. loved the place, worked with some nice guys there, Karl Rambach, don't know if you know him..) Anyway, I haven't looked much at your log or kernel parameter although am sure you suspicions are right to go there, but the error message starts with file system full warnings, this is possibly causing it all, run a "bdf" during the problem and find out which filessyetm is full..

What else is the box doing?.

[Harald Coeleveld]

Some more additional info:

# bdf -l
Filesystem kbytes used avail %used Mounted on
/dev/vg00/lvol3 4800512 2075002 2555203 45% /
/dev/vg00/lvol1 99669 25827 63875 29% /stand
/dev/vg00/lvol4 1777664 9219 1657961 1% /mnt/spool

Apache web server pages + config stored on account on separate file server:
bdf .
Filesystem kbytes used avail %used :/mnt/.../.../
8388608 724468 7468652 9%

[Harald Coeleveld]

Bill,

Karl's a direct (well-experienced) collegue of mine (Consumer Business dept.). Have forwarded the URL; perhaps he'll drop a line or two (he's quite busy at the time). I'll say hi for you next time we speak (this afternoon probably). Thanks for replying.

Harald Coeleveld
UNIX sysadmin / Local UNIX support business Line Cellular.

[Stefan Stechemesser]

Hi,

I had the same problem on a 10.20 E55 Box running apache 2.0.43.

I increased the parameters semmns => 150
semmnu => 50
semume => 20
shmmax => 0X6000000

Since then, I had no more problems with my webserver.

[Harald Coeleveld]

Stefan, thanks for your reply.

As we are using about 300 HP-UX servers, patch management / kernel parameter tuning is strictly managed. Consulted with UNIX dept to tweak params on web server node as a test case.

Note: We are using HP-UX 11.11 (not 11.0)

Current values:
sema 1 1 Static N/A Enable Sys V Semaphores
semaem 16384 16384 Static N/A Max Value for Adjust on Exit Semaphores
semmap 130 130 Static N/A Max Number of Semaphore Map Entries
semmni 128 128 Static N/A Number of Semaphore Identifiers
semmns 512 512 Static N/A Max Number of Semaphores
semmnu 30 30 Static N/A Number of Semaphore Undo Structures
semmsl 2048 2048 Dynamic N/A Maximum Number of Semaphores Per ID
semume 10 10 Static N/A Semaphore Undo Entries Per Process
semvmx 32767 32767 Static N/A Semaphore Maximum Value
shmmax 0x4000000 0x4000000 Dynamic N/A Max Shared Mem Segment (Hex)

Changes to be made:
semmns: 512 => 150 WILL **NOT** BE CHANGED
semmnu: 30 => 50 WILL BE CHANGED
semume: 10 => 20 WILL BE CHANGED
shmmax: 0x4000000 => 0X6000000 WILL BE CHANGED

I'll post later to see if things are stable.
Click on my name for e-mail address details if you want to me, if required.

Thanks to all for contributing !

[Harald Coeleveld]

Hmmm.. one last comment though.

mutex = mutual exclusion = semaphores = probably multi-threading, right ?

How to disable multi-threading in the Apache config as the web server load is not that high ?

I prefer this over kernel param changes for now...

[Harald Coeleveld]

Kernel parameter configuration changed december 17th 2002.

Web server has been up and running until today (Januari 2nd 2003).

Thank you all for your help & best wishes for 2003!

[Ross Druker]

Hi,
I ran into this same error, "[Thu Feb 13 09:00:25 2003] [emerg] (28)No space left on device: couldn't grab
the accept mutex" with apache 2.0.40 on HP-UX 11.0 and with apache 2.0.39 on HP-UX 10.20.

I had originally found this thread a while ago and upped semmnu on the 11.0 system from 30 to 60, still had a problem, then upped it to 120 and apache has run fine since.

However, I can't get apache to work on the 10.20 system. I upped semmnu to 120. No good.

The I followed the later suggestions I found:

semmns: 128 (left as it was)
semmnu: 120 (left as I'd changed it)
semume: 10 -> 20
shmmax: 0x4 -> 0x6

It still bombs. Any other ideas?

Thanks
Ross Druker

[Surendra Singh_1]

Hi , I am using 2.0.43 on HP-UX and facing the same problem. Is there any way that we can disable the multi-threading on web Sever as I am not having the Administrative permission on this machine.
Any Info on this will be very helpful.


The Error message
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:44 2003] [emerg] (36)Identifier removed: couldn't grab the accept mutex
[Thu Feb 27 12:50:50 2003] [alert] Child 3212 returned a Fatal error...
Apache is exiting!
[Thu Feb 27 13:38:05 2003] [emerg] (22)Invalid argument: couldn't release the ac

[Arthur Luimes_2]

This same problem I had was solved by setting
AcceptMutex fcntl
in httpd.conf.

No kernel adjustements necessary.
See also http://nagoya.apache.org/bugzilla/show_bug.cgi?id=22516

Arthur Luimes
arthur.luimes@ins.com