HP-UX 11.0, Apache 2.0.45 problem.

[Eric Ladner]

Sorry for the blank message.

I'm using HP's Apaache 2.0.45 (from software.hp.com) on a web server that serves a BUNCH of pages every day.

Recently, these messages started showing up in the apache error_log, and after some period of time, the server becomes unresponsive.

[Thu May 29 09:15:43 2003] [emerg] (28)No space left on device: apr_proc_mutex_lock failed. Attempting to shutdown process gracefully.

The machine DOES have plenty of disk space and I've checked the system tables that glance shows and there are no entries that are close to maxing out (nfile is the highest at 45 percent).

Has anyone else run into this problem?

[Pete Randall]

When I hear "mutex_lock", I think database. Is there one involved?


Pete

[Steven E. Protter]

Does the Apache server actually shut down under these circumstances?

Here are some threads with bunnies, so the problem was solved.

http://forums.itrc.hp.com/cm/QuestionAnswer/1,,0x7fe3cdec06f1d61190050090279cd0f9,00.html

http://search.hp.com/redirect.html?url=http%3A//forums.itrc.hp.com/cm/QuestionAnswer/1,,0x18a7ee3e323bd5118fef0090279cd0f9,00.html&qt=apache+apr_proc_mutex_lock+failed&hit=2

http://search.hp.com/redirect.html?url=http%3A//forums.itrc.hp.com/cm/QuestionAnswer/1,,0x28efcdec06f1d61190050090279cd0f9,00.html&qt=apache+apr_proc_mutex_lock+failed&hit=3

Here is a performance doc, oracle oriented but it can help. Apache uses a lot of shared memory, shmmax and shmseg need to be looked at. Never increase shmmax more than 25% of total memory, which is physical plus swap.

Performance Tuning doc

http://www1.itrc.hp.com/service/cki/search.do?category=c0&docType=Security&docType=Patch&docType=EngineerNotes&docType=BugReports&docType=Hardware&docType=ReferenceMaterials&docType=ThirdParty&searchString=UPERFKBAN00000726&mode=id&admit=-682735245+1053636168960+28353475&searchCrit=allwords&printable=true

SEP

[S.K. Chan · Accepted solution]

The error is somewhat related to process locking. Don't waste time looking for physical disk space utilization because what it is really compaining about is your kernel running out of space (table space I think). Take a look closely again to these parameters .. "nfile", "nproc", maybe "maxthreads". I'm not surprise if increasing any of those would help.

[Eric Ladner]

No databases involved.

The server just did it again, so I bumped up max_thread_count and rebuilt the kernel. I'll see if it makes a difference in about 20 minutes.

[Eric Ladner]

Setting max_thread_proc (or max_proc_threads) to a higher number and recompiling the kernel did the trick. Thanks guys!