Issue with Apache taking a dive

[Darrell Tyler_1]

Attached is a slice of the error_log file of apache. It started to go belly up at about 5:16 from my bigbrother log. I?m having some issues trying to locate the exact cause of error. ?Lil help please.

[A. Clay Stephenson]

Lil log please? The log appears to written using invisible ink.

[Darrell Tyler_1]

Lest try it again, I have attached a text file. Thanks

[Steven E. Protter]

Looks like some processes got killed or could not open around 5:13 a.m.

I would do some general tuning based on nproc, nfile and maxuprc

This could just be a resource problem.

SEP

[Darrell Tyler_1]

I've given all three of those tunable parameters a look-see and they are set high enough where it is hard to believe that they would be the culprits. They are
nfile=15017, nproc=5100 and maxuprc=500.

[John Payne_2]

Did your server get killed at 5:13 and rolled again at 6:28?

I used to get this when the operation's staff improperly brings apache down. The problem was that the ssl cache file was not deleted when apache was brought down.

To remedy this (You'd think it would just be easier to have the operators do what they should, wouldn't you? Not so here... :) ), I added the following line to the startup script:

rm /opt/apache/logs/ssl_scache

So that the Shared memory cache file is always non-existant when apache starts up. When you stop apache with apachectl, this cache file is normally deleted for you.

The location for this file is set in ssl.conf:

SSLSessionCache shmcb:/opt/hpws/apache/logs/ssl_scache(512000)

Hope it helps

John

[Steven E. Protter]

gpm graphical glance has a nice screen to monitor system parameters as things happen. If the high water mark is at or near capacity, adjust upward.

You might want to monitor resources with the scripts I'm attaching.

Next: If the server is exposed to the Internet, you may be the victim of a denial of service attack.

I have just closed down a DOS method that floods the web server with spurious content requests for documents not posted on your server. In the log, you'll see get and post requests that start with http, which you won't see in access_log for local pages.

Depending on your setup, if this is an issue, I can help you harden apache and stop this type of attack.

SEP

[A. Clay Stephenson]

Note those errors associated with errno 17 - EEXIST. You could get those kinds of errors trying to create a file that already exists but because the error message specifically mentions shared memory my best guess is that a shmget() system call is failing. Shmget can also set errno = 17 if trying to create a shared memory segment for which an identifier exists. Your best approach to absolutely nailing this is to download the Apache source and look through the code for the error message. Tusc could also be run against Apache and that would display the system calls and well as the arguments but looking through the source code is more definitive.