Segmentation Fault and Memory Bus Error

[Global Server Operation]

Server:N4000, HPUX 11i
I have a customer who uses Apache as their webserver and they are seeing the following problems in the Apache log.DC, we are not using Tomcat, however we are using a product called Stellent. Stellent is a content management application (www.stellent.com). They have been also trying to determine if it is their application that is causing the Apache Segmentation faults and bus errors but haven't found anything. Is there anything else we can do to trap the reason why Apache is having all the Segmentation faults and bus errors?
[Thu May 13 14:15:15 2004] [notice] child pid 2157 exit signal Segmentation fault (11)
[Thu May 13 14:15:15 2004] [notice] child pid 2156 exit signal Segmentation fault (11)
[Thu May 13 14:15:17 2004] [notice] child pid 2167 exit signal Segmentation fault (11)
[Thu May 13 14:15:18 2004] [notice] child pid 2171 exit signal Segmentation fault (11)
[Thu May 13 14:15:18 2004] [notice] child pid 2170 exit signal Segmentation fault (11)
[Thu May 13 14:15:20 2004] [notice] child pid 2200 exit signal Segmentation fault (11)
[Thu May 13 14:15:21 2004] [notice] child pid 2206 exit signal Segmentation fault (11)
[Thu May 13 14:15:21 2004] [notice] child pid 2205 exit signal Segmentation fault (11)
[Thu May 13 14:15:39 2004] [notice] child pid 2432 exit signal Segmentation fault (11)
[Thu May 13 14:15:39 2004] [notice] child pid 2431 exit signal Segmentation fault (11)
[Thu May 13 14:15:39 2004] [notice] child pid 2259 exit signal Segmentation fault (11)
[Thu May 13 14:15:41 2004] [notice] child pid 2545 exit signal Segmentation fault (11)
[Thu May 13 14:15:48 2004] [notice] child pid 2549 exit signal Segmentation fault (11)
[Thu May 13 14:15:48 2004] [notice] child pid 2548 exit signal Segmentation fault (11)
[Thu May 13 14:15:50 2004] [notice] child pid 2577 exit signal Segmentation fault (11)
[Thu May 13 14:15:54 2004] [notice] child pid 2581 exit signal Segmentation fault (11)
[Thu May 13 14:15:54 2004] [notice] child pid 2580 exit signal Segmentation fault (11)
[Thu May 13 14:15:56 2004] [notice] child pid 2592 exit signal Segmentation fault (11)
[Thu May 13 14:23:01 2004] [notice] child pid 2596 exit signal Segmentation fault (11)
[Thu May 13 14:23:01 2004] [notice] child pid 2595 exit signal Segmentation fault (11)
[Thu May 13 14:23:04 2004] [notice] child pid 4790 exit signal Bus error (10)
[Thu May 13 14:53:09 2004] [notice] child pid 4792 exit signal Segmentation fault (11)
[Thu May 13 14:56:13 2004] [notice] child pid 7330 exit signal Segmentation fault (11)
[Thu May 13 14:56:21 2004] [notice] child pid 7465 exit signal Segmentation fault (11)
[Thu May 13 14:56:21 2004] [notice] child pid 4793 exit signal Bus error (10)
[Thu May 13 14:56:23 2004] [notice] child pid 7473 exit signal Segmentation fault (11)
[Thu May 13 14:56:24 2004] [notice] child pid 7475 exit signal Segmentation fault (11)
[Thu May 13 14:56:24 2004] [notice] child pid 7474 exit signal Segmentation fault (11)
[Thu May 13 14:56:26 2004] [notice] child pid 7476 exit signal Segmentation fault (11)
[Thu May 13 14:56:27 2004] [notice] child pid 7480 exit signal Segmentation fault (11)
[Thu May 13 14:56:27 2004] [notice] child pid 7479 exit signal Segmentation fault (11)
[Thu May 13 14:56:29 2004] [notice] child pid 7485 exit signal Segmentation fault (11)
[Thu May 13 14:56:30 2004] [notice] child pid 7487 exit signal Segmentation fault (11)
[Thu May 13 14:56:30 2004] [notice] child pid 7486 exit signal Segmentation fault (11)
[Thu May 13 14:56:42 2004] [notice] child pid 7488 exit signal Segmentation fault (11)
[Thu May 13 14:56:43 2004] [notice] child pid 7495 exit signal Bus error (10)
[Thu May 13 14:56:44 2004] [notice] child pid 7497 exit signal Segmentation fault (11)
[Thu May 13 14:56:44 2004] [notice] child pid 7496 exit signal Segmentation fault (11)
[Thu May 13 14:56:46 2004] [notice] child pid 7498 exit signal Segmentation fault (11)
[Thu May 13 14:56:47 2004] [notice] child pid 7500 exit signal Segmentation fault (11)
[Thu May 13 14:56:47 2004] [notice] child pid 7499 exit signal Segmentation fault (11)
[Thu May 13 14:56:49 2004] [notice] child pid 7516 exit signal Segmentation fault (11)
[Thu May 13 14:56:50 2004] [notice] child pid 7518 exit signal Segmentation fault (11)
[Thu May 13 14:56:50 2004] [notice] child pid 7517 exit signal Segmentation fault (11)
[Thu May 13 14:56:52 2004] [notice] child pid 7519 exit signal Segmentation fault (11)
[Thu May 13 14:56:53 2004] [notice] child pid 7521 exit signal Segmentation fault (11)
[Thu May 13 14:56:53 2004] [notice] child pid 7520 exit signal Segmentation fault (11)
[Thu May 13 14:56:55 2004] [notice] child pid 7522 exit signal Segmentation fault (11)
[Thu May 13 14:56:56 2004] [notice] child pid 7523 exit signal Segmentation fault (11)
[Thu May 13 14:56:57 2004] [notice] child pid 7524 exit signal Segmentation fault (11)
[Thu May 13 14:56:58 2004] [notice] child pid 7525 exit signal Segmentation fault (11)
[Thu May 13 14:56:59 2004] [notice] child pid 7533 exit signal Segmentation fault (11)
[Thu May 13 14:56:59 2004] [notice] child pid 7532 exit signal Segmentation fault (11)
[Thu May 13 14:57:01 2004] [notice] child pid 7534 exit signal Bus error (10)
[Thu May 13 14:57:02 2004] [notice] child pid 7536 exit signal Segmentation fault (11)
[Thu May 13 14:57:02 2004] [notice] child pid 7535 exit signal Segmentation fault (11)
[Thu May 13 14:57:04 2004] [notice] child pid 7537 exit signal Segmentation fault (11)
[Thu May 13 14:58:37 2004] [notice] child pid 7539 exit signal Segmentation fault (11)
[Thu May 13 14:58:37 2004] [notice] child pid 7538 exit signal Segmentation fault (11)
[Thu May 13 14:58:39 2004] [notice] child pid 7725 exit signal Segmentation fault (11)
[Thu May 13 14:58:40 2004] [notice] child pid 7726 exit signal Segmentation fault (11)
[Thu May 13 14:58:42 2004] [notice] child pid 7728 exit signal Segmentation fault (11)
[Thu May 13 14:58:42 2004] [notice] child pid 7727 exit signal Segmentation fault (11)
[Thu May 13 14:58:44 2004] [notice] child pid 7742 exit signal Segmentation fault (11)
[Thu May 13 14:58:50 2004] [notice] child pid 7750 exit signal Segmentation fault (11)
[Thu May 13 14:58:50 2004] [notice] child pid 7749 exit signal Segmentation fault (11)
[Thu May 13 14:58:54 2004] [notice] child pid 7773 exit signal Segmentation fault (11)
[Thu May 13 14:58:55 2004] [notice] child pid 7776 exit signal Segmentation fault (11)
[Thu May 13 14:58:56 2004] [notice] child pid 7782 exit signal Segmentation fault (11)
[Thu May 13 14:58:56 2004] [notice] child pid 7781 exit signal Segmentation fault (11)
[Thu May 13 14:58:58 2004] [notice] child pid 7783 exit signal Segmentation fault (11)
[Thu May 13 14:58:59 2004] [notice] child pid 7785 exit signal Bus error (10)
[Thu May 13 14:58:59 2004] [notice] child pid 7784 exit signal Segmentation fault (11)
[Thu May 13 14:59:01 2004] [notice] child pid 7786 exit signal Segmentation fault (11)
[Thu May 13 14:59:02 2004] [notice] child pid 7788 exit signal Segmentation fault (11)
[Thu May 13 14:59:02 2004] [notice] child pid 7787 exit signal Segmentation fault (11)
[Thu May 13 14:59:04 2004] [notice] child pid 7789 exit signal

Any suggestions?

[Steven E. Protter]

I would suggest increasing shmmax which can be up to 25% of memory, which HP-UX defines as ram plus swap.

shmseg may be a factor.

maxusers which by default controls nproc and nfiles.

Also see maxuprc, it defaults to 75 and a heavy apache server may need more processes.

Also use mstm or xstm to check hardware.

SEP

[Global Server Operation]

Thanks. I'll review the parameters and adjust.

[Kent Ostby]

Also, you would want to review your systems overall swap space.

While this is not the most likely cause, it could be a factor if you are maxing it out.

HP recommendation is a minimum of disk swap space of 2x total ram.

Some DB heavy servers require 3x physical ram.

Best regards,

Kent M. Ostby

[A. Clay Stephenson]

You can play with kernel parameters and that may fix you (or at least better mask the problem) but the way to really diagnose this problem is through a debugger's stack trace. You can also get a hint of the problem using tusc.

The bad news about a stack trace is that without the source code, it's difficult to get precise information. If I really wanted to nail this puppy, I would download the source for Apache and compile with the -g option (this adds debugger data to the executable to make yoiur task easier) and turn off optimization. You then run this version of Apache and let it crash. You nowe have a version that is very easy to debug and it should take you to the exact offending line of code.

Apache is a rather easy build from source so although this may sound difficult, it's really not.