httpd package keeps halting on simple cluster

[Iain Soedring]

This is on HP11i v2, Serviceguard 11.17.
I've got a cluster up and running, but when I try to start the package which should start apache (httpd), it fails immediately.

The package log contains the following:

########### Node "hiptc079": Starting package at Wed Sep 24 17:12:38 BST
2008 ###########
Sep 24 17:12:38 - Node "hiptc079": Adding IP address 9.20.51.245 to subnet 9.20.
50.0
Sep 24 17:12:39 - Node "hiptc079": Starting service httpd using
"/opt/hpws/apache/bin/httpd -k start"
/usr/lib/hpux64/dld.so: End of liblist, invalid RTLD_NEXT argument.

########### Node "hiptc079": Package start completed at Wed Sep 24 17:12
:39 BST 2008 ###########

########### Node "hiptc079": Halting package at Wed Sep 24 17:12:39 BST
2008 ###########
Sep 24 17:12:39 - Node "hiptc079": Halting service httpd
cmhaltserv: Service name httpd is not running.
Sep 24 17:12:39 - Node "hiptc079": Remove IP address 9.20.51.245 from subnet 9.2
0.50.0

########### Node "hiptc079": Package halt completed at Wed Sep 24 17:12:
39 BST 2008 ###########


Obviously it looks like there's an error there when starting httpd, but this is a red-herring, httpd still starts ok. In fact, the httpd process are still running on the server after this failure:
hiptc079: 23 : ps -ef | grep http
www 26655 26639 0 17:12:39 ? 0:00 /opt/hpws/apache/bin/httpd -k start
www 26652 26639 0 17:12:39 ? 0:00 /opt/hpws/apache/bin/httpd -k start
root 26639 1 0 17:12:39 ? 0:00 /opt/hpws/apache/bin/httpd -k start
www 26644 26639 0 17:12:39 ? 0:00 /opt/hpws/apache/bin/httpd -k start
/etc/cmcluster/apach

What's strange is all these processes have the '-k start' in them, if I start httpd from the command line, then they don't.

The syslog.log contains:

Sep 24 17:12:38 hiptc079 cmcld[10988]: Request from node hiptc068 to start packa
ge iainpackage on node hiptc079.
Sep 24 17:12:38 hiptc079 cmcld[10988]: Executing '/etc/cmcluster/apach/apach.sh
start' for package iainpackage, as service PKG*63489.
Sep 24 17:12:38 hiptc079 CM-CMD[26630]: cmmodnet -a -i 9.20.51.245 9.20.50.0
Sep 24 17:12:39 hiptc079 CM-CMD[26635]: cmrunserv httpd >> /etc/cmcluster/apach/
apach.sh.log 2>&1 /opt/hpws/apache/bin/httpd -k start
Sep 24 17:12:39 hiptc079 cmcld[10988]: Service PKG*63489 terminated due to an ex
it(0).
Sep 24 17:12:39 hiptc079 cmcld[10988]: Started package iainpackage on node hiptc
079.
Sep 24 17:12:39 hiptc079 cmcld[10988]: Service httpd terminated due to an exit(0
).
Sep 24 17:12:39 hiptc079 cmcld[10988]: Service httpd in package iainpackage has
gone down.
Sep 24 17:12:39 hiptc079 cmcld[10988]: Disabled node hiptc079 from running packa
ge iainpackage.
Sep 24 17:12:39 hiptc079 cmcld[10988]: Executing '/etc/cmcluster/apach/apach.sh
stop' for package iainpackage, as service PKG*63489.
Sep 24 17:12:39 hiptc079 cmomd[26479]: Successfully started package iainpackage
on node hiptc079.
Sep 24 17:12:39 hiptc079 CM-CMD[26648]: cmhaltserv httpd
Sep 24 17:12:39 hiptc079 CM-CMD[26651]: cmmodnet -r -i 9.20.51.245 9.20.50.0
Sep 24 17:12:39 hiptc079 cmcld[10988]: Service PKG*63489 terminated due to an ex
it(0).
Sep 24 17:12:39 hiptc079 cmcld[10988]: Halted package iainpackage on node hiptc0
79.

The package script apach.sh has the following parameters set:

IP[0]=9.20.51.245
SUBNET[0]=9.20.50.0

SERVICE_NAME[0]=httpd
SERVICE_CMD[0]="/opt/hpws/apache/bin/httpd -k start"
SERVICE_RESTART[0]=""

Also, I set the 'Listen' parameter in httpd.conf to be the relocatable ip address 9.20.50.245

Other potentially interesting params, in the package .config file:

SERVICE_NAME httpd
SERVICE_FAIL_FAST_ENABLED NO
SERVICE_HALT_TIMEOUT 5

Could starting it with 'httpd -k start' be the problem?

Any ideas on this?

[Kenan Erdey]

Hi,

could you try starting with /sbin/init.d/apache start.

[Steven E. Protter]

Shalom,

Looks like a a problem in the package configuration file.

invalid RTLD_NEXT argument.

If apache runs normally without SG, its the package configuration script.

SEP

[Iain Soedring]

I fixed the RTLD_NEXT error, but still find it is halting immediately on startup:

Sep 25 11:13:29 - Node "hiptc068": Starting service httpd using
"/opt/hpws/apache/bin/httpd -k start"

########### Node "hiptc068": Package start completed at Thu Sep 25 11:13
:29 BST 2008 ###########

########### Node "hiptc068": Halting package at Thu Sep 25 11:13:29 BST
2008 ###########
Sep 25 11:13:29 - Node "hiptc068": Halting service httpd
cmhaltserv: Service name httpd is not running.

[Iain Soedring]

Made some progress, setting:

SERVICE_RESTART[0]="-R"

Httpd stays running in this situation, and the relocatable ip address stays active. However the server is continually restarting, and the log file is getting bigger and bigger, filling up with these messages:

hiptc068: 119 : tail -f apach.sh.log
httpd (pid 1534) already running
httpd (pid 1534) already running
httpd (pid 1534) already running
...
...

This is kind of a brute force way of getting it working, and obviously I can't leave it like this.

Still no idea why httpd halts itself.

[Iain Soedring]

Solved this problem by installing and using the HP Serviceguard "Enterprise Cluster Master Toolkit", this has readymade scripts for running apache in a cluster, and a README that takes you through all the steps to do this.