topic Re: packages can not failover when process down in Operating System - HP-UX

packages can not failover when process down

mk707 — Fri, 31 Aug 2018 23:18:42 GMT

I created 2 nodes cluster as failover type, and 4 packages, but my packages could not failover.
Could you please advice this issue?

node1 and node2 has AUTO_RUN setting.
all packages set aut_run yes, failover_policy configured_node.
pkg1 provides VIP with and monitored_subnet X.Y.Z.0.
pkg2 provides shared disk.
pkg3 provides ftp server as external_script with dependent on pkg1 up state and pkg2 up state.
external script invokes vsftpd using /etc/init.d/vsftpd start/stop.
pkg4 monitors vsftpd process and exit with status=1 when missing process.

I want to failover all packages when vsftpd down, but it was not.
When I killed vsftpd process, nothing happened. Only pkg4 move to down status, but the other packages were still running on node1.

I need another scripts or another settings to move all packages to standby node?

Best regards,
Keiko

Re: packages can not failover when process down

Bill Hassell — Sun, 02 Sep 2018 04:35:20 GMT

The logic used to monitor the availability of vsftpd may need some rework. The program vsftpd (https://security.appspot.com/vsftpd.html) seems to be a typical daemon which usually is scheduled through inittab with a respawn action in case the program disappears. This would be the preferred design to ensure that the daemon is always running.

If pkg4 detected the loss of vsftpd, then the other 3 packages need to be shutdown. So the settings you are using for all the packages needs some additional work.

Re: packages can not failover when process down

mk707 — Sun, 02 Sep 2018 04:56:51 GMT

Hi Bill, thank you for your advice.

I know Serviguguard package needs to "always running". At first, I defined /etc/init.d/vfstd as service_cmd in pkg3, but I got fail because /etc/init.d/vsftpd script immediately exited. (SG manual said this is bad definition.)
So, the second time, I temporally added sleep 1d at last of /etc/init./vsftpd and I put /etc/init.d/vsftpd into start function in external script. But this way did not success, because the package status was startting not up state.
The third time, I changed start function from my tempolally vsftpd to original /etc/init.d/vfstpd. Then pkg3 went to run state.
Now I recognized daemon program such as vsftpd must execute as external script.

My pkg4 watches vsftpd process. pkg4's function is as follows.

while true
do
echo `date`
# string=`ps -e | grep "vsftpd.conf"|grep -v grep` > /dev/null
string=`ps -aef | grep "vsftpd.conf"|grep -v grep`
if [ $? -eq 1 ]; then
exit 1
else
pid=`echo ${string} | awk '{print $1}'`
if [[ $pid = "" ]]; then
return 1
fi
fi
sleep 60s
done

Re: packages can not failover when process down

Bill Hassell — Sun, 02 Sep 2018 06:02:17 GMT

A few hints on your script:

The ps command can find a process by name (-C option). No need for the double grep.
The UNIX95=1 construct turns on extra options just for ps (see: man ps)

Not sure about vsftpd.conf. In the script below, I assume that the process you want to monitor is vsftpd.

while true
do
     UNIX95=1 ps -C vsftpd > /dev/null
     if [[ $? -ne 0 ]]
     then
        return 1
     fi
     sleep 60
done

Re: packages can not failover when process down

Patrick Wallek — Tue, 04 Sep 2018 14:13:35 GMT

Why do you have all of the different resources in different packages? This is not how a SG package is typically designed.

For a single service like this, you typically have your network, shared disk, running process, and process monitor all in a single SG package. Then if the process goes down the entire package, encompassing all needed resources, fails over to the other node.

Re: packages can not failover when process down

mk707 — Tue, 04 Sep 2018 22:16:10 GMT

Hi Patrick

Thank you for your advice,

I understand SG design by your advice. I will try it.

Keiko

Re: packages can not failover when process down

mk707 — Tue, 04 Sep 2018 22:24:06 GMT

Hi Bill

I could pickup ftp process perfectly. Thank you.