service guard question

[sheevm]

Hi,

I am new to serviceguard. We are planning to implement SG in production envirionment for Oracle database server and another application server. 2 nodes cluster for DB server and 2 node cluster for application in active/paasive mode.

Does anyone out there can provide me some sample package configuration/control scripts.

I am reading a lot about this subject.
Still it is unclear how does the service guard monitor the application/database?

Do we need to write the monitoring scripts? If we do how do I incorporate this in the serviceguard to detect the application failure?

Does anyone can provide me a sample service guard configuration file that monitors/stops/starts the application and/or oracle database

Thanks
Rajim

[Sung Oh]

Hello,

here is SG sample overview.
http://www.phptr.com/content/images/0131463969/samplechapter/0131463969_ch25.pdf

http://h71028.www7.hp.com/enterprise/downloads/SG%20mixed%20cluster_WP102704.pdf

Regards,
Sung

[Michael Steele_2]

ServiceGuard comes with a template to file in beginning with your cluster config., cmclconfig.ascii. In the broad strokes you begin on your node1 building the cluster. Then you build your volume groups and ftp the map over to nodeB which rebuilds the volume groups from the map.

Here's a brief outline of the step, it is by no way complete. In fact, I'd take the ServiceGuard class before attempting it.

1. cmquerycl -C cmclconfig.ascii -n node1 -n node2 -n node3, etc.
2. vi cmclconfig.ascii (fill in template)
3. cmcheckconf -C cmclconfig.ascii
4. cmaplyconf -C cmclconfig.ascii.
5. cmviewcl -v (verify report status)
6. on node1 make vol. groups.
7. export to node2,
vgchange -a n /dev/vg
vgexport -p -s -m /temp/vgmap /dev/vg
ftp /tmp/vgmap node2
8. mkdir, mknod then
vgexport -s -m /tmp/vgmap /dev/vg

9. mount file systems
10. create database
11. create package configuration file
12. create package run/halt script on node1
13. create package monitor script. (* ...Do we need to write monitoring scripts...*)
14. copy run/halt scripts to node2.
15. finish package configuration
16. test failover

[sheevm]

Michael, Thanks for the brief outline.

I have questions on steps 12 and 13.

run/halt/monitor scripts are created outside the service guard correct?

Do I call this script in the package configuration file? Does the serviceguard runs the monitoring scripts on certain intervals to detect the failure?

I am not in a position to attend the classes before implementation.

Any help is appreciated.

The links I received does not work.

Thanks.

[Geoff Wild · Accepted solution]

"Do we need to write the monitoring scripts? If we do how do I incorporate this in the serviceguard to detect the application failure? "

Yes - you do need to write monitoring scripts (or use ones that come with ServiceGuard - see /opt/cmcluster )

How to incorporate?" add a service to the package control script:

http://docs.hp.com/en/B3936-90079/ch06.html

Example:

In the cntl file:

SERVICE_NAME[0]="iprprtpkg_mon"
SERVICE_CMD[0]="/etc/cmcluster/IPRPRT/lp-script monitor"
SERVICE_RESTART[0]="-r 3"


# cat lp-script
#!/bin/sh
#
# iprprt startup/shutdown/monitor script
# Geoff Wild
#

LOCKF=/tmp/iprprt.lock
PACKAGE_NAME=iprprtpkg
HOST=`hostname`
DATE=`/usr/bin/date`
MONITOR_INTERVAL=60
set -A MONITOR_PROCESSES lpsched


###############################################################################
# Function: iprprt_run_cmds
#
# Start the HP LP Spooler
#
###############################################################################

function iprprt_run_cmds
{

echo "Checking to see if package is already running..."

STATE=`cmviewcl | grep iprprt | awk '{ print $2 }'`
NODE=`cmviewcl | grep iprprt | awk '{ print $5 }'`
if [[ $STATE = "down" ]]
then
echo "${PACKAGE_NAME} is down...starting up ${PACKAGE_NAME}"
else
if [[ $STATE = "up" ]]
then
if [[ $NODE = $(/usr/bin/hostname) ]]
then
print "${PACKAGE_NAME} is running...on the current ${HOST}: \c"
print "halting ${PACKAGE_NAME} on $NODE"
/usr/sbin/cmhaltpkg ${PACKAGE_NAME}
fi
fi
fi

lockfile=/var/spool/lp/SCHEDLOCK
if [ -f $lockfile ];
then
echo "looks like we have $lockfile, try to shut spooler first..."
/usr/sbin/lpshut
sleep 5
if [ -f $lockfile ];
then
echo "$lockfile still there but the spooler was not running, removing the lockfile..."
rm $lockfile
fi
fi
echo "Start the lp spooler..."
/usr/sbin/lpsched -v
sleep 10
SCHED=`/usr/bin/lpstat -r | awk '{print $3}'`
if [[ $SCHED = "not" ]]
then
echo "lp spooler not running..."
else
echo "lp spooler started successfully."
fi

}

###############################################################################
# Function: iprprt_shutdown_cmds
#
# Stop the HP LP Spooler
#
###############################################################################

function iprprt_shutdown_cmds
{

lockfile=/var/spool/lp/SCHEDLOCK
if [ -f $lockfile ];
then
echo "looks like we have $lockfile, try to shut spooler first..."
/usr/sbin/lpshut
sleep 5
if [ -f $lockfile ];
then
echo "$lockfile still there but the spooler was not running, removing the lockfile..."
rm $lockfile
fi
fi
sleep 10
SCHED=`/usr/bin/lpstat -r | awk '{print $3}'`
if [[ $SCHED = "not" ]]
then
echo "lp spooler stopped successfully."
else
echo "lp spooler is still running!!!"
set -m
nohup ${0} fault & # The script calls itself with the fault option.
set +m
sleep 999999

fi

}

###############################################################################
# Function: monitor_processes
#
# Monitor the HP LP Spooler processes by making sure that all required
# processes are running.
###############################################################################

function monitor_processes
{

print "\n *** getting pids ***"
get_monitor_processes_pid
sleep ${MONITOR_INTERVAL}
while true
do
if [[ -f ${LOCKF} ]]
then
no_monitor_processes
else

for i in ${MONITOR_PROCESSES_PID[@]}
do
kill -s 0 ${i} > /dev/null
if [[ $? != 0 ]]
then
print "\n\n"
ps -ef
print "\n *** ${i} has failed. Aborting HP LP Spooler. ***"
set -m
nohup ${0} fault & # The script calls itself with the fault option.
set +m
sleep 999999
fi
done
sleep ${MONITOR_INTERVAL}
fi
done

}

###############################################################################
# Function: no_monitor_processes
#
# Do NOT Monitor the HP LP Spooler processes
#
##############################################################################

function no_monitor_processes
{
while [[ -f ${LOCKF} ]]
do
DATE=`/usr/bin/date`
print " ${DATE} *** $LOCKF exists? - monitoring disabled. ***"
sleep ${MONITOR_INTERVAL}
done
print "\n *** getting new pids ***"
get_monitor_processes_pid
sleep ${MONITOR_INTERVAL}
}

###############################################################################
# Function: get_monitor_processes_pid
#
# get the pid's of all the processes we are using
#
##############################################################################

function get_monitor_processes_pid
{
typeset -i n=0

for i in ${MONITOR_PROCESSES[@]}
do

# MONITOR_PROCESSES_PID[$n]=`ps -fu lp | awk '/'${i}$'/ { print $2 }'`
# We only want lpsched if the PPID is 1
MONITOR_PROCESSES_PID[$n]=`ps -fu lp | awk '/'${i}'/ { if ( $3 == 1 ) print $2 }'`
print "Monitored process = ${i}, pid = ${MONITOR_PROCESSES_PID[$n]}"
if [[ ${MONITOR_PROCESSES_PID[$n]} = "" ]]
then
print "\n\n"
ps -ef
print "\n *** ${i} has failed at startup time. Aborting HP LP Spooler. ***"
set -m
nohup ${0} fault & # The script calls itself with the fault option.
set +m
sleep 999999
fi
(( n = n + 1 ))

done
}

###############################################################################
# Function: halt_package
#
# Because there is no move command in MC/Serviceguard, the package must first
# be halted, disabled from running on the host, and then enabled to run in the
# cluster.
###############################################################################

function halt_package
{
/usr/sbin/cmhaltpkg ${PACKAGE_NAME}
/usr/sbin/cmmodpkg -d -n ${HOST} ${PACKAGE_NAME}
sleep 1
/usr/sbin/cmmodpkg -e ${PACKAGE_NAME}
}


###############################################################################
# MAIN
#
# Check the command-line option and take the appropriate action.
###############################################################################

print "\n *** $0 called with $1 argument. ***\n"

case $1 in

fault)
halt_package
;;

monitor)
monitor_processes
;;

start)
print "\n \"${HOST}\": Starting the lp spooler package at ${DATE} "
iprprt_run_cmds
;;


shut)
print "\n \"${HOST}\": Shutting down the lp spooler package at ${DATE} "
iprprt_shutdown_cmds
;;

*)
print "Usage: ${0} [ shut | start | monitor ]"
;;
esac




Rgds...Geoff

[melvyn burnard]

for Oracle it is best to use the toolkit supplied with the Enterprise Cluster Master Toolkit, this makes incorporating Oracle into a package far easier, and supplies the monitor as well.
If you have Mission Critical OE, then ECMT is included

[Michael Steele_2]

Within the oracle.pkg package configuration file there are data fields that point to oracle monitoring.

SERVICE_NAME
SERVICE_FAIL_FAST_ENABLED
SERVICE_HALT_TIMEOUT

Within the oracle package 'script' which is separate from the config. file 'oracle.cnt1', for example

SERVICE_NAME[0]="ORCLE1.SH"
SERVICE_CMD[0]="..../ORACLE1.SH MONTIOR"

copy to nodeB
cmcheckconf
cmapplyconf
cmmodpkg -e oracld1 (package)

Test failover

su - oracle
test.sql

[sheevm]

Thanks to all.

Special Thanks to Geoff for the scripts.

I will review and seek your help if I have more questions.

Can you guys suggest any good practical book that comes close to the HP servicegurd training class?

Thanks.
Raji