Alert if the package and node is down.

[Srinikalyan]

Hello to all,
I have a 2 node Active active configuration cluster based around Oracle RAC and SG. I need to send an email to admin if there is any failure in the cluster i.e. either package failure or the node itself and the packages currently active in the node etc. Any idea on which method I can use so that I will get the required functionality. Any sample scripts will help me to proceed in best way.

Thanks,
Srini

[Ivan Krastev]

You can use 2 methods:
1. User script in the package control file - when package goes down, script will send an email.
2. Crontab script to monitor package/cluster status every 5-10 min.


regards,
ivan

[Srinikalyan]

Thanks Ivan.
Any sample scripts which can be used in general to know the status of all the packages? I can run cmviewcl -v -p pkg_name |grep current|awk '{print $4}' to identify which node is running that particular package. I need the generailzed script to know the status of all the packages.
Regards,
Srini

[Kenan Erdey]

hi,

why don't you just use cmviewcl command ?

[Srinikalyan]

Hi,
Currently I have a script like

primary=prinode
running=`cmviewcl -v -p prodnfs |grep current | awk '{print $4}'`
if [ "$running" = "$primary" ]
then
exit
else
/usr/bin/bttrap sysFailureMajor application is not running on its Primary Node.Contact admin Immediately
fi

What I need is intead of one package, I need the status of all the packages in the cluster even if I add a new package.

Regards,
Srini.

[Kenan Erdey · Accepted solution]

Hi,
Hope it helps:

for pkg_name in $(cmviewconf | grep 'package name'| awk '{print $3}')
do
running_node=$(cmviewcl -v -p $pkg_name | grep current | awk '{print $1}');


if [ "$running_node" = "Primary" ]
then
//do something if package is running in it's primary node
else
//do something if package is not running in it's primary node
fi
done

for pkg_name in $(cmviewcl | grep 'down' | awk '{print $1}')
do
//do something with $pkg_name
done

[Stephen Doud]

The goal is to know when a package is legitimately down.
Using the package control script to spawn an email when a package is halted may not produce the results you need. There are two cases that come to mind
1) node outage = no way the package halt script will be run
2) manual package maintenance, moving it to another node

Though not as timely, a cron job may produce more legitimate information.

To determine package status, check this:

To find all package status
# cmviewcl -v -f line | grep ^package: | grep "|status=" | grep -v node:
package:SG-CFS-pkg|status=up
package:CFSdg|status=up
package:CFSdg-vol1|status=up
package:CFSdg-vol2|status=up
package:CFSdg-vol3|status=up
package:P1|status=down
package:p2|status=down
package:henry|status=up
package:testpkg|status=up

To filter just the down packages:
# cmviewcl -v -f line | grep ^package: | grep "|status=" | grep -v node: | grep =down
package:P1|status=down
package:p2|status=down

And to get just the names of the packages that are down:

# cmviewcl -v -f line | grep ^package: | grep "|status=" | grep -v node: | grep =down | cut -d : -f 2,2 | cut -d "|" -f 1,1
P1
p2


Now, you can write a script to send an email if any results return, and link it to a cron job.

[Srinikalyan]

Thanks Stephen and Kenan,
Stephen,
cmviewcl -f line -l package|grep -e status give only the status of the packages but not on the node it's running. Any other formatting to acheive this?
Kenan,
How can I eliminate the multi-node packages in the output.
Regards,
Srini

[Srinikalyan]

I arrived with this solution.
#!/usr/bin/sh
for pkg_name in $(cmviewcl -f line -l package|grep -e type|grep -v multi_node|cut -d : -f 2,2 | cut -d "|" -f 1,1)
do
running_node=$(cmviewcl -v -p $pkg_name | grep current | awk '{print $1}');
if [ "$running_node" = "Primary" ]
then
echo $pkg_name running in Primary
else
echo $pkg_name not running in Primary
fi
done

[Aashique]

Hi,
check this script

#!/usr/bin/ksh
##########################
DATE=`date +%Y.%m.%d`
TIME=`date +%X`
cmviewcl -v >/tmp/clstatus-file

#Checking TESTPKG
check1=`more clstatus-file|awk '{if($1=="testpkg")
if($5=="node1") print "TEST IS FAILOVER"
else if($2=="down") print "TEST IS HALTED"
else continue}'`


if [ "$check1" ]
then
mailx -s "$check1-$DATE-$TIME" admin@node1.com
//send sms also
fi
====================================
set this script in ur cron in every 5 min.

Thanks & Regards
Aashique

[Srinikalyan]

Hi,
I wrote the generalised script for the required functionality.

#!/usr/bin/sh
if [ -f /tmp/output.txt ]
then
rm /tmp/output.txt
fi

for pkg_name in $(cmviewcl -f line -l package|grep -e type|grep -v multi_node|cut -d : -f 2,2 | cut -d "|" -f 1,1)
do
running_node=$(cmviewcl -v -p $pkg_name | grep current | awk '{print $1}');
node_name=$(cmviewcl -v -p $pkg_name | grep current | awk '{print $4}')
if [ "$running_node" = "Primary" ]
then
echo $pkg_name running in Primary node $node_name. >> /tmp/output.txt
elif [ "$(cmviewcl -f line -l package -p $pkg_name|grep -e status|cut -d = -f 2,2)" = "down" ]
then
echo $pkg_name package totally down. Not running in Primary $node_name as well as secondary $node_name. >> /tmp/output.txt
else
echo $pkg_name running in Secondary node $node_name. >> /tmp/output.txt
fi
done
#mailx -s "Cluster status" oracle < /tmp/output.txt

[Srinikalyan]

Thanks everyone who contributed.