Cluster Troubleshooting

[Eric Jacklin]

Hi

Would you please provide me the document which can help me out to do step by step Trouble shoot of cluster.

Cluster : MCSG
OS : 11.11


Regards,
Jitendra Patel

[Suraj K Sankari]

Hi,

For trouble shooting the cluster there is no such doc.

You need to read the full pdf doc providing by HP.
here is the link

http://docs.hp.com/en/oshpux11iv3.html#Serviceguard

Suraj

[Duncan Edmonstone]

Actually there's a chapter in the manual that covers this:

http://docs.hp.com/en/B3936-90079/ch08.html

Or you could share your problem with us...

HTH

Duncan

[Md. Farhan A Azam]

Hello,

Check this doc...

http://docs.hp.com/en/B3936-90100/B3936-90100.pdf

[sujit kumar singh]

hi


the cluster truoble shooting u need to refer to the log files well


the system LOG file that is the SYSLOG :
/var/adm/syslog/syslog.log

and the correcponding package control files:

/etc/cmcluster/pkg1/pkg1.cntl.log

on all the nodes that are the member of the cluster.


hoever these MCSG commands help u understand and know about the states of the cluser

#cmviewcl
#cmviewcl -v
#cmviewcl -l package -l node

etc

also u can use the commands as

#cmviewconf
this u can use even if the cluster is down

#cmscancl


refer to the manpages of these commands for more details and also remeber that basic of the SG is a standard but u have to know that there can diff types of cluster like RAC, SGeSAP, Metro anc Continental.


with the basic tools that i have stated above and the understanding the geometry of the cluster u can become a master of the Cluster Troubleshooting.


Regards
Sujit

[Stephen Doud]

Chapter 8 in the Managing Serviceguard manual at http://docs.hp.com/en/ha.html#Serviceguard is devoted to troubleshooting.
Problems with Serviceguard range across the board in installation, configuration, administration and update trouble. No troubleshooting document can cover all issues, so this ITRC forum was devised to help you, your peers and HP moderators look at current topics for resolution.
As stated before, if you have a specific issue, you can read the manual and use the ITRC knowledge database to begin your research. After that, check here for assistance and if that fails to get you a solution, open a software support case with HP.

[sdass]

Here are some typical examples:
# cmviewcl
cmviewcl : Cannot view the cluster configuration.
Either this node is not configured in a cluster, user
doesn't have
access to view the cluster configuration, or there is some
obstacle
to viewing the configuration. Check the syslog file for more
information.
For a list of possible causes, see the Serviceguard manual
for cmviewcl.
# cmviewconf
cmviewconf: Unable to get cluster configuration information.
Unable to open communications to configuration daemon:
Connection refused
Unable to connect to configuration database.
# cmviewconf
cmviewconf: Either binary file does not exist, or the user
doesn't
have access to view the cluster configuration.
# cmhaltpkg dbciP01
cmhaltpkg : Unable to open handle to local cluster
Either no cluster configuration file exists, the file is
corrupted,
cmclconfd is unable to run, or user root on node nero
doesn't have access to view the configuration.
# cmviewcl
CLUSTER STATUS
alwayson up
Failed to get dlm configuration.


Troubleshooting cheklist

1. Do you try to run a SG command on a node not having SG
configured and that accesses a node that is running in a cluster?
E.g. you run cmcheckconf to add a node that is not currently
member of the cluster and you run the cmcheckconf on the node
that is to be added? For SG A.11.16 and later you should run the
command on the node that has SG already configured. Otherwise
you would need to change the Role Based Access Policies in the
cluster.ascii of the existing cluster to allow external nodes to
modify the configuration
2. /usr/lbin/cmclconfd exists and has appropriate execution rights.
It's cksum and what string match what is documented (e.g. in
patch texts for SG).
# ls -l /usr/lbin/cmclconfd
-r-xr--r-- 1 bin bin 3725848 Mar 14
2005 /usr/lbin/cmclconfd
3. The hacl-cfg/tcp and hacl-cfg/udp ports are listed in /etc/services.
In a NIS environment the command "ypcat services" list the
ports.
# grep hacl-cfg /etc/services
hacl-cfg 5302/tcp # HA Cluster TCP configuration
hacl-cfg 5302/udp # HA Cluster UDP configuration
4. The hacl-cfg/tcp and hacl-cfg/udp ports are listed
in /etc/inetd.conf on HP-UX as
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd
cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd
cmclconfd -c
5. inetd is running and it registered the ports correctly when it was
restarted last
# grep hacl-cfg /var/adm/syslog/syslog.log
Aug 22 14:25:30 nero inetd[980]: hacl-cfg/udp: Added
service, server /usr/lbin/cmclconfd
Aug 22 14:25:30 nero inetd[980]: hacl-cfg/tcp: Added
service, server /usr/lbin/cmclconfd
Netstat -an shows that inetd is listening on hacl-cfg/tcp.
# netstat -an | grep 5302 | grep LISTEN
tcp 0 0 *.5302 *.* LISTEN
6. Make sure that /var/adm/inetd.sec does not deny access for
cluster nodes to hacl-cfg ports.
7. Make sure the subnet masks for a subnet have the same value on
all nodes
# ifconfig lan0
lan0: flags=843
inet 16.25.249.67 netmask ffffff00 broadcast
16.25.249.255
If the subnet mask is oncorrect, the udp broadcast sent by most
commands may not reach other nodes resulting in a "refusing
communications" message
8. Enable inetd connection logging (inetd -l) to verify that a local
SG command (e.g. cmviewconf) connects to inetd which in turn
starts the cmclconfd server process.
# tail -f /var/adm/syslog.log &
# inetd -l
# Apr 20 14:57:14 nero inetd[1189]: Connection logging
enabled
# cmviewconf > /dev/null 2>&1
Apr 20 14:57:47 nero inetd[1395]: hacl-cfg/tcp:
Connection from localhost (127.0.0.1) at Fri Apr 20
14:57:47 2007
Apr 20 14:57:48 nero inetd[1396]: ident/tcp: Connection
from localhost (127.0.0.1) at Fri Apr 20 14:57:47 2007
If there are no messages for connection
9. Firewall software (like IP Filter) must not disallow access to haclcfg
ports from and to all other cluster nodes on all IP addresses the cluster nodes can potentially talk on
10. Make sure to list all cluster IP addresses in /etc/hosts. Make sure
the cluster IP addresses are listed at the top of the file. Use
an /etc/nsswitch.conf file of the form:
hosts: files [NOTFOUND=continue] dns
11. For SG versions SG A.11.15 and earlier: Make sure
either /etc/cmcluster/cmclnodelist or if this file does not exist
$HOME/.rhosts contain the IP addresses of all cluster nodes and
of all subnets the nodes can potentially talk on (not only those
configured in the cluster binary /etc/cmcluster/cmclconfig). Make
sure these SG versions use most recent patch levels. If you use $HOME/.rhosts make sure to list the cluster IP addresses at
the top of the file.

[Eric Jacklin]

Thanks