Node is refusing Serviceguard communication

[Kehad Snydewel]

Hi all,
I am busy with a new serviceguard 11.18 installation on HPUX 11.23 (2 node cluster).
I successfully created the cluster (cmquerycl, cmcheckconf, cmapplyconf). Did not configure the packet yet. I had to change the hostnames of both nodes and did it in the following manner:

-Cmhaltcl -v

Both nodes:
-set _parms hostname
-change hostnames and reboot

Node 1 and 2 :
-Changed the /etc/hosts, /.rhosts, /etc/cmcluster/cmclnodelist and /etc/cmcluster/cmclconf.ascii files to reflect the new node names.
-cmcheckconf –C cmclconf.ascii
Node olprd1 is refusing Serviceguard communication.
Please make sure that the proper security access is configured on node
olprd1 through either file-based access (pre-A.11.16 version) or role-based
access (version A.11.16 or higher) and/or that the host name lookup
on node olprd1 resolves the IP address correctly.
cmcheckconf: Failed to gather configuration information

/etc/hosts of both nodes
127.0.0.1 localhost loopback
192.168.250.163 olprd1 #LAN1
192.168.250.164 olprd2 #LAN1
10.10.10.10 hb1 # heartbeat of olprd1 LAN0
10.10.10.11 hb2 # heartbeat of olprd2 LAN0
I ‘ve gone through all the forums and checked everything but still the same error. i tried to cmcheckcl –C cmconf.ascii on both nodes but no luck. On both node I receive the same error and refers to the same node which is:
Node olprd1 is refusing Serviceguard communication.

I also tried cmquerycl –v –C cmclconfig.ascii -n olprd1 -n olprd2 on both nodes and recived the same error:
Node1 (olprd1)
Node olprd2 is refusing Serviceguard communication
Node2 (olprd2)
Node olprd1 is refusing Serviceguard communication.
I renamed the cmclconfig binary on both nodes as well but still the same error.

Any advice???

[skt_skt]

verify your hacl entries in /etc/inetd.conf and /etc/services. (restart the inet daemon).

Could put the exact error from the syslog.?

[Kehad Snydewel]

i checked for this and it seems that all hacl and auth entries exist on both nodes.

i do not have the syslog entries at the moment but will provide this as well later.

i also tried to remove both nodes from cmclnodelist and only add a +. and also tried to add the nodes and +.

first attempt/try
cat cmclnodelist
olprd1 root
olprd2 root

second attempt/try
cat cmclnodelist
olprd1 root
olprd2 root

third attempt/try
cat cmclnodelist
+


root@olprd1:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd1


root@olprd2:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd2


root@olprd1:/etc/cmcluster# cat /etc/inetd.conf |grep -i hacl
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmm
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c


root@olprd2:/etc/cmcluster# cat /etc/inetd.conf |grep -i hacl
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmm
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c


root@olprd1:/etc/cmcluster#cat /etc/inetd.conf |grep -i auth
auth stream tcp6 wait bin /usr/lbin/identd identd


root@olprd2:/etc/cmcluster#cat /etc/inetd.conf |grep -i auth
auth stream tcp6 wait bin /usr/lbin/identd identd



root@olprd1:/etc/cmcluster# cat /etc/services |grep -i hacl
hacl-hb 5300/tcp # High Availability (HA) Cluster heartbeat
hacl-gs 5301/tcp # HA Cluster General Services
hacl-cfg 5302/tcp # HA Cluster TCP configuration
hacl-cfg 5302/udp # HA Cluster UDP configuration
hacl-probe 5303/tcp # HA Cluster TCP probe
hacl-probe 5303/udp # HA Cluster UDP probe
hacl-local 5304/tcp # HA Cluster Commands
hacl-test 5305/tcp # HA Cluster Test
hacl-dlm 5408/tcp # HA Cluster distributed lock manager


root@olprd2:/etc/cmcluster# cat /etc/services |grep -i hacl
hacl-hb 5300/tcp # High Availability (HA) Cluster heartbeat
hacl-gs 5301/tcp # HA Cluster General Services
hacl-cfg 5302/tcp # HA Cluster TCP configuration
hacl-cfg 5302/udp # HA Cluster UDP configuration
hacl-probe 5303/tcp # HA Cluster TCP probe
hacl-probe 5303/udp # HA Cluster UDP probe
hacl-local 5304/tcp # HA Cluster Commands
hacl-test 5305/tcp # HA Cluster Test
hacl-dlm 5408/tcp # HA Cluster distributed lock manager

[Matti_Kurkela]

ServiceGuard A.11.18 uses the bootstrap authentication file /etc/cmcluster/cmclnodelist only when cluster configuration does not exist yet.

The old configuration contains the old host names and may be interfering with your cmapplyconf.

You may have to run "cmdeleteconf" on all your nodes to remove all traces of the old configuration, then cmapplyconf to apply your new cluster configuration as if you were setting up a new cluster.

MK

[Kehad Snydewel]

this is what i get when issueing the command cmdeleteconf on both nodes:

NODE1
root@olprd1:/etc/cmcluster#cmdeleteconf
cmdeleteconf: Unable to get cluster configuration information: Permission denie.

NODE1 SYSLOG
Apr 30 13:01:19 olprd1 syslog: cmdeleteconf -v
Apr 30 13:01:19 olprd1 cmclconfd[29575]: Permission denied for root@olprd1 (RBA)
Apr 30 13:01:26 olprd1 syslog: cmdeletecon


NODE2
root@olprd2:/etc/cmcluster# cmdeleteconf
Checking current status
Unable to perform the security token exchange with cmclconfd on node olprd2
Unable to perform the security token exchange with cmclconfd on node OLPRD1
Unable to refresh view of cluster olclu: No such file or directory

NODE2 SYSLOG has no entries

[Eric SAUBIGNAC]

Bonjour,

I would do the following (2 ideas)

A) stop cluster, revert back all nodes to their old name, reboot, then cmdeleteconf

change both servers to new names, reboot.

modify cluster configuration file, packages configuration files, check and apply the new conf.

B) stop cluster if running (?) (cmhaltcl) , then "/sbin/init.d/cmcluster stop"

remove on all nodes /etc/cmcluster/cmclconfig
restart "/sbin/init.d/cmcluster start"

modify cluster configuration file, packages configuration files, check and apply the new conf.


Regards

Eric

[Mridul Shrivastava]

Did you set any parameters in identd ?
You may need to add -i
Then don't forget to inetd -c

Other ideas:
Have you used /var/adm/inetd.sec ?
If so, you will have to tailor it.

Have you commented out any hacl... services in /etc/services?

If you use cmclnodelist then also add 127.0.0.1 root and all the other IP addresses that you want to use (e.g. heartbeat lans etc). Do this on all nodes.

Set the permissions of cmclnodelist to 644 on all nodes.

Check the contents of /etc/rc.config.d/cmcluster
- you will have to change that eventually anyway.

[Rita C Workman]

I just have one question first......

From your HPUX box run a reverse lookup:
nslookup
nslookup

Did it resolve properly ?

If you use DNS (no matter what runs DNS) from a PC command line do the same thing and run a reverse lookup:

Did it resolve exactly the same as the one from HPUX?

Thanks,
Rita

[Eric SAUBIGNAC]

I would like to underline one thing before leaving : since MC/SG A.11.15 (or A.11.16, I don't remember exactly) the concept of "Access Control Policies" has been introduced in MC/SG. And since this modification, once a cluster is created, host name is a real source of problem when trying to alter cluster configuration. I am pretty sure that your problem is there (as already said by Matti) and not in configuration of inetd.sec, services, and anything else.

So to avoid wasting time, I think the better way for you is to delete this cluster then create it again. Really.

Eric

[Kehad Snydewel]

Mridul
i do not have an entry in my inetd.conf file with -i.

root@olprd1:/etc/cmcluster# cat /etc/inetd.conf |grep -i hacl
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmm
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c


Other ideas:
Have you used /var/adm/inetd.sec ?
If so, you will have to tailor it.

how should i edit this file because i am not using it:

root@olprd1:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd1

root@olprd2:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd2

i did reverse lookups and eveything resolved properly

i think i will just reinstall the OS and Serviceguard on both nodes becuase i would like to configure the 2x BL860c with hardware RAID and not software. its a new installation anyway.

thank you guys for your assistance and advice.

much appreciated!!