topic Re: Node is refusing Serviceguard communication in Operating System - HP-UX

Node is refusing Serviceguard communication

Kehad Snydewel — Tue, 29 Apr 2008 17:51:53 GMT

Hi all,
I am busy with a new serviceguard 11.18 installation on HPUX 11.23 (2 node cluster).
I successfully created the cluster (cmquerycl, cmcheckconf, cmapplyconf). Did not configure the packet yet. I had to change the hostnames of both nodes and did it in the following manner:

-Cmhaltcl -v

Both nodes:
-set _parms hostname
-change hostnames and reboot

Node 1 and 2 :
-Changed the /etc/hosts, /.rhosts, /etc/cmcluster/cmclnodelist and /etc/cmcluster/cmclconf.ascii files to reflect the new node names.
-cmcheckconf –C cmclconf.ascii
Node olprd1 is refusing Serviceguard communication.
Please make sure that the proper security access is configured on node
olprd1 through either file-based access (pre-A.11.16 version) or role-based
access (version A.11.16 or higher) and/or that the host name lookup
on node olprd1 resolves the IP address correctly.
cmcheckconf: Failed to gather configuration information

/etc/hosts of both nodes
127.0.0.1 localhost loopback
192.168.250.163 olprd1 #LAN1
192.168.250.164 olprd2 #LAN1
10.10.10.10 hb1 # heartbeat of olprd1 LAN0
10.10.10.11 hb2 # heartbeat of olprd2 LAN0
I ‘ve gone through all the forums and checked everything but still the same error. i tried to cmcheckcl –C cmconf.ascii on both nodes but no luck. On both node I receive the same error and refers to the same node which is:
Node olprd1 is refusing Serviceguard communication.

I also tried cmquerycl –v –C cmclconfig.ascii -n olprd1 -n olprd2 on both nodes and recived the same error:
Node1 (olprd1)
Node olprd2 is refusing Serviceguard communication
Node2 (olprd2)
Node olprd1 is refusing Serviceguard communication.
I renamed the cmclconfig binary on both nodes as well but still the same error.

Any advice???

Re: Node is refusing Serviceguard communication

skt_skt — Tue, 29 Apr 2008 19:05:06 GMT

verify your hacl entries in /etc/inetd.conf and /etc/services. (restart the inet daemon).

Could put the exact error from the syslog.?

Re: Node is refusing Serviceguard communication

Kehad Snydewel — Tue, 29 Apr 2008 20:06:08 GMT

i checked for this and it seems that all hacl and auth entries exist on both nodes.

i do not have the syslog entries at the moment but will provide this as well later.

i also tried to remove both nodes from cmclnodelist and only add a +. and also tried to add the nodes and +.

first attempt/try
cat cmclnodelist
olprd1 root
olprd2 root

second attempt/try
cat cmclnodelist
olprd1 root
olprd2 root

third attempt/try
cat cmclnodelist
+

root@olprd1:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd1

root@olprd2:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd2

root@olprd1:/etc/cmcluster# cat /etc/inetd.conf |grep -i hacl
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmm
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c

root@olprd2:/etc/cmcluster# cat /etc/inetd.conf |grep -i hacl
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmm
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c

root@olprd1:/etc/cmcluster#cat /etc/inetd.conf |grep -i auth
auth stream tcp6 wait bin /usr/lbin/identd identd

root@olprd2:/etc/cmcluster#cat /etc/inetd.conf |grep -i auth
auth stream tcp6 wait bin /usr/lbin/identd identd

root@olprd1:/etc/cmcluster# cat /etc/services |grep -i hacl
hacl-hb 5300/tcp # High Availability (HA) Cluster heartbeat
hacl-gs 5301/tcp # HA Cluster General Services
hacl-cfg 5302/tcp # HA Cluster TCP configuration
hacl-cfg 5302/udp # HA Cluster UDP configuration
hacl-probe 5303/tcp # HA Cluster TCP probe
hacl-probe 5303/udp # HA Cluster UDP probe
hacl-local 5304/tcp # HA Cluster Commands
hacl-test 5305/tcp # HA Cluster Test
hacl-dlm 5408/tcp # HA Cluster distributed lock manager

root@olprd2:/etc/cmcluster# cat /etc/services |grep -i hacl
hacl-hb 5300/tcp # High Availability (HA) Cluster heartbeat
hacl-gs 5301/tcp # HA Cluster General Services
hacl-cfg 5302/tcp # HA Cluster TCP configuration
hacl-cfg 5302/udp # HA Cluster UDP configuration
hacl-probe 5303/tcp # HA Cluster TCP probe
hacl-probe 5303/udp # HA Cluster UDP probe
hacl-local 5304/tcp # HA Cluster Commands
hacl-test 5305/tcp # HA Cluster Test
hacl-dlm 5408/tcp # HA Cluster distributed lock manager

Re: Node is refusing Serviceguard communication

Matti_Kurkela — Wed, 30 Apr 2008 07:24:11 GMT

ServiceGuard A.11.18 uses the bootstrap authentication file /etc/cmcluster/cmclnodelist only when cluster configuration does not exist yet.

The old configuration contains the old host names and may be interfering with your cmapplyconf.

You may have to run "cmdeleteconf" on all your nodes to remove all traces of the old configuration, then cmapplyconf to apply your new cluster configuration as if you were setting up a new cluster.

MK

Re: Node is refusing Serviceguard communication

Kehad Snydewel — Wed, 30 Apr 2008 11:05:23 GMT

this is what i get when issueing the command cmdeleteconf on both nodes:

NODE1
root@olprd1:/etc/cmcluster#cmdeleteconf
cmdeleteconf: Unable to get cluster configuration information: Permission denie.

NODE1 SYSLOG
Apr 30 13:01:19 olprd1 syslog: cmdeleteconf -v
Apr 30 13:01:19 olprd1 cmclconfd[29575]: Permission denied for root@olprd1 (RBA)
Apr 30 13:01:26 olprd1 syslog: cmdeletecon

NODE2
root@olprd2:/etc/cmcluster# cmdeleteconf
Checking current status
Unable to perform the security token exchange with cmclconfd on node olprd2
Unable to perform the security token exchange with cmclconfd on node OLPRD1
Unable to refresh view of cluster olclu: No such file or directory

NODE2 SYSLOG has no entries

Re: Node is refusing Serviceguard communication

Eric SAUBIGNAC — Wed, 30 Apr 2008 11:19:30 GMT

Bonjour,

I would do the following (2 ideas)

A) stop cluster, revert back all nodes to their old name, reboot, then cmdeleteconf

change both servers to new names, reboot.

modify cluster configuration file, packages configuration files, check and apply the new conf.

B) stop cluster if running (?) (cmhaltcl) , then "/sbin/init.d/cmcluster stop"

remove on all nodes /etc/cmcluster/cmclconfig
restart "/sbin/init.d/cmcluster start"

modify cluster configuration file, packages configuration files, check and apply the new conf.

Regards

Eric

Re: Node is refusing Serviceguard communication

Mridul Shrivastava — Wed, 30 Apr 2008 12:44:35 GMT

Did you set any parameters in identd ?
You may need to add -i
Then don't forget to inetd -c

Other ideas:
Have you used /var/adm/inetd.sec ?
If so, you will have to tailor it.

Have you commented out any hacl... services in /etc/services?

If you use cmclnodelist then also add 127.0.0.1 root and all the other IP addresses that you want to use (e.g. heartbeat lans etc). Do this on all nodes.

Set the permissions of cmclnodelist to 644 on all nodes.

Check the contents of /etc/rc.config.d/cmcluster
- you will have to change that eventually anyway.

Re: Node is refusing Serviceguard communication

Rita C Workman — Wed, 30 Apr 2008 14:30:11 GMT

I just have one question first......

From your HPUX box run a reverse lookup:
nslookup
nslookup

Did it resolve properly ?

If you use DNS (no matter what runs DNS) from a PC command line do the same thing and run a reverse lookup:

Did it resolve exactly the same as the one from HPUX?

Thanks,
Rita

Re: Node is refusing Serviceguard communication

Eric SAUBIGNAC — Wed, 30 Apr 2008 15:02:40 GMT

I would like to underline one thing before leaving : since MC/SG A.11.15 (or A.11.16, I don't remember exactly) the concept of "Access Control Policies" has been introduced in MC/SG. And since this modification, once a cluster is created, host name is a real source of problem when trying to alter cluster configuration. I am pretty sure that your problem is there (as already said by Matti) and not in configuration of inetd.sec, services, and anything else.

So to avoid wasting time, I think the better way for you is to delete this cluster then create it again. Really.

Eric

Re: Node is refusing Serviceguard communication

Kehad Snydewel — Wed, 30 Apr 2008 15:30:11 GMT

Mridul
i do not have an entry in my inetd.conf file with -i.

root@olprd1:/etc/cmcluster# cat /etc/inetd.conf |grep -i hacl
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmm
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c

Other ideas:
Have you used /var/adm/inetd.sec ?
If so, you will have to tailor it.

how should i edit this file because i am not using it:

root@olprd1:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd1

root@olprd2:/etc/cmcluster# more /var/adm/inetd.sec
dtspc allow 127.0.0.1 loopback olprd2

i did reverse lookups and eveything resolved properly

i think i will just reinstall the OS and Serviceguard on both nodes becuase i would like to configure the 2x BL860c with hardware RAID and not software. its a new installation anyway.

thank you guys for your assistance and advice.

much appreciated!!

Re: Node is refusing Serviceguard communication

Rita C Workman — Wed, 30 Apr 2008 17:23:49 GMT

If your lookups are ok, then Eric's suggestion is your easiest and most effective fix.

If you've decided to go through the extra work of reloading an O/S and software, to address a cluster problem, that's your choice and alot of work to address a cluster issue.

/rcw

Re: Node is refusing Serviceguard communication

skt_skt — Wed, 30 Apr 2008 17:35:42 GMT

here is your problem. you dont have 113 port listening.

[/root] grep -i auth /etc/services
ident 113/tcp authentication # RFC1413.

root [/root] netstat -na|grep 113
tcp 0 0 *.113 *.* LISTEN

Re: Node is refusing Serviceguard communication

skt_skt — Wed, 30 Apr 2008 17:37:59 GMT

[/root] grep -i ident /etc/inetd.conf
ident stream tcp wait bin /usr/lbin/identd identd

Re: Node is refusing Serviceguard communication

Mridul Shrivastava — Wed, 30 Apr 2008 18:09:40 GMT

Try moving those inetd.sec files and add -i in inetd.conf and see the result if it makes any difference.

Re: Node is refusing Serviceguard communication

Aashique — Thu, 01 May 2008 11:48:29 GMT

Hi,
As per "Mridul" posting i also think moving the file "inetd.sec" and try.

Thanks & Regards
Aashique

Re: Node is refusing Serviceguard communication

Kehad Snydewel — Fri, 02 May 2008 06:39:52 GMT

i will try to add the necassary port in inetd and let you know.

thanks

Re: Node is refusing Serviceguard communication

Stephen Doud — Fri, 02 May 2008 17:34:38 GMT

If you have a software support contract, you have access to the knowledge database on the ITRC - search for document emr_na-c01215677-1 titled, "HPUX & LINUX Serviceguard - is refusing Serviceguard communication"

Re: Node is refusing Serviceguard communication

IT Response — Tue, 11 Nov 2008 14:26:37 GMT

We ran into the same issue when setting up a new SG 11.16 cluster on a fresh install of HP-UX 11.11 EOE.

Running # cmquerycl -v logged tips regarding port 113 in syslog.

I investigated /etc/services for port 113 as suggested by Santhosh Kumar Theyyan on April 30th.

# netstat -an | grep 113 did not display a LISTEN service on any node.

Edit /etc/services for the line "auth 113/tcp authentication" to "ident 113/tcp authentication"

Do that on all nodes and bounce inetd as well.

Then # netstat -an | grep 113 will display
tcp 0 0 *.113 *.* LISTEN

# cmquerycl -C (...) then worked like a charm...

Running cmquerycl -v logged tips regarding port 113 in syslog.

Thanks Santhosh !

Re: Node is refusing Serviceguard communication

Ninad_1 — Tue, 11 Nov 2008 16:07:33 GMT

Hi,

Might be a silly suggestion and not sure if it will work - but is easiest to try.
In /etc/hosts - can you add the earlier defined names of hosts as aliases to the current host name entries and then try deleting the cluster .
Then remove the entries in /etc/hosts and create a new cluster with the new host names.

Regards,
Ninad