Node XXXX is refusing Serviceguard communication

[Paul Condren]

Hello.

I'm getting the error below when trying to build a new serviceguard cluster, 2 RP3440 nodes, SG version 11.16. This error occors when using check, apply or query commands. I tried creating the conf file fom another 11.16 cluster file and applying it but it still errors.

Error Is...

Checking nodes ... Done
Checking existing configuration ... Done
Warning: Unable to get configuration for cluster bacstel.
Error: Node ira70043 is refusing Serviceguard communication.
Please make sure that the proper security access is configured on node
ira70043 through either file-based access (pre-A.11.16 version) or role-based
access (version A.11.16 or higher) and/or that the host name lookup
on node ira70043 resolves the IP address correctly.
cmapplyconf : Failed to gather configuration information

I have...
1. created a cmcmnodelist
2. created a /.rhosts.
3. added all required entries to the /etc/hosts
4. checked an nslookup resolved the hostames OK - it does.
5. Tested a traceroute for both the nodes - no probs.

Ive build clusters before and not had this problem. The only 'new' thing here is that we have applied bastille to this server, although I have now reverted the bastille changes suspecting them as the cause. I have an ignite recovery tape taken p[rior to the bastile install so may revert the system to that worst case.

Any suggestion would be great!
Paul

[Bharat Katkar]

Hi Paul,
Did you use revert-action script to undo bastellie changes. If not then this will happen.
Simply removing bactille software won't help.

Regards,

[Paul Condren]

I ran the revert script and then rebooted to revert the kernel. Also ran the undo permission channges script.

Is bastile a no go with serviceguard?

[Bharat Katkar · Accepted solution]

Hi Paul,
I have never used Bastile with SG but it seems it's a critical job to go with. Currently don't know what all ports remained blocked on your nodes but i found out the ports that MC/serviceguard is using. See if you can check and free those ports for communication.
Also refer to this links below:

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=944107
http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=729393

Port Requirements:
-----------------

ServiceGuard uses the ports listed below.
Before installing, check /etc/services and be sure no other program has reserved these ports.

clvm-cfg 1476/tcp HA LVM Configuration
hacl-qs 1238/tcp HA Quorum Server
hacl-hb 5300/tcp High Availability (HA) Cluster heartbeat
hacl-hb 5300/udp High Availability (HA) Cluster heartbeat
hacl-gs 5301/tcp HA Cluster General Services
hacl-cfg 5302/tcp HA Cluster TCP configuration
hacl-cfg 5302/udp HA Cluster UDP configuration
hacl-probe 5303/tcp HA Cluster TCP probe
hacl-probe 5303/udp HA Cluster UDP probe
hacl-local 5304/tcp HA Cluster commands
hacl-test 5305/tcp HA Cluster test
hacl-dlm 5408/tcp HA Cluster distributed lock manager

In addition, ServiceGuard also uses dynamic ports (typically in the range 49152-65535) for some cluster services. If you have adjusted the dynamic port range using kernel tunable parameters alter your rules accordingly.


System Firewalls


When using a system firewall such as HP-UX IPFilter with ServiceGuard, specific communications must be allowed to ensure proper cluster operation. Specific IPFilter rules required by ServiceGuard are documented in the HP-UX IPFilter Release Notes, available from http://www.docs.hp.com -> Internet and Security Solutions.

General guidelines for using a system firewall with ServiceGuard are listed below.

To enable intra-cluster communications, each HEARTBEAT_IP network on every node within the cluster must allow the following communications in both directions with all other nodes in the cluster:

tcp on port numbers 5300-5304, and 5408 - and allow only packets with the SYN flag
udp on port numbers 5300 and 5302
tcp and udp on dynamic ports (typically 49152-65535)

If your ServiceGuard configuration uses a quorum server, all nodes within the cluster must allow the following communication to the quorum server IP address:
tcp on port 1238 - and allow only packets with the SYN flag
Any node providing quorum service for another cluster must allow the following communication from that
cluster's nodes:
tcp on port 1238 - and allow only packets with the SYN flag
Running the cmscancl command requires the "shell" port be open.

There are additional firewall considerations to enable execution of ServiceGuard commands from nodes outside the cluster, such as those listed in cmclnodelist. To allow execution of ServiceGuard commands, follow the guidelines below.

All nodes in the cluster must allow the following communications:

From the remote nodes:
tcp on ports 5302 - and allow only packets with the SYN flag
udp on port 5302
To the remote nodes:
tcp and udp on port numbers 49152-65535

The remote nodes must allow the following communications:
From the cluster nodes:
tcp and udp on port numbers 49152-65535

To the cluster nodes
tcp on ports 5302 - and allow only packets with the SYN flag
udp on port 5302

Hope that helps,
Regards,

[Bharat Katkar]

More details regarding cluster communication can be found here:
http://docs.hp.com/en/B3935-90068/ch01s03.html

Regards,

[Alex Glennie]

Hi Paul

What's the O/S here if 11.23 let me know, otherwise I'd initailly be checking :

grep identd inetd.conf
grep auth /etc/services
netstat -an | grep 113
syslogs yield any clues here ?
nsswitch.conf ?
debug inetd -i ?

in case this is an authentication issue of some sort ?

[Mark Nieuwboer]

Hi Paul,

This is a known issue for serviceguard 11.16.
Has nothing to do with bastille.
You need to install some patches i know 2
PHSS_32733 PHSS_32732. But i thing there are more.
What sometimes did the trick was rebooting the servers.

grtz. Mark

[Steven E. Protter]

Shalom Paul,

inetd.conf

May need new options:

example.

hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmomd -i -f /var/opt/cmom/cmomd.log -r /var/opt/cmom
#registrar stream tcp nowait root /etc/opt/resmon/lbin/registrar /etc/opt/resmon/lbin/registrar
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c -i


SEP

[rariasn]

Hello, Paul,

User "root" user for cmgetconf, cmappyconf, etc.

Sguard A.11.16 not use security cmclnodelist or .rhosts: use "security roles"

Install utility " SG-Manager A.05.00 Serviceguard Java GUI" for hpux

Salud

[Paul Condren]

Hi All.

Thanks for all the help and suggestions.

First up Roles and access - according to the book I dont need to set up roles etc to create a cluster fropm the command line as root - these are just for admin afterwards and setting up no root users with things like view only access. We have some 50 odd serviceguard clusters here and we've never had to do this in the past with 11.16

Next - reverting Bastille. I ran the bastile -r and also the revert permission changes cript. Still no joy. I also ran thru some suggestion from Alex but due to the traditional unrealistic project deadlines and timescales I was under pressure to complete the build - not having access to their app on the SG disks was upsetting the developers!

So, I luckily had an ignite taked just before the application of the bastille config. I revered to this and the cluster checked and applied first time with no issues. I have sice build the package and applied this and taken another ignite.

Ive have now applied bastille again today and everythings fine. ALl the serviceguard commands still work with no issues.

I can only conclude that if your applying bastille to a serviceguard environment than you need to do this after installing and building the cluster. There could be something in bastille that detects servicegard and afects its changes.