Unable to perform the security token exchange with cmclconfd on node

[RajuD]

I getting below error message when starting the package in 2 node cluster.

# cmruncl
cmruncl: Validating network configuration...
cmruncl: Network validation complete
Unable to perform the security token exchange with cmclconfd on node ch02cslp
Unable to perform the security token exchange with cmclconfd on node ch01cslp


# cmcheckconf -v -C sl_cluster.conf
Checking cluster file: sl_cluster.conf
Checking nodes ... Done
Checking existing configuration ... Done
Node ch01cslp is refusing Serviceguard communication.
Please make sure that the proper security access is configured on node
ch01cslp through either file-based access (pre-A.11.16 version) or role-based
access (version A.11.16 or higher) and/or that the host name lookup
on node ch01cslp resolves the IP address correctly.
cmcheckconf: Failed to gather configuration information

[Ivan Krastev]

Check for errors in /etc/cmcluster/cmnodelist .

regards,
ivan

[RajuD]

Hi,

Thanks for your reply...

But there is no error found in cmclnodelist...

[root@ch01cslp:/etc/cmcluster]
# more cmclnodelist
ch01cslp root
ch02cslp root

i think this is related to auth issue when i do rlogin to other node it take min 3minutes to switch?

[Ivan Krastev]

Is there are any problems with network configuration? Check also for proper resolving: on both nodes chech nslookup other_node.

regards,
ivan

[RajuD]

hi,

nslookup is happening.

Using /etc/hosts on: ch01cslp

looking up FILES
Name: ch02cslp.aen.nts.co.id
Address: 10.22.130.47
Aliases: ch02cslp

i checked and found cmcld service is not running whenstarting the package.

As a workaround i have started the cmcld service manually and started the package but this not solution for this problem, the cmcld service should start automatically when we start or form a cluster please correct me if i am wrong?

please let me know how to overcome this?

i think it is auth issue i have compared the /etc/inet.d, nsswitch.conf, .rhost file with the working fine cluster?

And also it is taking more time when we do rlogin and query any cluster command?
Please guide do i need to check install or update any cluster patch?

Find the cmcld output below
In Node1
# cmviewcl -v

CLUSTER STATUS
sl_cluster up

NODE STATUS STATE
ch01cslp up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vglock /dev/dsk/c4t0d3 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/2/1/0 lan2
STANDBY up 0/5/1/0 lan4

NODE STATUS STATE
ch02cslp up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vglock /dev/dsk/c4t0d3 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/2/1/0 lan2
STANDBY up 0/5/1/0 lan4

PACKAGE STATUS STATE AUTO_RUN NODE
pkg_FS_SYSLOGPRD up running enabled ch02cslp

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Service up 0 0 SYSLOG.MON
Subnet up 10.22.130.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary up enabled ch01cslp
Alternate up enabled ch02cslp (current)

[root@ch01cslp:/root]
#


Im Node2
# cmviewcl -v
cmviewcl: Cannot view the cluster configuration: No such file or directory.
Either this node is not configured in a cluster, user doesn't have
access to view the cluster configuration, or there is some obstacle
to viewing the configuration. Check the syslog file for more information.
For a list of possible causes, see the Serviceguard manual for cmviewcl.

[Stephen Doud]

Though not exhaustive, check the following:

CAUSE 1: "auth" line commented out in /etc/inetd.conf
#auth stream tcp6 wait bin /usr/lbin/identd identd

Serviceguard uses identd to validate Serviceguard commands are being performed
by nodes in the cluster. If this line is disabled, Serviceguard commands will
fail in various ways; one being the "security token exchange' error.

--------------------------------------------------------------------------------
CAUSE 2: In another case, it was found that /etc/nsswitch.conf did not have
the following line:

ipnodes: files

This is an essential line, and adding it corrected the problem. Due to
Serviceguard's need for local hostname lookup, it is recommended that
/etc/nsswitch.files be copied to /etc/nsswitch.conf as a starting configuration
for hostname resolution.
--------------------------------------------------------------------------------
CAUSE 3: It was suspected that a differential between Serviceguard patch
levels between nodes caused the problem. The recommendation was to install to
the same Serviceguard patch level. To identify the Serviceguard patch level,
run:

# what /usr/lbin/cmcld

The patch level will be listed with the version of Serviceguard. Example:

/usr/lbin/cmcld:
HP92453-02A.11.00 HP-UX SYMBOLIC DEBUGGER (END.O ILP32) $Revision: 75.02
$
Build date: Sun Oct 23 20:17:15 PDT 2005
Build id: ibld_sg_a1116patch_1111_product
Build platform: hpux
Cluster Monitor Product $Revision: 82.2 $
Cluster Monitor Product Only $Revision: 82.2 $
Daemon
A.11.16.00 Date: 10/23/05 Patch: PHSS_33834 <<---- HERE

Check the patch level on all nodes in the cluster. If it is different, schedule
the node with the older version for an outage, cmhaltnode that node and update it's
Serviceguard level to match the other node.
--------------------------------------------------------------------------------
CAUSE 4: This cause is primarily based on the message in syslog.log.
The cause - permissions on the /etc/passwd file were 400, not 444 as was
expected.

[RajuD]

Hi Stephen,

Cause 1:

For this there is no comment(#) found in /etc/inetd.conf file.

Steps taken : i have copied the /etc/inetd.conf file from the working cluster and copied to not working cluster but problem didnt solved.

Cause 2:

Need more clarrification.
Below is the output from the server theres is no ipnodes:files entry in nsswitch.conf.
Node 1
# more /etc/nsswitch.conf
#
# /etc/nsswitch.hp_defaults:
#
# @(#)B.11.11_LR
#
# An example file that could be copied over to /etc/nsswitch.conf; it
# uses NIS (YP) in conjunction with files.
#

#passwd: compat
#group: compat
#hosts: dns [NOTFOUND=return] nis [NOTFOUND=return] files
hosts: files [NOTFOUND=continue] dns
#networks: nis [NOTFOUND=return] files
#protocols: nis [NOTFOUND=return] files
#rpc: nis [NOTFOUND=return] files
#publickey: nis [NOTFOUND=return] files
#netgroup: nis [NOTFOUND=return] files
#automount: files nis
#aliases: files nis
#services: nis [NOTFOUND=return] files

Node 2
# /etc/nsswitch.hp_defaults:
#
# @(#)B.11.11_LR
#
# An example file that could be copied over to /etc/nsswitch.conf; it
# uses NIS (YP) in conjunction with files.
#

#passwd: compat
#group: compat
#hosts: dns [NOTFOUND=return] nis [NOTFOUND=return] files
hosts: files [NOTFOUND=continue] dns
#networks: nis [NOTFOUND=return] files
#protocols: nis [NOTFOUND=return] files
#rpc: nis [NOTFOUND=return] files
#publickey: nis [NOTFOUND=return] files
#netgroup: nis [NOTFOUND=return] files
#automount: files nis
#aliases: files nis
#services: nis [NOTFOUND=return] files

Cause 3:

Node 1
# cmversion
A.11.17.00

# what /usr/lbin/cmcld
/usr/lbin/cmcld:
Build platform: hpux
Build date: Tue Nov 8 15:47:46 PST 2005
Build id: ibld_sg_a1117patch_1123_product
Cluster Monitor Product $Revision: 82.2 $
Cluster Monitor Product Only $Revision: 82.2 $
Daemon
A.11.17.00 Date: 11/08/05 Patch: PHSS_33840

Node 2
# cmversion
A.11.17.00

# what /usr/lbin/cmcld
/usr/lbin/cmcld:
Build platform: hpux
Build date: Tue Nov 8 15:47:46 PST 2005
Build id: ibld_sg_a1117patch_1123_product
Cluster Monitor Product $Revision: 82.2 $
Cluster Monitor Product Only $Revision: 82.2 $
Daemon
A.11.17.00 Date: 11/08/05 Patch: PHSS_33840

Working Cluster Output.
# cmversion
A.11.16.00

# what /usr/lbin/cmcld
/usr/lbin/cmcld:
HP92453-02A.11.00 HP-UX SYMBOLIC DEBUGGER (END.O ILP32) $Revision: 75.02 $
Build date: Tue Jan 31 09:51:29 PST 2006
Build id: ibld_sg_a1116patch_1111_product
Build platform: hpux
Cluster Monitor Product $Revision: 82.2 $
Cluster Monitor Product Only $Revision: 82.2 $
Daemon
A.11.16.00 Date: 01/31/06 Patch: PHSS_33836

Cause 4:

Permission is 444 in /etc/passwd file but i cannt change it to 400 because it will create problem for database user, it will not display user name if i change it to 400.