problem adding a re-installed node back to cluster

[avik]

hi
my node1 had got failed and i reinstalled the OS and other packages back. Now when i add the node back to the cluster service which is already running on second node, it just doesnt get added.

"cmrunnode node1 " lasts forever and "cmrunnode -v -w none node1" throws
Unable to perform the security token exchange with cmclconfd on node node1 error. You have the outputs attached below.

Post your thoughts and gain some points.


node1:/# cmviewcl

CLUSTER STATUS
oidcl up

NODE STATUS STATE
node1 down unknown
node2 up running

PACKAGE STATUS STATE AUTO_RUN NODE
oidpkg up running enabled node2
node1:/# clear
node1:/# cmviewcl -v

CLUSTER STATUS
oidcl up

NODE STATUS STATE
node1 down unknown

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vg01 /dev/dsk/c40t0d6 unknown

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY unknown 0/1/2/0 lan0
PRIMARY unknown 0/2/1/0/6/0 lan1
PRIMARY unknown 0/4/1/0/6/1 lan4
STANDBY unknown 0/4/1/0/6/0 lan3
STANDBY unknown 0/3/1/0/6/0 lan2

NODE STATUS STATE
node2 up running

Cluster_Lock_LVM:
VOLUME_GROUP PHYSICAL_VOLUME STATUS
/dev/vg01 /dev/dsk/c40t0d6 up

Network_Parameters:
INTERFACE STATUS PATH NAME
PRIMARY up 0/1/2/0 lan0
PRIMARY up 0/2/1/0/6/0 lan1
PRIMARY up 0/4/1/0/6/1 lan4
STANDBY up 0/4/1/0/6/0 lan3
STANDBY up 0/3/1/0/6/0 lan2

PACKAGE STATUS STATE AUTO_RUN NODE
oidpkg up running enabled node2

Policy_Parameters:
POLICY_NAME CONFIGURED_VALUE
Failover configured_node
Failback manual

Script_Parameters:
ITEM STATUS MAX_RESTARTS RESTARTS NAME
Subnet up 172.16.100.0

Node_Switching_Parameters:
NODE_TYPE STATUS SWITCHING NAME
Primary down node1
Alternate up enabled node2 (current)
node1:/#
node1:/#
node1:/# cmrunnode tirgris1
cmrunnode: Node tirgris1 is not a configured node name.
node1:/# cmrunnode node1
cmrunnode: Validating network configuration...
node1:/#
node1:/#
node1:/#
node1:/# cmrunnode -v -w none node1
Unable to perform the security token exchange with cmclconfd on node node1
node1:/#

[David Bellamy]

did you copy the ascii files back to the node after you reinstalled the OS. did you also do a cmapplyconf and make all the volume groups cluster aware.

[avik]

I had copied the ascii and package files to the recovered node. further the vgexport map was taken from the running node and have successfully imported the VG into the second node. cmapplyconf was not run but now when i am runnig it now, its taking forever...

node11:/etc/cmcluster# cmapplyconf -C oid.ascii
Begin cluster verification...

[avik]

though i have been able to import the VG sucesfully in the second node, i hadn't activated them as i dont hv downtime for apps.

[Ivan Krastev]

Check also if your identd daemon is running.

regards,
ivan

[avik]

inetd daemons are running and were restarted...still no luck.

node1:/var/adm/syslog# ps -ef | grep inetd
root 930 1 0 11:04:16 ? 0:00 /usr/sbin/inetd
root 6957 3609 0 17:33:16 pts/0 0:00 grep inetd
node1:/var/adm/syslog# cd /sbin/init.d
node1:/sbin/init.d# ./inetd stop
Internet Services stopped
node1:/sbin/init.d# ./inetd start
Internet Services started
node1:/sbin/init.d# ps -ef | grep inetd
root 6971 3609 1 17:33:37 pts/0 0:00 grep inetd
root 6966 1 0 17:33:29 ? 0:00 /usr/sbin/inetd
node1:/sbin/init.d#

[Steve Lewis]

In future, always take a recovery tape/net recovery image of both nodes after you make any changes, so that when you need to recover the server you can simply use that to get up and running again.

Anyway, check your cmnodelist or .rhosts files on all nodes in the cluster.
Then manually check the lan interfaces using linkloop and ping.
Then I suggest the use of a cmquerycl to see what it does find out there before you re-apply the cluster config.

[NMory]

Check if:

1) identd is patched

2) /etc/nsswitch.conf =
hosts: files dns

3) /etc/hosts contains a list of every IP-bearing NIC on each cluster node

4) nslookup and "who -Rm" shows the correct hostname

5) Port 113 is not denied in /var/adm/inetd.sec

NM

[avik]

verified cmnodelist and .rhosts files on both the nodes in the cluster. interfaces are able to ping each other.

cmquerycl is stucking at gathering storage information as below.

node1:/etc/cmcluster# cmquerycl -v -C cmclconfig -n node1 -n node2
Looking for other clusters ... Done
Gathering storage information

the next step is to take a downtime and active the VG in second node to see?

[Steve Lewis]

You didn't indicate the recovery method or versions being used on both nodes.

When you recovered the node, did you bring it up to the same patch level as the existing node, or put it higher with a later patch bundle, or leave it at a lower patch level?
Check that your Serviceguard versions are the same using swlist.

[Rita C Workman]

...hmm...you say you re-installed the O/S. You did not say you re-ignited the O/S from your Ignite tape. So since you loaded the packages again, I can only 'assume' that you loaded the O/S from something other than the Ignite.

If you did, we are all shooting at a moving target with our eyes closed. No way to know exactly what is there.

Your cluster knows the node, but your node doesn't look like the way the cluster remembers it. And the cluster remembers. Run cmviewcl -o

and see things as the cluster sees things.
My guess....you'll have to remove the node from the cluster and the take the node you rebuilt and treat it as a new node and follow the steps to add it like new.

Just my concensus,based on what I read...
Rgrds,
Rita

[avik]

trust the machine hs come up from an ignited backup.

[avik]

service guard versions are same across the box.

it doesnt post anything in any of the logs nor it does throw any error to check for anything.

[Stephen Doud]

For the message, "Unable to perform the security token exchange with cmclconfd on node ..."
I found that /etc/nsswitch.conf did not have the following
line:

ipnodes: files

This is an essential line for Serviceguard, and adding it prevented the message.
Due to Serviceguard's need for local hostname lookup, it is recommended that
/etc/nsswitch.files be copied to /etc/nsswitch.conf as a starting
configuration for hostname resolution.

In another case, it was suspected that dissimilar Serviceguard patch levels
across nodes in the cluster was the root of the problem. Unable to confirm that however.

[avik]

stephen, i will be trying that...

could bastille be an issue when setting up clusters? though i never seen "bastille -r" working to rolback the bastille effect'.