Problem with starting cluster from one node.

[ashokan_1]

My newly installed MC/ServiceGuard 11.16 on HPUX 11.11 with two nodes refuses to startup from Node 1, but can be started and halted without errors from Node 2. What has changed recently is the DNS info, and permissions on the logical volumes in the Lockdisk, but have been restored to the original.
Following is the startup log from Node 1:

moldb1:/#cmruncl -v
cmruncl : Validating network configuration...
Gathering configuration information ..
Gathering Network Configuration ..... Done
Error: Permission denied accessing node moldb2.
Error: Permission denied accessing node moldb2.
Error: Unable to determine network configuration: failed to setup for probing the network on node moldb2: Permission denied
Failed to probe network
Error: Detected a partition of IP subnet 10.10.10.0.
Partition 1
moldb1 lan1
Partition 2
moldb2 lan1
Error: Detected a partition of IP subnet 10.1.0.0.
Partition 1
moldb1 lan3
Partition 2
moldb2 lan3
Failed to evaluate network
cmruncl: Failed to validate the network configuration as reported above but will try to start the cluster anyway.
cmruncl : Network validation complete
cmruncl failed: Permission denied
.........moldb1:/#

I am new to MC/Serviceguard. Please help.

[G. Vrijhoeven]

Hi,

You could start one node:
cmruncl -n moldb1
This only starts one node, but make sure the DNS changes and the lock disk gets fixed. If not to the original settings mind that it will take a cluster restart to alter the lock disk parameter.

HTH,

Gideon

[Armin Kunaschik]

Did you change the hostname?
It's always a bad idea to mess with hostnames/nodenames in a cluster.

Change every occurence of the old name to
the new one (/etc/hosts,/etc/cmcluster/cmclnodelist,
/etc/cmcluster/cmclconfig.ascii...)
and re-apply the cluster configuration!

Hope this helps,
Armin

[Stanimir]

Obviosly you have a cluster-configuration problem with network. Pay attention on your lan-configuration:

#lanscan
#netstat -in
#lanadmin
#vi /etc/rc.config.d/netconf

The check your cmcluster.ascii /net-part:lans, IPs/ and after that create newcluster-configuration.

[Stephen Doud]

Check to see if the hostname and alias are swapped in /etc/hosts. Put hostname to the left of the alias.

Load /etc/cmcluster/cmclnodelist with both (SIMPLE) hostnames - on both nodes. eg:
node1 root
node2 root

Don't use fully qualified names


Determine if the same IP is assigned to different NICs (check both nodes)


-sd

[Nguyen Anh Tien]

1. You should read this docs
http://docs.hp.com/en/B3936-90079/ch01.html
2. check /etc/cmcluster/cmclnodelist. this file has format
hostname1 username1
hostname2 username1
hostname: hostname or IP of nodes in cluster
username: user you login and run cmruncl and other cluster command.
After checking and correct this file any problem you encouter. pls post again
Regard
tienna

[ashokan_1]

I verified all the above suggestions. As Stanimir suggested, it looks like a network issue. The heartbeat_IP has a subnetmask 255.255.255.0 and the public IP has a subnet mask 255.255.0.0. Is it mandatory to keep the subnet mask of the heartbeat IP as 255.255.255.0, can't it be in the same subnet of the Public IPs? For the Oracle10g RAC installation, the cache fusion IP is with same subnet mask of the public IP, and this is added into cmclconf.ascii. The cluster was redone with all IP addresses with subnet mask 255.255.0.0, at this time the error from the second node while starting the cluster is "permission denied to 'Public IP'"

[Stephen Doud]

Verify the HEARTBEAT_IP LAN 'network' (netstat -in) is set to use the 255.255.255.0 netmask.
Create a new cluster.ascii file (full cmquerycl) and compare it to the result of a cmgetconf - if the network configurations do not match (particularly the comments at the bottom of each node section), then the network configuration is different than when the cluster was created.
-SD