cmcheckconf : Unable to verify cluster file:

[Philip Hull]

Hi,
Recently we attempted to introduce a new node into our tri-node cluster. Despite having everything lined up we checked the config and encountered the below error


# cmcheckconf -v -C /etc/cmcluster/cluster.ascii -P /etc/cmcluster/db/db.ascii -P /etc/cmcluster/app1/app1.ascii

Checking cluster file: /etc/cmcluster/cluster.ascii.bkp
Checking nodes ... Done
Checking existing configuration ... Done
Gathering configuration information ... Done
Gathering configuration information ... Done
Gathering configuration information .............. Done
Cluster psa1 is an existing cluster
Parsing package file: /etc/cmcluster/db/db.ascii.bkp.
Package db already exists. It will be modified.
Parsing package file: /etc/cmcluster/app1/app1.ascii.
Package app1 already exists. It will be modified.
cmcheckconf : Unable to verify cluster file: /etc/cmcluster/cluster.ascii



Now, I have read several posts regarding errors reported by cmcheckconf, but they all deal with different issues. The problem is that I cannot identify what exactly cmcheckconf has a problem with.


Any Ideas?

Thanks!

[Mike Shilladay]

Hi Philip,

Just checking the obvious - you have got the /etc/cmcluster/cluster.ascii on all three nodes, the spelling the same too.

Mike.

[Philip Hull]

Hi Mike,
Yes I just double-checked - sometimes the obvious can just slip you by. But in this case all configs are distributed across all nodes.

[Steven E. Protter]

Shalom,

cmcheckconf : Unable to verify cluster file: /etc/cmcluster/cluster.ascii

This error is generic.

The problem could be anything.

Check the log files for more information.

use ioscan to insure all nodes have access to the shared storage and the lock disk.

SEP

[Philip Hull]

Hi,
Any specific logs I should be checking?
There isn't anything in syslog.log to indicate a problem.

[Mike Shilladay]

Hi Philip,

Check the logs in the package directory, there should a log file in there. That may help.

Mike.

[Philip Hull]

Hi Mike,
Unfortunately those logs have not been written to since the last failover (July 1st).

As has been stated, the error I see is very generic so tracking down more information as to where it is stopping is a major headache.

[Philip Hull]

For those who are interested, I have attached the cluster.ascii file.

[melvyn burnard]

Firstly, there is no requirement to have th ecluster ascii file on all nodes
Secondly, this will not write to any package log.
Have you tried simply adding in the node, without specifiying the packages?
cmcheckconf -v -C /etc/cmcluster/cluster.ascii
What version is this? are they all at the same version? and patch level?
Is there anything in teh syslog?


Also, how did you add the third node in? Did you use cmquerycl to create a new ascii file?
You might also consider turning up the logging level when doing the cmcheckconf

[Philip Hull]

Hi, yes I have tried - same result;

# cmcheckconf -v -C /etc/cmcluster/cluster.ascii

Checking cluster file: /etc/cmcluster/cluster.ascii
Checking nodes ... Done
Checking existing configuration ... Done
Gathering configuration information ... Done
Gathering configuration information ... Done
Gathering configuration information .............. Done
Cluster psa1 is an existing cluster
cmcheckconf : Unable to verify cluster file: /etc/cmcluster/cluster.ascii.

[Vihang.]

Hi Philip,

Have you checked the syslog? What does it say? There would be some errors related to this that we might be ignoring thinking them as unimportant. Can you paste the syslog output after you run cmcheckconf.



Regards,
Vihang.

[Stephen Doud]

I scanned our database. This generic message has occurred when one of the following was found:

1) EMC BCVs are involved
- the documented fix required that the BCV was split and vgchgid'd, however, I suggest you perform an ioscan and locate any NO_HW device files and remove them.

2) A NIC in the cluster ASCII file is unavailable.
- run ifconfig on each NIC, insuring NICs bearing an IP are assigned the correct subnet mask, and NICs intended to operate as standby NICs are assigned inet 0.0.0.0 netmask 0

[Deepak Kr]

If not yet tried then Try this also:

Create a cluster configuration file from running cluster

cmgetconf -v -c
Add new node in this config file and run:

# cmquerycl -c -C -n

Review the file, and if it looks good, apply it:

# cmapplyconf -f -C

See if you get same error.

[Philip Hull]

Hmm that's very interesting. I had to make a slight change to your command, since the cluster is already running, but I ran

cmquerycl -c psa1 -C cluster.ascii.tmp -n psa1d -n psa1b

And it did not report an error at all, only a warning

Warning: Unable to determine local domain name for psa1d

[Deepak Kr]

Can you please provide following


cat /etc/hosts
cat /etc/cmclnodelist

[Michael Steele_2]

Bet cmgetconf will tell you the exact error

[Philip Hull]

Alas, no. cmgetconf gives no error at all - it seems very happy in fact.

I have attached the hosts file. The cmnodelist has been changed to a single entry, "+", because when I initially tried to check the config I ran into all sorts of problems with this new node trying to talk to the other nodes...

[Michael Steele_2]

You haven't added a domain name to the local host. This is just a warning and will do nothing nasty.

[Deepak Kr]

Agree with Michael...domain name is missing here.


Try using :

IP Addr hostname hostname.domainname

Also in cmclnodelist :

hostname1 root
hostname2 root

No FQDN needed.


provide cat /etc/resolv.conf

#hostname
#nslookup hostname

[Philip Hull]

All

Solution was found - it turns out there was a problem with both standby LANs on node psa1a (lan1 and lan3).

This could not be determined until we bought the entire cluster down in a scheduled maintenance window, whereupon cmcheckconf finally reported that lan1 and lan3 could not be added.

Thanks to everyone for their input.

[Deepak Kr]

:-))

Please remember this:

http://forums12.itrc.hp.com/service/forums/helptips.do?#33

Have fun!