cmclconf unusual syslog error

[Matthew Ghofrani]

In a 8 node cluster running SG 11.15, I get this unusual message in syslog on only one of the nodes. This started after patching all servers with PHSS_32660 and inserting -i in inetd.conf area of hacl lines.
Does anyone know how to resolve this:
@@@@@@@@@
Sep 20 09:36:14 cmcld: Cannot connect to configuration daemon (cmclconfd) on node
Sep 20 09:36:14 cmcld: Failed to dump Flight Recorder log buffer.
Sep 20 09:40:10 cmcld: Unable to write updated configuration file $SGCONF/cmclconfig to disk: No such file or directory.
Sep 20 09:40:10 cmcld: Will retry to write out configuration file to disk in 5 min
Sep 20 09:45:11 cmcld: Unable to write updated configuration file $SGCONF/cmclconfig to disk: No such file or directory.
Sep 20 09:45:11 cmcld: Will retry to write out configuration file to disk in 5 min
@@@@@@@@@@

[Hakan Aribas]

Please look at the following link, may be useful

http://forums1.itrc.hp.com/service/forums/questionanswer.do?threadId=206968

[Sameer_Nirmal]

It seems that cmclconfd daemon is not running on the node. Check if it is running
#ps -ef | grep cmclconfd

Just wondering what made to put -i as an attribute to cmclconfd in inetd.conf???
what are other attributes being used?
Have you used -i in all other nodes as well?

Since this daemon is not working , Flight Recorder is failing as well.

[melvyn burnard]

it looks like the changes you made may not have been read by inetd.
verify these and issue the inetd -c command to check.

[Matthew Ghofrani]

The patch "to do list" was big in which placing -i disables using sendmail criteria.
And yes all nodes are identical and cmclconfd does run fine on all nodes including the troubled one.

What else?

[melvyn burnard · Accepted solution]

I assume tis was working prior to the updates beiong done?

A question to ask.
Do you have a file /etc/cmcluster.conf?
If so, is it the same as on the other nodes?
It seems that the daemons do not have the information they require for accessing file locations.

[Matthew Ghofrani]

The /etc/cmcluster.conf is identical on all nodes but the /etc/cmcluster/cmclconfig is different than the rest on the troubled node.
I think it can't be updated due the errors in syslog.

Even cmcheckconf & cmapplyconf work ok!

[melvyn burnard]

that sounds like it was unable to distribute the binary file from another node after a cmapplyconf change.
Not normally recommended, but if the other 7 nodes have the same cmclconfig file, you could check the permissions/ownerships of this file, then rcp a "good" copy of the file onto thios node.
Have a backup of the /etc/cmcluster directory made first!


I also wonder what would happen if you did a cmhaltnode then a cmrunnode on this node.

If you do, you may want to turn up the logging in inetd.conf to see if this can be tracked any further and log a call with your local HP Response Centre

[Sameer_Nirmal]

As you are saying -i is being used with cmclconfd on all cluster nodes , then identd funcationality is not used.
It seems that after patching installed and disabling the identd , inetd -c has not been run as Melvyn said before.
Probably you need run
# /usr/sbin/inetd -c

Confirm you have entries as given below in the inetd.conf file on the node.

The patch special instruction say this:

If you would prefer to install this patch and choose not to utilize the identd functionality (thus obviating the need to enable identd), do the following steps on each node after installing PHSS_31067 and PHSS_31069 but before having each node re-join the cluster (e.g. before issuing a cmrunnode or cmruncl).

1) Change the cmomd entry in /etc/inetd.conf to appear as:

hacl-probe stream tcp nowait root \
/opt/cmom/lbin/cmomd /opt/cmom/lbin/cmomd -i -f \
/var/opt/cmom/cmomd.log -r /var/opt/cmom

The cmclconfd line should appear as:

hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd \
cmclconfd -c -i

2) Reload inetd: "/usr/sbin/inetd -c"

[Matthew Ghofrani]

I have done inetd -c many times, but I am afraid to do 2 things. 1. To copy the file from another node incase it brings the package down, and 2. Can't bring the node down (at least not yet) this is a production mission critical application.

PS. I do have HP involved in this, but were exploring other inputs since this issue has been going on for 11 days!!