topic Re: cmclconfd takes over 90% of one CPU in Operating System - HP-UX

cmclconfd takes over 90% of one CPU

mario.schroeder — Tue, 10 May 2005 07:12:30 GMT

We have since yesterday the problem, that the cmclconfd takes over 90% of our CPU.

So at this time I killed them 3-4 times but it wont go away. On the Server runs no cluster like mc/SG cluster.

143 processes: 126 sleeping, 17 running
Cpu states:
CPU LOAD USER NICE SYS IDLE BLOCK SWAIT INTR SSYS
0 1.00 37.3% 0.0% 62.7% 0.0% 0.0% 0.0% 0.0% 0.0%
1 0.05 0.0% 0.2% 0.2% 99.6% 0.0% 0.0% 0.0% 0.0%
--- ---- ----- ----- ----- ----- ----- ----- ----- -----
avg 0.53 18.7% 0.0% 31.5% 49.8% 0.0% 0.0% 0.0% 0.0%

Memory: 153308K (91208K) real, 396380K (307696K) virtual, 2317168K free Page# 1/4

CPU TTY PID USERNAME PRI NI SIZE RES STATE TIME %WCPU %CPU COMMAND
0 ? 13386 root 232 20 1196K 516K run 22:07 99.70 99.52 cmclconfd
1 ? 1804 root -16 20 7036K 6568K run 120:23 4.12 4.11 midaemon
1 ? 34 root 152 20 0K 7648K run 58:51 0.76 0.76 vxfsd
1 ? 1805 root 152 20 8016K 11068K run 161:01 0.36 0.36 prm3d

Re: cmclconfd takes over 90% of one CPU

melvyn burnard — Tue, 10 May 2005 07:21:24 GMT

Are you saying that this node is NOT part of a Serviceguard cluster?
If so, then simply swremove Serviceguard.

Re: cmclconfd takes over 90% of one CPU

Robert-Jan Goossens — Tue, 10 May 2005 07:22:00 GMT

Mario,

Check if this doc helps you.

Document description: cmclconfd consumes a lot of CPU time
Document id: UMCSGKBRC00014425

http://www5.itrc.hp.com/service/cki/docDisplay.do?docLocale=en_US&docId=200000078241146

Robert-Jan

Re: cmclconfd takes over 90% of one CPU

Steven E. Protter — Tue, 10 May 2005 07:23:12 GMT

You might want to try and shut down Serviceguard, because it certainly seems like a cluster is running, or someone is trying to configure one.

It appears to me that a deamon is spawning this process and killing the process is not going to help.

swlist -l product | grep -i service

Lets see what's installed. Then see whats running at boot time in /etc/rc.config.d and maybe shut the daemon down.

SEP

Re: cmclconfd takes over 90% of one CPU

mario.schroeder — Tue, 10 May 2005 07:41:21 GMT

It is a MC S/G Cluster installed. Of course, but the cluster should be inactive.

The Process take the CPU time until since yesterday 16:15h

#cmviewcl
cmviewcl : Cannot view the cluster configuration. Either this node is not confi
gured in a cluster, or else there is some obstacle to viewing the configuration.
Check the syslog file for more information. For a list of possible causes, see
the ServiceGuard manual for cmviewcl.

#swlist -l product | grep -i service
ATS-CORE A.11.14 Service Guard Advanced Tape Services
CIFS-Server A.01.08 CIFS/9000 (Samba) File and Print Services

Ignite-UX B.4.2.110 HP-UX System Installation Services
NFS B.11.11 ONC/NFS; Network-File System,Information
Services,Utilities
PHNE_22722 1.0 NTP timeservices upgrade plus utilities
SG-Domain-Tool B.01.08 MC/ServiceGuard HP Domain Script Template
s
SG-FasTrack-Tool B.01.08 MC/ServiceGuard Netscape FastTrack Script
Templates
SG-Fndation-Tool B.01.08 MC/ServiceGuard Foundation Bundle Script
Templates
SG-Informix-Tool B.01.08 MC/ServiceGuard Informix Script Templates

SG-NFS-Tool A.11.11.02 MC/ServiceGuard NFS Script Templates
SG-Oracle-Tool B.01.08 MC/ServiceGuard Oracle Script Templates
SG-Progress-Tool B.01.08 MC/ServiceGuard Progress Script Templates

SG-Sybase-Tool B.01.08 MC/ServiceGuard Sybase Script Templates
ServControlMgr A.02.05.01 HP-UX ServiceControl Manager
ServiceGuard A.11.14 Service Guard
TechPrintServ B.11.11 HP-UX Technical Image Printing Service

Now I typed "cmhaltcl" and the Cluster should stop

Re: cmclconfd takes over 90% of one CPU

mario.schroeder — Tue, 10 May 2005 07:43:41 GMT

Hmmm .... it seems not good

#cmhaltcl
cmhaltcl : Unable to determine the nodes on the current cluster
cmhaltcl : Either no cluster configuration file exists, or the file is corrupted, or cmclconfd is unable t
o run

Re: cmclconfd takes over 90% of one CPU

Denver Osborn — Tue, 10 May 2005 07:57:04 GMT

Have you recently installed any mc/sg patches? later patches add identd auth for mc/sg. If you don't follow the steps outlined for the patch install... the nodes in the cluster could experience the comm problem you've described.

If no patches, etc. have been installed... the I would turn to the cmclnodelist file. Verify the /etc/cmcluster/cmclnodelist entries are good. If they contain hostname entries, verify dns is good or that they both resolve using /etc/hosts 1st.

It would also help troubleshooting if you post the output of `grep hacl /etc/inetd.conf` and `what /usr/lbin/cmcld`

Hope this helps,
-denver

Re: cmclconfd takes over 90% of one CPU

mario.schroeder — Tue, 10 May 2005 08:02:21 GMT

Hi,

here are the outpost.
Until yesterday we havenot problems with this machine.

rbz16:/#grep hacl /etc/inetd.conf
hacl-probe stream tcp nowait root /opt/cmom/lbin/cmomd /opt/cmom/lbin/cmomd -f /var/opt/cmom/cmomd.l
og
hacl-cfg dgram udp wait root /usr/lbin/cmclconfd cmclconfd -p
hacl-cfg stream tcp nowait root /usr/lbin/cmclconfd cmclconfd -c

And the output of the what command

/usr/lbin/cmcld:
HP92453-02A.10.20 HP-UX SYMBOLIC DEBUGGER (END.O) $Revision: 74.03 $
Build date: Mon Dec 10 14:40:36 PST 2001
Build id: ibld_sgops_a1114
A.11.14 Date: 12/15/2001
Daemon
Config DB
Cluster Monitor
Command Srv
CommunicationSrv
Config
Dlm
Local Comm
Network Sensor
Package Manager
Remote Comm
Quorum Module
API
Service Sensor
Cluster LVM
Status DB
Sync
Util

Re: cmclconfd takes over 90% of one CPU

melvyn burnard — Tue, 10 May 2005 08:02:26 GMT

What patch level do you have for this Serviceguard revision? do:
what /usr/lbin/cmcld
and look at what PHSS_ is installed.

Re: cmclconfd takes over 90% of one CPU

Alex Lavrov. — Tue, 10 May 2005 08:03:10 GMT

Also disable it on the next reboot:

change:
AUTOSTART_CMCLD=0
to
AUTOSTART_CMCLD=1
in /etc/rc.config.d/cmcluster.

Alex.

Re: cmclconfd takes over 90% of one CPU

melvyn burnard — Tue, 10 May 2005 08:09:48 GMT

so you have an unpatched SG version.
I would suggest you obtain and install the relevant Serviceguard 11.14 patch on all nodes.
This may have been triggered by some spurious networking "traffic jams"

you may have to reboot the node to clear the issue, although kill -9 should work on it

Re: cmclconfd takes over 90% of one CPU

mario.schroeder — Tue, 10 May 2005 16:43:58 GMT

Thanx to all.

after a reboot it works fine.

schroedi
http://www.schroedi.org