MC/SG node timeout

[Pedro Sousa]

Hello.
Can you tell me why am I receiving these messages under syslog:
Mar 22 07:15:34 poru0011 cmcld: timers delayed 2.70 seconds
Mar 22 07:15:34 poru0011 cmcld: Warning: cmcld process was unable to run for the
last 2 seconds,
Mar 22 07:15:34 poru0011 cmcld: which is longer than the node timeout (2 seconds
)

thanks

[Carsten Krege]

The messages indicate that there is a kernel problem that leads to SG's main process cmcld being starved out.

First cmcld's timer loop thread finds out that a SG timer is delayed that should have popped earlier, hence the "timer delayed" message.
Next the realtime process cmcld logs a message that it couldn't get CPU time for more then NODE_TIMEOUT seconds (here: default value 2 seconds).

It is hard to say what sort of kernel problem this is, therefore I strongly recommend to open a call with your local HP Response Center. It is a good idea to increase the NODE_TIMEOUT value to 5-8 seconds (the recommended values, see latest SG manual Dec 2000) to avoid unplanned downtime, as long as the problem has not been fixed (a downtime (i.e. a TOC) would occur if cmcld is not able to update the safety timer within the failover time; the failover time is scaled with NODE_TIMEOUT).

If the delays are not occurring too often and are not lasting longer than ~2 seconds, this is for most admins nothing to worry about (although still not nice).


It might be helpful to watch out whether the problem happens always at the same point of time, as this could be an indication that a periodic event (job scheduled by cron, startup of Predictive, stop of an major application etc.) causes the problem.

Carsten

[Pedro Sousa]

Thanks Carsten.
I'll open a call just in case.