VoIP - CMM/Catalyst troubles

[Marco_168]

Hi!

I wonder if anyone might have an idea what this syslog message indicates:

Apr 13 01:58:22.653 MEZT: %PM_SCP-SP-4-UNK_OPCODE: Received unknown unsolicited
message from module 6, opcode 0x101
Apr 13 01:58:38.897 MEZT: %OIR-SP-3-PWRCYCLE: Card in module 6, is being power-c
ycled off (Module not responding to Keep Alive polling)
Apr 13 01:58:39.173 MEZT: %MLSM-6-MMLS_MODE_CHANGE: MMLS: Replication Mode Chang
e Detected because oir event
Apr 13 01:58:38.965 MEZT: %C6KPWR-SP-4-DISABLED: power to module in slot 6 set o
ff (Module not responding to Keep Alive polling)
Apr 13 02:00:32.222 MEZT: %MLSM-6-MMLS_MODE_CHANGE: MMLS: Replication Mode Chang
e Detected because oir event
Apr 13 02:00:32.199 MEZT: %DIAG-SP-6-NO_TEST: Module 6: No test to run
Apr 13 02:00:35.482 MEZT: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces
are now online

No one was doing a physical reset nor was anyone logged in to the cat. Is the HW defect? We already have a TAC case opened. Comments are appreciated.

Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
6 5 Communication Media Module WS-SVC-CMM SAD083-----

Mod MAC addresses Hw Fw Sw Status
--- ---------------------------------- ------ ------------ ------------ -------
6 xxxx.xxxx.xxxx to xxxx.xxxx.xxxx 2.6 12.3(8)XY4, 12.3(8)XY4, Ok

[Philip Neeson]

Hi Marco,

Did you find the resolution here.. I have exactly the same issue with the same hardware and version 12.3(8)XY4

Any help would be appriciated.

Regards,

Philip.

[Marco_168]

Cisco IOS Software, Cat6K-lc Software (wscmm-I6S-M), Version 12.3(8)XY4.

My collegues already tried these things:

* downgrade to XY2 -> stopped the reboots
* delet all SNMP related config -> no effect

We have two CMMs (one active, one standby). The active one was upgraded to XY4 again but no more reboots!

a CISCO design engineer is currently looking into our TAC case. No result yet! Seems like an OS bug to me.

Bye,

Marco

[Marco_168]

CISCO's answer to the reboot problem:

The DE has identified the root cause of the problem, with the help of the core files and will provide the fix in the coming XY5 release in Jun end.

If you still have further question on this, please let me know.

Guess there is no alternative to waiting...

[Philip Neeson]

Hi Marco,

I'm not having much luck here; Cisco TAC have said;

---

>"%PM_SCP-SP-4-UNK_OPCODE: Received unknown unsolicited message from module 8, opcode 0x101" message.

The SCP opcode of 0x101 is a message sent when a module is reloading. The message is considered to be unsolicited because it is not a user triggered reload.

The supervisor code is not expecting the module to send an SCP message with this opcode when it is not a user triggered reload. We can classify this as a minor software defect with the CMM code. It does not cause the supervisor to reload the CMM. It is clear from the logs that the module is not responding to SCP pings.

Other causes of SCP ping failures is high CPU utilization on a line card processor. To rule this out you would have to check the CPU utilization frequently on the CMM module to determine if the CPU utilization is high. Then at the same time, check the "remote command switch show scp counter" output for any SCP retransmissions for that module.

---

They suggested we replace the module, which we did but the issue still occurs.

We did swap the CMM (from slot 8) with a FW-SM (in slot 9) and the CMM has been stable now for more than 72hrs.

Do you have a Cisco Bug Id for the work the DE is doing? I could point my CE in this direction and they can compare notes.

Was your CMM reloading often?

Philip.

[Marco_168]

HI!

CISCO Bug ID is CSCeh67918

Reboots happen almost every night at the same time! Sniffing didn't yield anything. I really guess it's a CISCO bug.

Bye,

Marco