High CPU usage by FMCK task on S5800 stack

[AA666]

Hello there,

I read that HPE experts will be online and  can help to solve some puzzles :)

I have a issue with high CPU usage by FMCK task on my 2xS5800 and 2xA5800 IRF stack.

Below is output of some commands from both 2xS5800 affected nodes.

I marked "Email me when someone replies", so, just let me know if you need more information.

Regards,

i

[DC-SW4-hidecmd]disp version
H3C Comware Platform Software
Comware Software, Version 5.20, Release 1808P17
Copyright (c) 2004-2014 Hangzhou H3C Tech. Co., Ltd. All rights reserved.
H3C S5800-60C-PWR uptime is 241 weeks, 0 day, 1 hour, 10 minutes

H3C S5800-60C-PWR with 2 Processors
512M    bytes SDRAM
4M      bytes Nor Flash Memory
512M    bytes Nand Flash Memory
Config Register points to Nand Flash

Hardware Version is Ver.B
CPLDA Version is 003, CPLDB Version is 003
BootRom Version is 220
[SubSlot 0] 48GE+4SFP Hardware Version is Ver.B
[SubSlot 1] 16SFP Hardware Version is Ver.A
[SubSlot 2] 4SFP Plus Hardware Version is Ver.B
[SubSlot 3] No Module
[SubSlot 4] No Module

[DC-SW4-hidecmd]dis cpu tas
===== Current CPU usage info =====
CPU Usage Stat. Cycle: 11 (Second)
CPU Usage            : 89%
CPU Usage Stat. Time : 2018-10-09  20:22:27
CPU Usage Stat. Tick : 0x2285e7(CPU Tick High) 0x9e243c64(CPU Tick Low)
Actual Stat. Cycle   : 0x0(CPU Tick High) 0x2c3b729e(CPU Tick Low)

TaskName        CPU        Runtime(CPU Tick High/CPU Tick Low)
VIDL            11%               0/ 53ffc99
TICK             0%               0/  29f2fe
STMR             0%               0/   d29a4
DSTK             0%               0/   4eda5
SYSM             0%               0/   11f3f
L2X1             2%               0/ 121dd56
bC.1             7%               0/ 35ee89f
bLK1             2%               0/  fdbdea
L2X0             2%               0/ 122e8a2
bC.0             7%               0/ 31a3e26
bLK0             2%               0/  fac707
DQFD             1%               0/  a70912
DQIT             2%               0/ 12b5087
STAT             0%               0/   8bbb7
FMCK            51%               0/16c4f8a9
PSED             0%               0/  19dbb0
T_DM             0%               0/   158da
mIPC             0%               0/   19bac
T_VA             0%               0/    2bc3
DARP             0%               0/   1a501
T_ST             0%               0/   15244
T_SI             0%               0/   260c3
TPBR             0%               0/   32a4a
BGRT             0%               0/   2bea1
MARP             0%               0/   35ba3
VMAC             0%               0/    2f6b
SCAR             0%               0/  1da4b1
RTMC             0%               0/   62478
bRX1             0%               0/  27c5b8
bRX2             0%               0/   dc33b

[DC-SW4-hidecmd]dis cpu tas slot 2
===== Current CPU usage info =====
CPU Usage Stat. Cycle: 32 (Second)
CPU Usage            : 88%
CPU Usage Stat. Time : 2018-10-09  20:25:36
CPU Usage Stat. Tick : 0x2285e7(CPU Tick High) 0xfb6dab47(CPU Tick Low)
Actual Stat. Cycle   : 0x0(CPU Tick High) 0x804426bb(CPU Tick Low)

TaskName        CPU        Runtime(CPU Tick High/CPU Tick Low)
VIDL            12%               0/105236f2
TICK             0%               0/  72bf7f
STMR             0%               0/  261c75
DSTK             0%               0/   f99db
SYSM             0%               0/   3e73b
L2X1             2%               0/ 31699e4
bC.1             6%               0/ 8e96c64
bLK1             2%               0/ 306dc58
L2X0             2%               0/ 3440c11
bC.0             6%               0/ 8f0d28f
bLK0             2%               0/ 2df1bab
DQFD             1%               0/ 1e22347
DQIT             2%               0/ 3467bb5
LPDT             0%               0/     388
STAT             0%               0/  2c60c5
FMCK            52%               0/430d0f7c
PSED             0%               0/  454c26
T_DM             0%               0/   4af5e
mIPC             0%               0/   58a5e
T_VA             0%               0/    b36f
DARP             0%               0/  173678
T_ST             0%               0/ 1000bd2
T_SI             0%               0/   818a6
TPBR             0%               0/   94813
BGRT             0%               0/   8774d
MARP             0%               0/   9c8a1
VMAC             0%               0/    9e8c
SCAR             0%               0/  7623f0
RTMC             0%               0/  120f1f
bRX1             0%               0/  de2e38
bRX2             0%               0/  2c0fa5
STKT             0%               0/  3c1cbf
IUCT             0%               0/  25393b
tEXP             0%               0/   13053
bIP0             0%               0/   1a546
bIP1             0%               0/   127c4
NSET             0%               0/  1a6255

[MstWntd]

Dear Customer,

The FMCK is a Switch management task.

Below things may be a cause for this High CPU usage :

=> Configuration Restoration,
=> Active / Standby MPU Switchovers.
=> When there is a Line card insertion.

The CPU usage of these task may rise for a short time. These are a normal phenomenon.

=> If there is any H/W faults, due to which we notice many interupts, we may see the CPU usage of these tasks also rises abnormally.

Also check for any Network Loop Exists
Check this below link :
http://support.huawei.com/onlinetoolsweb/ptmngsys/Web/tsrev_s/en/content/s/15_edesk_high_cpu_usage/edesk_high_cpu_usage_edesk003.html

If the above infromation alleviates your query, don't forget to give Kudos.

[AA666]

 Thanks for the answer, hovewer, there are no loops, no active/standby switchovers or line cards insertions.

 FMCK task CPU consumption is constant, not for a short time.

Below is additional information regarding this issue, please let me know if you need additional info,

Thanks

[DC-SW4]dis loopback-detection
 Loopback-detection is running.
 Detection interval is 30 seconds.
 No port is detected with loopback.

[DC-SW4]dis stp down-port
 Down Port                     Reason
[DC-SW4]

[DC-SW4-hidecmd]dis cpu task | inc FMCK
FMCK            35%               0/2cee0bc0
[DC-SW4-hidecmd]dis cpu task | inc FMCK
FMCK            36%               0/3290ffe2
[DC-SW4-hidecmd]dis cpu task | inc FMCK
FMCK            37%               0/38255d16
[DC-SW4-hidecmd]dis cpu task | inc FMCK
FMCK            38%               0/3d5327ed
[DC-SW4-hidecmd]dis cpu task slot 2 | inc FMCK
FMCK            52%               0/7c1a4b1f
[DC-SW4-hidecmd]dis cpu task slot 2 | inc FMCK
FMCK            49%               0/ 3fda9a7
[DC-SW4-hidecmd]dis cpu task slot 2 | inc FMCK
FMCK            52%               0/ 8c53ce8
[DC-SW4-hidecmd]

[MstWntd]

Dear Customer, 

 Did you happen to reboot the switch and check was the issue still persisting?

What is the current firmware version?  

Any abnormal information / warnings in the logs?, "Display logbuffer Reverse"

How long you are facing this issue?
Prior noticing this High CPU Usage issue, was there any changes in your network? if yes, could that be causing this? Anything that you remember here?

[AA666]

Comware Software, Version 5.20, Release 1808P17

Unfortunately this stack is a part of our critical network infrastructure and its not rebooted and not updated for a long time. It is not an easy task to get a maintenance window and approval for reboot or FW upgrade.

"dis log re" shows only iMC activity and no any anomalies nor errors.

This issue appears on 1st slot for about a month ago, and on a 2nd slot about a 2 weeks ago.

There are no changes in our network, SFP modules, boards, etc,  which can reflect this Fault Management ChecK (FMCK) task for the past month.

Hovewer, there are few CRC errors per second on one of IRF interfaces on slot 1. Can it be a root of cause for this issue?

<DC-SW4>dis int | inc [^0] input err
 Input:  2 input errors, 0 runts, 0 giants, 0 throttles
 Input:  9588018 input errors, 0 runts, 0 giants, 0 throttles
<DC-SW4>

[MstWntd]

Dear customer, 

We have to check those 2 Input errors / CRC errors are they accumulated ones over a period of time. orelse are they still incremental?

What is the switch uptime here?

And are we able to see what are those 2 inputs error details under, "display logbuffer"?

If you are still seeing on going CRC errors on the IRF Ports of Slot 1, then there is a high chances of questioning how the control plane traffic is being passed here? and BTW, what is the status of the IRF Link status? "display irf link"

I understand the situation that it is critical network at your infrastructure, however having a CPU usage of 89% it not normal. Even a little spike would bring your network down. You may have to create a downtime to trouble shoot this one.

[AA666]

These Input CRC errors are accumulated, below is a graph of XGE 1/2/4 errors per second for past 24h, uptime and all IRF info, "disp logbuffer re" shows nothing about interface errors nor any other error messages:

<DC-SW4>dis irf
MemberID  Role   Priority  CPU-Mac         Description
 *+1     Master  25        0023-89ed-b3ee  -----
   2     Slave   20        0023-89cd-ba3e  -----
   3     Slave   15        d07e-2879-df80  -----
   4     Slave   10        d07e-2879-c909  -----
--------------------------------------------------

 * indicates the device is the master.
 + indicates the device through which the user logs in.

 The Bridge MAC of the IRF is: 0023-89ed-b3ed
 Auto upgrade                : yes
 Mac persistent              : 6 min
 Domain ID                   : 0
<DC-SW4>dis irf top
<DC-SW4>dis irf topology
                           Topology Info
 -------------------------------------------------------------------------
               IRF-Port1                IRF-Port2
 MemberID    Link       neighbor      Link       neighbor    Belong To
 1           UP         2             UP         4           0023-89ed-b3ee
 2           UP         3             UP         1           0023-89ed-b3ee
 3           UP         4             UP         2           0023-89ed-b3ee
 4           UP         1             UP         3           0023-89ed-b3ee

<DC-SW4>dis irf con
<DC-SW4>dis irf configuration
 MemberID NewID    IRF-Port1                     IRF-Port2
  1       1        Ten-GigabitEthernet1/2/3      Ten-GigabitEthernet1/2/4
  2       2        Ten-GigabitEthernet2/2/3      Ten-GigabitEthernet2/2/4
  3       3        Ten-GigabitEthernet3/0/27     Ten-GigabitEthernet3/0/28
  4       4        Ten-GigabitEthernet4/0/27     Ten-GigabitEthernet4/0/28
<DC-SW4>

H3C S5800-60C-PWR uptime is 241 weeks, 0 day, 1 hour, 10 minutes

[MstWntd]

Dear Customer, 

I would try a reboot and if that does not fix, then a firmware upgrade, in such situation.

You may log a case and check if any more investigation is left here, You would require the serial number of the device and depending upon the warranty entitlement, they can provide you assistance here.

Note : Even for the support and R&D to work on such situation, they would expect to have the minimum firmware version on the devices. Or else they may not be able to assist : https://support.hpe.com/hpsc/doc/public/display?docId=a00050829en_us

Support Numbers : 
Australia: 13 11 47 or +61 2 8031 8316 | New Zealand: 0800 664 747 or +61 2 8031 8315 | Malaysia: 1800 88 1811 or +60 3 6207 4533 | Philippines: +63 2395 3565
Singapore: 18002724333 or +65 6272 4333 | India: 1800 419 8080 or 1800 419 4994 | Ireland: 1890-946-500 (Warranty) or 1850-924-590 (Care pack or Contract)
UK: 0845-161-0030 (Warranty) or 0845-161-0050 (Care pack or Contract) | US & Canada: 1-844-806-3425 (Warranty) or 1-800-633-3600 (Care pack or Contract)
Other Countries: https://www.hpe.com/us/en/contact-hpe.html
Web HPE Support Centre: https://h20566.www2.hpe.com/portal/site/hpsc

[AA666]

Ok, seems like there are no other options rather than reboot and FW upgrade. Will try this approach, thanks.