- Community Home
- >
- Networking
- >
- Switching and Routing
- >
- Comware Based
- >
- Re: High CPU usage by FMCK task on S5800 stack
Categories
Company
Local Language
Forums
Discussions
Forums
- Data Protection and Retention
- Entry Storage Systems
- Legacy
- Midrange and Enterprise Storage
- Storage Networking
- HPE Nimble Storage
Discussions
Discussions
Discussions
Forums
Forums
Discussions
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
- BladeSystem Infrastructure and Application Solutions
- Appliance Servers
- Alpha Servers
- BackOffice Products
- Internet Products
- HPE 9000 and HPE e3000 Servers
- Networking
- Netservers
- Secure OS Software for Linux
- Server Management (Insight Manager 7)
- Windows Server 2003
- Operating System - Tru64 Unix
- ProLiant Deployment and Provisioning
- Linux-Based Community / Regional
- Microsoft System Center Integration
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Discussion Boards
Community
Resources
Forums
Blogs
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 10:31 AM
тАО10-09-2018 10:31 AM
High CPU usage by FMCK task on S5800 stack
Hello there,
I read that HPE experts will be online and can help to solve some puzzles :)
I have a issue with high CPU usage by FMCK task on my 2xS5800 and 2xA5800 IRF stack.
Below is output of some commands from both 2xS5800 affected nodes.
I marked "Email me when someone replies", so, just let me know if you need more information.
Regards,
i
[DC-SW4-hidecmd]disp version H3C Comware Platform Software Comware Software, Version 5.20, Release 1808P17 Copyright (c) 2004-2014 Hangzhou H3C Tech. Co., Ltd. All rights reserved. H3C S5800-60C-PWR uptime is 241 weeks, 0 day, 1 hour, 10 minutes H3C S5800-60C-PWR with 2 Processors 512M bytes SDRAM 4M bytes Nor Flash Memory 512M bytes Nand Flash Memory Config Register points to Nand Flash Hardware Version is Ver.B CPLDA Version is 003, CPLDB Version is 003 BootRom Version is 220 [SubSlot 0] 48GE+4SFP Hardware Version is Ver.B [SubSlot 1] 16SFP Hardware Version is Ver.A [SubSlot 2] 4SFP Plus Hardware Version is Ver.B [SubSlot 3] No Module [SubSlot 4] No Module
[DC-SW4-hidecmd]dis cpu tas ===== Current CPU usage info ===== CPU Usage Stat. Cycle: 11 (Second) CPU Usage : 89% CPU Usage Stat. Time : 2018-10-09 20:22:27 CPU Usage Stat. Tick : 0x2285e7(CPU Tick High) 0x9e243c64(CPU Tick Low) Actual Stat. Cycle : 0x0(CPU Tick High) 0x2c3b729e(CPU Tick Low) TaskName CPU Runtime(CPU Tick High/CPU Tick Low) VIDL 11% 0/ 53ffc99 TICK 0% 0/ 29f2fe STMR 0% 0/ d29a4 DSTK 0% 0/ 4eda5 SYSM 0% 0/ 11f3f L2X1 2% 0/ 121dd56 bC.1 7% 0/ 35ee89f bLK1 2% 0/ fdbdea L2X0 2% 0/ 122e8a2 bC.0 7% 0/ 31a3e26 bLK0 2% 0/ fac707 DQFD 1% 0/ a70912 DQIT 2% 0/ 12b5087 STAT 0% 0/ 8bbb7 FMCK 51% 0/16c4f8a9 PSED 0% 0/ 19dbb0 T_DM 0% 0/ 158da mIPC 0% 0/ 19bac T_VA 0% 0/ 2bc3 DARP 0% 0/ 1a501 T_ST 0% 0/ 15244 T_SI 0% 0/ 260c3 TPBR 0% 0/ 32a4a BGRT 0% 0/ 2bea1 MARP 0% 0/ 35ba3 VMAC 0% 0/ 2f6b SCAR 0% 0/ 1da4b1 RTMC 0% 0/ 62478 bRX1 0% 0/ 27c5b8 bRX2 0% 0/ dc33b
[DC-SW4-hidecmd]dis cpu tas slot 2 ===== Current CPU usage info ===== CPU Usage Stat. Cycle: 32 (Second) CPU Usage : 88% CPU Usage Stat. Time : 2018-10-09 20:25:36 CPU Usage Stat. Tick : 0x2285e7(CPU Tick High) 0xfb6dab47(CPU Tick Low) Actual Stat. Cycle : 0x0(CPU Tick High) 0x804426bb(CPU Tick Low) TaskName CPU Runtime(CPU Tick High/CPU Tick Low) VIDL 12% 0/105236f2 TICK 0% 0/ 72bf7f STMR 0% 0/ 261c75 DSTK 0% 0/ f99db SYSM 0% 0/ 3e73b L2X1 2% 0/ 31699e4 bC.1 6% 0/ 8e96c64 bLK1 2% 0/ 306dc58 L2X0 2% 0/ 3440c11 bC.0 6% 0/ 8f0d28f bLK0 2% 0/ 2df1bab DQFD 1% 0/ 1e22347 DQIT 2% 0/ 3467bb5 LPDT 0% 0/ 388 STAT 0% 0/ 2c60c5 FMCK 52% 0/430d0f7c PSED 0% 0/ 454c26 T_DM 0% 0/ 4af5e mIPC 0% 0/ 58a5e T_VA 0% 0/ b36f DARP 0% 0/ 173678 T_ST 0% 0/ 1000bd2 T_SI 0% 0/ 818a6 TPBR 0% 0/ 94813 BGRT 0% 0/ 8774d MARP 0% 0/ 9c8a1 VMAC 0% 0/ 9e8c SCAR 0% 0/ 7623f0 RTMC 0% 0/ 120f1f bRX1 0% 0/ de2e38 bRX2 0% 0/ 2c0fa5 STKT 0% 0/ 3c1cbf IUCT 0% 0/ 25393b tEXP 0% 0/ 13053 bIP0 0% 0/ 1a546 bIP1 0% 0/ 127c4 NSET 0% 0/ 1a6255
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 11:06 AM
тАО10-09-2018 11:06 AM
Re: High CPU usage by FMCK task on S5800 stack
Dear Customer,
The FMCK is a Switch management task.
Below things may be a cause for this High CPU usage :
=> Configuration Restoration,
=> Active / Standby MPU Switchovers.
=> When there is a Line card insertion.
The CPU usage of these task may rise for a short time. These are a normal phenomenon.
=> If there is any H/W faults, due to which we notice many interupts, we may see the CPU usage of these tasks also rises abnormally.
Also check for any Network Loop Exists
Check this below link :
http://support.huawei.com/onlinetoolsweb/ptmngsys/Web/tsrev_s/en/content/s/15_edesk_high_cpu_usage/edesk_high_cpu_usage_edesk003.html
If the above infromation alleviates your query, don't forget to give Kudos.
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 11:22 AM - edited тАО10-09-2018 11:50 AM
тАО10-09-2018 11:22 AM - edited тАО10-09-2018 11:50 AM
Re: High CPU usage by FMCK task on S5800 stack
Thanks for the answer, hovewer, there are no loops, no active/standby switchovers or line cards insertions.
FMCK task CPU consumption is constant, not for a short time.
Below is additional information regarding this issue, please let me know if you need additional info,
Thanks
[DC-SW4]dis loopback-detection Loopback-detection is running. Detection interval is 30 seconds. No port is detected with loopback.
[DC-SW4]dis stp down-port Down Port Reason [DC-SW4]
[DC-SW4-hidecmd]dis cpu task | inc FMCK FMCK 35% 0/2cee0bc0 [DC-SW4-hidecmd]dis cpu task | inc FMCK FMCK 36% 0/3290ffe2 [DC-SW4-hidecmd]dis cpu task | inc FMCK FMCK 37% 0/38255d16 [DC-SW4-hidecmd]dis cpu task | inc FMCK FMCK 38% 0/3d5327ed [DC-SW4-hidecmd]dis cpu task slot 2 | inc FMCK FMCK 52% 0/7c1a4b1f [DC-SW4-hidecmd]dis cpu task slot 2 | inc FMCK FMCK 49% 0/ 3fda9a7 [DC-SW4-hidecmd]dis cpu task slot 2 | inc FMCK FMCK 52% 0/ 8c53ce8 [DC-SW4-hidecmd]
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 12:29 PM
тАО10-09-2018 12:29 PM
Re: High CPU usage by FMCK task on S5800 stack
Dear Customer,
Did you happen to reboot the switch and check was the issue still persisting?
What is the current firmware version?
Any abnormal information / warnings in the logs?, "Display logbuffer Reverse"
How long you are facing this issue?
Prior noticing this High CPU Usage issue, was there any changes in your network? if yes, could that be causing this? Anything that you remember here?
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 01:01 PM
тАО10-09-2018 01:01 PM
Re: High CPU usage by FMCK task on S5800 stack
Comware Software, Version 5.20, Release 1808P17
Unfortunately this stack is a part of our critical network infrastructure and its not rebooted and not updated for a long time. It is not an easy task to get a maintenance window and approval for reboot or FW upgrade.
"dis log re" shows only iMC activity and no any anomalies nor errors.
This issue appears on 1st slot for about a month ago, and on a 2nd slot about a 2 weeks ago.
There are no changes in our network, SFP modules, boards, etc, which can reflect this Fault Management ChecK (FMCK) task for the past month.
Hovewer, there are few CRC errors per second on one of IRF interfaces on slot 1. Can it be a root of cause for this issue?
<DC-SW4>dis int | inc [^0] input err Input: 2 input errors, 0 runts, 0 giants, 0 throttles Input: 9588018 input errors, 0 runts, 0 giants, 0 throttles <DC-SW4>
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 01:17 PM
тАО10-09-2018 01:17 PM
Re: High CPU usage by FMCK task on S5800 stack
Dear customer,
We have to check those 2 Input errors / CRC errors are they accumulated ones over a period of time. orelse are they still incremental?
What is the switch uptime here?
And are we able to see what are those 2 inputs error details under, "display logbuffer"?
If you are still seeing on going CRC errors on the IRF Ports of Slot 1, then there is a high chances of questioning how the control plane traffic is being passed here? and BTW, what is the status of the IRF Link status? "display irf link"
I understand the situation that it is critical network at your infrastructure, however having a CPU usage of 89% it not normal. Even a little spike would bring your network down. You may have to create a downtime to trouble shoot this one.
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 01:40 PM
тАО10-09-2018 01:40 PM
Re: High CPU usage by FMCK task on S5800 stack
These Input CRC errors are accumulated, below is a graph of XGE 1/2/4 errors per second for past 24h, uptime and all IRF info, "disp logbuffer re" shows nothing about interface errors nor any other error messages:
<DC-SW4>dis irf MemberID Role Priority CPU-Mac Description *+1 Master 25 0023-89ed-b3ee ----- 2 Slave 20 0023-89cd-ba3e ----- 3 Slave 15 d07e-2879-df80 ----- 4 Slave 10 d07e-2879-c909 ----- -------------------------------------------------- * indicates the device is the master. + indicates the device through which the user logs in. The Bridge MAC of the IRF is: 0023-89ed-b3ed Auto upgrade : yes Mac persistent : 6 min Domain ID : 0 <DC-SW4>dis irf top <DC-SW4>dis irf topology Topology Info ------------------------------------------------------------------------- IRF-Port1 IRF-Port2 MemberID Link neighbor Link neighbor Belong To 1 UP 2 UP 4 0023-89ed-b3ee 2 UP 3 UP 1 0023-89ed-b3ee 3 UP 4 UP 2 0023-89ed-b3ee 4 UP 1 UP 3 0023-89ed-b3ee <DC-SW4>dis irf con <DC-SW4>dis irf configuration MemberID NewID IRF-Port1 IRF-Port2 1 1 Ten-GigabitEthernet1/2/3 Ten-GigabitEthernet1/2/4 2 2 Ten-GigabitEthernet2/2/3 Ten-GigabitEthernet2/2/4 3 3 Ten-GigabitEthernet3/0/27 Ten-GigabitEthernet3/0/28 4 4 Ten-GigabitEthernet4/0/27 Ten-GigabitEthernet4/0/28 <DC-SW4>
H3C S5800-60C-PWR uptime is 241 weeks, 0 day, 1 hour, 10 minutes
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-09-2018 09:30 PM - edited тАО10-09-2018 10:13 PM
тАО10-09-2018 09:30 PM - edited тАО10-09-2018 10:13 PM
Re: High CPU usage by FMCK task on S5800 stack
Dear Customer,
I would try a reboot and if that does not fix, then a firmware upgrade, in such situation.
You may log a case and check if any more investigation is left here, You would require the serial number of the device and depending upon the warranty entitlement, they can provide you assistance here.
Note : Even for the support and R&D to work on such situation, they would expect to have the minimum firmware version on the devices. Or else they may not be able to assist : https://support.hpe.com/hpsc/doc/public/display?docId=a00050829en_us
Support Numbers :
Australia: 13 11 47 or +61 2 8031 8316 | New Zealand: 0800 664 747 or +61 2 8031 8315 | Malaysia: 1800 88 1811 or +60 3 6207 4533 | Philippines: +63 2395 3565
Singapore: 18002724333 or +65 6272 4333 | India: 1800 419 8080 or 1800 419 4994 | Ireland: 1890-946-500 (Warranty) or 1850-924-590 (Care pack or Contract)
UK: 0845-161-0030 (Warranty) or 0845-161-0050 (Care pack or Contract) | US & Canada: 1-844-806-3425 (Warranty) or 1-800-633-3600 (Care pack or Contract)
Other Countries: https://www.hpe.com/us/en/contact-hpe.html
Web HPE Support Centre: https://h20566.www2.hpe.com/portal/site/hpsc
I am an HPE Employee
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
тАО10-10-2018 06:40 AM
тАО10-10-2018 06:40 AM
Re: High CPU usage by FMCK task on S5800 stack
Ok, seems like there are no other options rather than reboot and FW upgrade. Will try this approach, thanks.