High CPU Usage

[L1nklight]

Hey Everyone,

I recently replaced a pair of A5800-24Gs (JC100A) with a set of A5800AF-48Gs (JG225A). These switches are serving core switch functionality in my DR site. I had to replace the switches for increased growth in our DR site. The AFs are bonded together using IRF and I have MAD bfd enabled.

For some reason, I seem to have high cpu usage and I believe it's causing some issues on my network. It looks like my latency is higher than it was with the JC100As. I have another site setup with a pair of AFs and it's only reporting 5-10% useage on both members as opposed to the new ones that have a 25-30% cpu use. 

Any ideas on how I can start troubleshooting?

[Peter_Debruyne]

Hi,

This can be checked using the hidden commands, that will show you which process is taking up the cpu resources:

<switch>sys
System View: return to User View with Ctrl+Z.
[switch]_hidecmd
Now you enter a hidden command view for developer's testing, some commands may
affect operation by wrong use, please carefully use it with our engineer's
direction.
[switch-hidecmd]dis cpu task
===== Current CPU usage info =====
CPU Usage Stat. Cycle: 26 (Second)
CPU Usage            : 12%
CPU Usage Stat. Time : 2000-07-16  10:14:00
CPU Usage Stat. Tick : 0x1a7f2(CPU Tick High) 0xca7dd6c2(CPU Tick Low)
Actual Stat. Cycle   : 0x0(CPU Tick High) 0x6a14fa50(CPU Tick Low)

TaskName        CPU        Runtime(CPU Tick High/CPU Tick Low)
VIDL            88%               0/5e52f721
TICK             0%               0/  837d32
STMR             0%               0/   f6cbc
DSTK             0%               0/   1d088
SYSM             0%               0/   2c6a4
L2X0             2%               0/ 28f63d8
bC.0             2%               0/ 2ab7f44
bLK0             0%               0/  f13733
DQFD             0%               0/  116f6c
DQIT             1%               0/ 158ce25
LPDT             0%               0/     2f6
STAT             0%               0/   51bf9
FMCK             0%               0/  114eeb
T_DM             0%               0/   4be51
mIPC             0%               0/   4c67b
...

Best regards,Peter

[L1nklight]

Hey Peter! Thanks so much for the response. I really appreciated it. While messing around a bit with the CLI, I also found out you can get the same results without going into the developer mode by typing the following:

[sys]disp cpu-usage 1 verbose

===== CPU usage info (no: 0 idx: 52) =====
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage : 27%
CPU Usage Stat. Time : 2013-11-20 09:24:31
CPU Usage Stat. Tick : 0x12aa(CPU Tick High) 0xa49357f3(CPU Tick Low)
Actual Stat. Cycle : 0x0(CPU Tick High) 0xee6cddfe(CPU Tick Low)

TaskName CPU Runtime(CPU Tick High/CPU Tick Low)
VIDL 73% 0/aef3337e
TICK 0% 0/ 8a8562
STMR 0% 0/ 18b4c3
DSTK 0% 0/ 12d2f2
SYSM 0% 0/ 6f926
MSGR 0% 0/ 5f2d45
bDPC 0% 0/ 2f9b4
L2X1 4% 0/ b6caf5f
bC.1 3% 0/ 8f1c339

This doesn't seem to give me as much information as I would have expected. Here's a summary of my overall concern with these switches: I have 2 cores at different data centers. Both cores have a pair of 5800AF-48Gs setup in an IRF configuration. There is one major difference between the two cores, one core runs 1211P01 and the other core system is running 1808P11. The 1808 core is currently registering 25-30% cpu load and the pings are very volitile by comparison to the 1211 core. The 1808 core recently replaced a set of 5800-24Gs that were running 1211P01 as well. The ping metrics have substantially changed and I am concerned it relates to the CPU overhead or even something thats misconfigured on the switch. This graph here illustrates the differences I am seeing in performance on the switch by comparison to the statistics I was pulling from the 5800-24G running 1211P01. Smokeping (the application forming the graph) is clearly showing that there is a significant "drift" being shown. The green dot represents the "average" ms but the darker shaded area is the spread of the pings. As you can see the spread varies quite a bit more. 

Does anyone have any input on why this might be happening? Is it standard for a 5800AF-48G to be running 30% CPU with an IRF configruation? This switch really isn't doing much more than the 5800-24G that it replaced. There is an extra IRF link added to the mix and I've configred BFD MAD. 

[Peter_Debruyne]

Hi,

in my experience it is quite normal to see 10-30% CPU load with IRF (many control plane protocols must be hot-synced between the members). The difference does seem very large between the 2 releases IMO.

Can you post the full process and cpu usage list ?

Generally speaking, ICMP has a very low prio on the CPU, so other processes will typically be served ahead of any ICMP reply.

Best regards,Peter

[L1nklight]

Sure no prob. 

===== CPU usage info (no: 0 idx: 38) =====
CPU Usage Stat. Cycle: 60 (Second)
CPU Usage : 26%
CPU Usage Stat. Time : 2013-11-20 15:17:26
CPU Usage Stat. Tick : 0x13f3(CPU Tick High) 0x508b4ff2(CPU Tick Low)
Actual Stat. Cycle : 0x0(CPU Tick High) 0xee6b2834(CPU Tick Low)

TaskName CPU Runtime(CPU Tick High/CPU Tick Low)
VIDL 74% 0/b199ef4f
TICK 0% 0/ 8b8114
STMR 0% 0/ 19bbba
DSTK 0% 0/ 12eb47
SYSM 0% 0/ 6d89d
MSGR 0% 0/ 6266cc
bDPC 0% 0/ 1dc2c
L2X1 4% 0/ bbd9893
bC.1 3% 0/ 90cadf2
bLK1 1% 0/ 31bf775
L2X0 4% 0/ b9bf3ed
bC.0 3% 0/ 9065e9a
bLK0 1% 0/ 2dcc72c
DQFD 0% 0/ 1e07440
DQIT 1% 0/ 31d14d0
LPDT 0% 0/ 421
STAT 0% 0/ 10fcb0
FMCK 0% 0/ 19fc3b
T_DM 0% 0/ 51ef6
mIPC 0% 0/ 4fe1a
T_VA 0% 0/ 8adc
DARP 0% 0/ 1b402e
T_ST 0% 0/ 5911e
T_SI 0% 0/ 6b962
TPBR 0% 0/ 7d474
BGRT 0% 0/ a8382
MARP 0% 0/ 88a17
VMAC 0% 0/ 8014
SCAR 0% 0/ 3b52db
RTMC 0% 0/ 16ee32
bRX1 0% 0/ 2dfca9
bRX2 0% 0/ 2cb12b
STKT 0% 0/ 1724cb
IUCT 0% 0/ 2cf7da
tEXP 0% 0/ 9f5d
bIP0 0% 0/ ff2e
bIP1 0% 0/ 191bf
NSET 0% 0/ 246053
NSTM 0% 0/ 1e7c23e
IPCB 0% 0/ 8f2a67
IPCD 0% 0/ 1bfc
RPCQ 0% 0/ 3f67cb
RPCD 0% 0/ 294bb
VP 0% 0/ 4ae
STM 0% 0/ 844eb2
STMH 0% 0/ 20f066
PMOT 0% 0/ 1f09b2
EXCP 0% 0/ db124
DRVR 0% 0/ 30a57
IPCM 0% 0/ cb07
INFO 0% 0/ cde8
HAMT 0% 0/ 2887
HACT 0% 0/ 47865
DEV 0% 0/ cf1ea
SOCK 0% 0/ 4f7be
ADJ4 0% 0/ 3e69d
SFLW 0% 0/ 21329
ACL 0% 0/ 23ac8
mac 0% 0/ b1d6
LAGG 0% 0/ 11608
MSTP 0% 0/ 1865b
GARP 0% 0/ e919b
CLST 0% 0/ 2f935
NDP 0% 0/ 316b6
NTDP 0% 0/ e585
HABP 0% 0/ d229
CFD 0% 0/ 111e69
RRPP 0% 0/ 426ca
LLDP 0% 0/ 650ca2
DCBX 0% 0/ 232eb
DLDP 0% 0/ 34bda
SMLK 0% 0/ 1ae2d
ARP 0% 0/ 2ca9ff
IP 0% 0/ 3c1a73
FSLH 0% 0/ 985f
FSLR 0% 0/ 6dd4e
NTPT 0% 0/ 468f8
VTYD 0% 0/ 95c13
DHCC 0% 0/ 135e1
ND 0% 0/ 11cc23
TRAP 0% 0/ 709
CWMP 0% 0/ 2484b
ACM 0% 0/ 1af3c4
LS 0% 0/ 65ebc
RDSO 0% 0/ 4d3c4
RDS 0% 0/ 7dc2
SC 0% 0/ 181ff
NAT 0% 0/ 8e0b
BFD 0% 0/ 361119
MFIB 0% 0/ 1573ca
STND 0% 0/ 27de2
ROUT 0% 0/ 572d0a
TNLM 0% 0/ c30c
IFNT 0% 0/ f863
GMAC 0% 0/ 1817a
DNS 0% 0/ 6ef7f
vt0 0% 0/ 4f96c