High cpu usage 5412zl - please help me troubleshoot

[Ryan_D]

I'm verifying some settings with our newly installed 5412zl and was alarmed to see the cpu usage on the switch at 66% (yikes).

I have PCM+ installed on a member server that monitors the switch however I have disabled it for troubleshooting purposes (and didn't seem to effect cpu usage at all).

sh system shows:
Up Time: 6 days
Memory Total: 139,754,400
Free: 91,360,640
CPU Util (%): 71

sh spanning-tree shows:
STP Enabled : Yes
Force Version : MSTP-operation
IST Mapped VLANs : 1-4094
Switch MAC Address : 001ffe-fee400
Switch Priority : 4096
Max Age : 20
Max Hops : 20
Forward Delay : 15

Topology Change Count : 64
Time Since Last Change : 46 hours

CST Root MAC Address : 001ffe-fee400
CST Root Priority : 4096
CST Root Path Cost : 0
CST Root Port : This switch is root

IST Regional Root MAC Address : 001ffe-fee400
IST Regional Root Priority : 4096
IST Regional Root Path Cost : 0
IST Remaining Hops : 20

Root Guard Ports:
TCN Guard Ports:
BPDU Protected Ports:
BPDU Filtered Ports:
PVST Protected Ports:
PVST Filtered Ports:

I didn't see anything in the sh tech log regarding "IP host route table is nearing max capacity" or "Unable to learn HW IP host routes, table FULL.". Reference page:

http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1277056

I have looked at http://www.hp.com/rnd/support/faqs/2650_6108.htm#question181 and have determined that if our default route statement is "ip route 0.0.0.0 0.0.0.0 192.7.221.1" (which is what we're currently using), then our summarized route statement would be "ip route 192.7.0.0/16 192.7.221.1". Would this help with performance?

I also noticed that the "addrmgrmovelist" in "sh tech" is showing weird behavior. It lists items that go from B1 to A1, then A1 to B1, then back again...all within minutes of each other. Ports A1, B1, C1, C2, and D1 are all uplink ports to other switches connected via 1Gb GBIC cards to Nortel 350T or 450T switches in unmanaged mode.

Please see the attached sh tech log file attached for more information.

We are still trying to troubleshoot ports D4, D6 and D7 that are dropping an unusually high amount of packets. Could this be what's causing high cpu?

Thanks for your help.

Ryan

[Pieter 't Hart]

Ryan,
Do the nortels have intrconnects? or is the 5412 a centre of a star-topology?

multiple mac-adresses apear on ports A1,A2,B1,B2,C1,C17,C2,D1,D19,D2,D23,D24
This may indicate there are more "uplinks" then only the A1,B1,C1,D1 that you want.

It's also possibly you go out of a port untaged in vlan-a and incomming to vlan-b, so you got a loop over different vlan's.

vlan 6
name "226WORKSTN"
untagged A1,A14,A16,A18,A20,A22,A24-B1,B13-B24,C1-C2,C6,C8,C10,C12,C14,C16,C18,C20,C22,D1-D22
looks OK.
But allthough the nortels are unmanaged, they DO have knowledge of vlan's, they receive these packets as incomming to vlan-1 and forward these packets internally with a vlan-1 tag.

draw a picture of your physical connections between the switches.
Match this with the vlan's you use, hereby take into acount tagged and untagged.

Maybe configuring all ports where multiple mac-adresses apear as "tagged all" may help, so no untagged packets are sent or processed on this uplinks that do not have a vlan-tag.

[cenk sasmaztin]

hi Ryan
commonly hight cpu usage cause static routing routing problem.
but you have one static routing

I think have loop on your network (may be int D13)

stp prevent loop on port but stp unaware some loop occurs

please make loop protech commad on int D13
I can see many log for D13

plase test loop protech command and result say me

sw(config)# loop-protect 1 receiver-action send-disable

if loop occur on port.
port is trun disable state and you must be monitoring cpu usage

please say me result

[Ryan_D]

Hey guys, thanks for replying to my question.

So is it recommended that I summarize my route to "ip route 192.7.0.0/16 192.7.221.1" rather than just leaving the default route?

Pieter - Sorry, I forgot to mention that some of the uplinks are actually from switch stacks. In any given stack we have 2 or 3 Nortel switches cascaded together, and then one switch in that stack has a 1Gb GBIC uplink to the core 5412. Almost every switch we have is in the same room so to make this work more efficiently, if I have to have 1 uplink for EVERY switch instead of 1 uplink for 3 different stacks, then that's no problem at all.

All switch uplinks:
A1 (switch stack-226 vlan)
B1 (switch stack-226 vlan)
B2 (single switch-225 vlan)
C1 (switch stack-226 vlan)
C2 (single switch-226 vlan)
D1 (single switch-226 vlan)
D2 (single switch-226 vlan)
D23 (single switch-224 vlan)(temporary)
D24 (single switch-224 vlan)(temporary)

Some of these switches can be cascaded (Nortel 450) and some cannot (Nortel 350, SMC), that's why some are in stacks and others aren't.

I will also try to optimize the vlan paths through tagging and try out some loop-protect options.

Thanks,
Ryan

[Matt Hobbs]

Try K.13.45. It fixes and ICMP redirect issue that could cause high CPU when the next hop router is in the same VLAN as other hosts.

[Ryan_D]

I tried enabling the loop-protect on just about every port except the switch uplink ports but high cpu usage still exists.

I tried summarizing the default route to "ip route 192.7.0.0/16 192.7.221.1" which drastically reduced the cpu usage, however I couldn't access 192.7.221.1 anymore from any vlans (which meant no internet/outside access).

I will try updating to K13.45 and see if that helps...thanks for the suggestion.

[Pieter 't Hart]

summarizing the default route to "ip route 192.7.0.0/16 192.7.221.1"

will not help if 192.7.221.1 has no other paths to the separate subnets.
It cannot route to 192.7.221.254, it probably it gets it right back.

look into this:
All switch uplinks:
A1 (switch stack-226 vlan)
B1 (switch stack-226 vlan)
B2 (single switch-225 vlan)
C1 (switch stack-226 vlan)
C2 (single switch-226 vlan)
D1 (single switch-226 vlan)
D2 (single switch-226 vlan)
D23 (single switch-224 vlan)(temporary)
D24 (single switch-224 vlan)(temporary)

multiple mac-adresses apear on ports A1,A2,B1,B2,C1,C17,C2,D1,D19,D2,D23,D24

pls check C17 and D19 ?

Look at these port-counters:
They most occur on your 100FDx ports, but you probably switched some ports in time.
I would say you got a speed/duplex mismatch.
You can also try enabling flow control.
Clear these counters and monitor again for details.

Status and Counters - Port Counters
Port Total Bytes Total Frames Errors Rx Drops Tx Ctrl Limit
-------- -------------- -------------- ------------ ------------ ----- ------
A4 4,174,237,174 27,902,486 63 7,292,780 off 0
A6 2,356,090,064 42,969,499 0 7,291,384 off 0
A10 2,207,900,851 20,906,621 0 7,290,420 off 0
A12 1,858,533,401 63,699,805 0 4,498,004 off 0
C2 1,694,863,064 12,938,181 0 11,112,634 off 0
C4 1,848,429,850 47,716,699 0 7,296,176 off 0
C21 2,193,623,189 20,847,633 0 7,287,480 off 0
D1 1,875,774,841 34,131,454 0 11,134,569 off 0
D2 1,421,327,748 34,375,472 0 22,232,748 off 0
D4 1,200,784,533 12,109,253 0 211,567,337 off 0
D6 928,737,867 3,110,679 0 146,816,698 off 0
D7 1,269,684,177 12,192,426 0 122,483,064 off 0
D23 3,776,029,371 15,214,271 0 11,134,858 off 0


Status and Counters - Port Status
| Intrusion MDI Flow Bcast
Port Type | Alert Enabled Status Mode Mode Ctrl Limit
-------- --------- + --------- ------- ------ ---------- ----- ----- ------
A4 100/1000T | No Yes Up 100FDx MDIX off 0
A6 100/1000T | No Yes Up 100FDx MDI off 0
A10 100/1000T | No Yes Up 100FDx MDIX off 0
A12 100/1000T | No Yes Up 100FDx MDI off 0
C2 100/1000T | No Yes Up 100HDx MDIX off 0
C4 100/1000T | No Yes Up 100FDx MDIX off 0
C6 100/1000T | No Yes Up 100FDx MDIX off 0
C21 100/1000T | No Yes Up 100FDx MDI off 0
D1 100/1000T | No Yes Up 100FDx MDI off 0
D2 100/1000T | No Yes Up 1000FDx MDIX off 0
D4 100/1000T | No Yes Down 1000FDx MDI off 0
D6 100/1000T | No Yes Up 100FDx MDI off 0
D7 100/1000T | No Yes Up 1000FDx MDIX off 0
D23 100/1000T | No Yes Up 100FDx MDI off 0
D24 100/1000T | No Yes Up 100FDx MDI off 0

Check the host that switches between ports A1,B1,D2
The ip-adress may give you a clue.
It may be just your management station, connected to different sittches.
Then there is no problem.
Or it may mean the stack is tripple connected to the main switch.
(but then again, i would expect more moves in the list).

show arp
IP ARP table
IP Address MAC Address Type Port
--------------- ----------------- ------- ----
192.7.226.125 000ffe-00821b dynamic B1

addrmgrmovelist
mac address vid old port new port timestamp
------------- ---- -------- -------- -----------------
000ffe-00821b 6 B1 A1 11/25/05 20:25:45
000ffe-00821b 6 A1 B1 11/25/05 20:25:46
000ffe-00821b 6 D2 B1 11/25/05 20:38:53
000ffe-00821b 6 A1 D2 11/25/05 20:38:53
000ffe-00821b 6 B1 A1 11/25/05 20:38:53
000ffe-00821b 6 D2 B1 11/25/05 21:07:34

[Ryan_D]

Well I haven't had a chance to upgrade to the latest firmware yet, but I might have figured out what was causing it. We have a test workstation in our computer room that was plugged into a 5 port workgroup switch, and the port on the ProCurve that the switch was plugged in to was configured for a different vlan than what the test machine was configured for (workstation configured for 224, ProCurve switch port configured for 226). Now that we put the test machine on a correct port, cpu usage is now averaging 9%, sometimes spiking to 21% at times (but haven't seen it higher than that since we made the change).

I'm surprised that what appears to be a small issue could cause such an increase in cpu usage on a switch as big and powerful as the 5412 - and we're only using 4 modules out of 12.

I've been configuring spanning-tree on our other Nortel switches to assist in diagnosing this problem. Should I enable spanning-tree on the individual Nortel switch ports or just the port links between the Nortel and the ProCurve core switch? Or should I enable Normal Learning on the switch links but enable Fast Learning on the individual switch ports?

Thanks again for your help!