Aruba & ProVision-based
1753317 Members
5280 Online
108792 Solutions
New Discussion

Weird Broadcast Storm

 
One234
Occasional Advisor

Weird Broadcast Storm

Hello All,

 

I’m having weird issue on 1 of our switches. The switch is a:

 

ProCurve J9089A Switch 2610-48-PWR

Software revision R.11.116

 

Every now and then the network on that switch goes down for a few seconds. When I look in the switch the CPU usage was over 90% and the errors are through the roof.  All ports receive and send between 1Gb and 4Gb of data in just a few seconds. The Rx errors and Tx Drops go from 0 to as high as 548,000 in seconds. And the weirdest thing is that ports that are NOT connected send and receive data as well. For example, port 1 is down and there is no patch cable in the port. But the counters are as shown below:

 

Status and Counters - Port Counters for port 1                      

 

  Name  : Office                                                         

  Link Status     : Down

  Totals (Since boot or last clear) :                                   

   Bytes Rx        : 2,630,109,833      Bytes Tx        : 3,851,483,045    

   Unicast Rx      : 8,879,255          Unicast Tx      : 12,249,457       

   Bcast/Mcast Rx  : 112,936            Bcast/Mcast Tx  : 1,673,512        

  Errors (Since boot or last clear) :                                   

   FCS Rx          : 22                 Drops Tx        : 211              

   Alignment Rx    : 30                 Collisions Tx   : 0                

   Runts Rx        : 0                  Late Colln Tx   : 0                

   Giants Rx       : 0                  Excessive Colln : 0                

   Total Rx Errors : 42                 Deferred Tx     : 0                

  Rates (5 minute weighted average) :

   Total Rx  (bps) : 0                  Total Tx  (bps) : 0        

   Unicast Rx (Pkts/sec) : 0            Unicast Tx (Pkts/sec) : 0        

   B/Mcast Rx (Pkts/sec) : 0            B/Mcast Tx (Pkts/sec) : 0        

   Utilization Rx  :     0 %            Utilization Tx  :     0 %

 

The statistics for all the ports are:

 

Status and Counters - Port Counters

 

                                                              Flow  Bcast

  Port    Total Bytes  Total Frames Errors Rx    Drops Tx     Ctrl  Limit

  ------- ------------ ------------ ------------ ------------ ----- ------

  1       2,186,625... 22,915,160   42           211          off   0    

  2       2,898,408... 61,699,756   0            88,038       off   0    

  3       1,177,410... 6,361,223    4            405          off   0    

  4       369,968,242  16,137,814   2            127          off   0    

  5       2,949,623... 384,356,113  0            754,306      off   0    

  6       1,572,208... 2,955,900    268          20           off   0    

  7       729,817,448  44,298,074   1            157,873      off   0    

  8       1,775,326... 42,946,214   0            158,578      off   0    

  9       1,009,498... 29,551,087   0            155,284      off   0    

  10      3,634,673... 34,794,969   273          155,535      off   0    

  11      0            0            0            0            off   0    

  12      3,656,204... 69,785,609   157,969      156,177      off   0    

  13      648,508,487  2,350,959    3            0            off   0    

  14      1,686,030... 44,087,419   5            5545         off   0    

  15      774          4            1            0            off   0    

  16      4,143,056... 138,345,593  24           161,565      off   0    

  17      166,970,809  1,194,749    2            32           off   0    

  18      255,391,213  334,981,928  6            175,149      off   0    

  19      452,525,822  40,150,657   2591         148          off   0    

  20      3,349,658... 170,835,943  548,538      22,436       off   0    

  21      134,446,920  1,100,442    2            27,992       off   0    

  22      2,415,604... 69,948,238   10           159,945      off   0    

  23      269,819,677  428,033      0            36           off   0    

  24      1,000,376... 2,154,290    172          155,175      off   0    

  25      2,319,175... 3,027,659    1            335          off   0    

  26      1,323,527    13,993       0            0            off   0    

  27      1,762,488... 29,318,736   3            158,185      off   0    

  28      3,132,146... 22,249,274   0            158,164      off   0    

  29      699,493,241  233,104,023  2            182,939      off   0    

  30      0            0            0            0            off   0    

  31      1,247,824... 33,402,322   2            158,164      off   0    

  32      0            0            0            0            off   0    

  33      2,253,144... 796,034,730  0            191,424      off   0    

  34      0            0            0            0            off   0    

  35      735,813,511  204,875,091  7            173,114      off   0    

  36      0            0            0            0            off   0    

  37      0            0            0            0            off   0    

  38      1,774,448... 5,324,206    2            19           off   0    

  39      345,588,799  114,448,187  1            86,461       off   0    

  40      1,435,537... 186,851,033  0            3858         off   0    

  41      4,189,278... 21,833,785   0            158,165      off   0    

  42      3,391,831... 18,263,350   5            435,142      off   0    

  43      2,711,372... 20,032,558   14           75,637       off   0    

  44      2,708,333... 14,612,315   0            78,136       off   0    

  45      1,043,213... 8,635,449    0            0            off   0    

  46      1,002,041... 26,210,443   0            158,165      off   0    

  47      2,770,060... 100,330,850  1            0            off   0    

  48      1,169,738... 10,400,757   0            227          off   0    

  49-Trk1 3,703,533... 3,847,487... 0            863          off   0    

  50-Trk1 3,737,030... 2,816,498... 0            249          off   0    

  51      0            0            0            0            off   0    

  52      0            0            0            0            off   0    

 

Ports 1, 2, 3, 4, 6, 11, 13, 14, 15, 19, 20, 23, 25, 26, 30, 32, 34, 36, 37, 51 and 52 are empty ports with no patch cable in it. Port 20 is even disabled in software and still sending and receiving data.

 

interface 20

   disable

   name "Mogelijk defect"

exit

 

Spanning tree is enabled in the switch. I disabled spanning tree in the switch to see what the counters would look like during a “normal” broadcast storm. I connected 2 ports with a patch cable to create a loop and the counters of a “normal” broadcast storm are shown below.

 

Multiple Spanning Tree (MST) Information

 

  STP Enabled   : No

 

switch01(config)# show interface

 

 Status and Counters - Port Counters

 

                                                              Flow  Bcast

  Port    Total Bytes  Total Frames Errors Rx    Drops Tx     Ctrl  Limit

  ------- ------------ ------------ ------------ ------------ ----- ------

  1       0            0            0            0            off   0    

  2       0            0            0            0            off   0    

  3       0            0            0            0            off   0    

  4       0            0            0            0            off   0    

  5       159,228,129  667,678      0            154,956      off   0    

  6       0            0            0            0            off   0    

  7       70,606,276   557,311      0            154,871      off   0    

  8       70,559,303   557,212      0            154,871      off   0    

  9       71,755,868   576,603      0            154,870      off   0    

  10      70,819,847   557,946      0            154,871      off   0    

  11      0            0            0            0            off   0    

  12      70,589,586   557,309      0            154,871      off   0    

  13      0            0            0            0            off   0    

  14      0            0            0            0            off   0    

  15      0            0            0            0            off   0    

  16      1,017,955... 4,827,501    0            155,220      off   0    

  17      134,446,994  1,100,430    2            28           off   0    

  18      155,252,099  697,433      0            154,877      off   0    

  19      0            0            0            0            off   0    

  20      0            0            0            0            off   0    

  21      134,446,920  1,100,442    2            27,992       off   0    

  22      236,587,767  843,325      0            155,048      off   0    

  23      0            0            0            0            off   0    

  24      92,074,859   598,973      0            154,871      off   0    

  25      0            0            0            0            off   0    

  26      0            0            0            0            off   0    

  27      82,049,803   617,718      0            158,169      off   0    

  28      70,634,741   554,153      0            158,164      off   0    

  29      1,005,993... 1,556,212    0            158,168      off   0    

  30      0            0            0            0            off   0    

  31      70,806,465   554,358      0            158,164      off   0    

  32      0            0            0            0            off   0    

  33      70,806,010   554,372      0            158,165      off   0    

  34      0            0            0            0            off   0    

  35      133,232,516  672,983      0            158,176      off   0    

  36      0            0            0            0            off   0    

  37      0            0            0            0            off   0    

  38      15,404,065   223,642      0            0            off   0    

  39      86,343,666   467,160      0            84,169       off   0    

  40      219,213,861  448,576      0            22           off   0    

  41      70,575,828   553,964      0            158,165      off   0    

  42      9,736,418    62,166       0            434,692      off   0    

  43      70,260,428   421,219      0            75,637       off   0    

  44      69,823,860   418,717      0            78,133       off   0    

  45      14,291,113   219,036      0            0            off   0    

  46      71,510,347   556,221      0            158,165      off   0    

  47      52,745,885   421,484      0            0            off   0    

  48      14,291,177   219,037      0            0            off   0    

  49-Trk1 4,286,713... 6,631,332    0            0            off   0    

  50-Trk1 2,151,721... 3,460,206    0            0            off   0    

  51      0            0            0            0            off   0    

  52      0            0            0            0            off   0    

 

Does anyone have any idea where to start troubleshooting this?

 

Thank you in advance.

 

Best regards,

 

Marc D.

1 REPLY 1
One234
Occasional Advisor

Re: Weird Broadcast Storm

Hello All,

 

I have a small update.

 

I created a new VLAN yesterday and assigned it to a few ports. Nothing is connected to the ports, but they still get a bunch of errors.

 

Status and Counters - Port Counters

 

                                                              Flow  Bcast

  Port    Total Bytes  Total Frames Errors Rx    Drops Tx     Ctrl  Limit

  ------- ------------ ------------ ------------ ------------ ----- ------

  15      774          4            1            0            off   0    

  17      166,970,809  1,194,749    2            32           off   0    

  19      452,525,822  40,150,657   2591         148          off   0    

  21      134,446,920  1,100,442    2            27,992       off   0    

  23      269,819,677  428,033      0            36           off   0    

 

Status and Counters - VLAN Information - Ports - VLAN 999

 

  VLAN ID : 999   

  Name : Isolate                        

  Status : Port-based

  Voice : No

  Jumbo : No

 

  Port Information Mode     Unknown VLAN Status   

  ---------------- -------- ------------ ----------

  15               Untagged Learn        Down     

  17               Untagged Learn        Down     

  19               Untagged Learn        Down     

  21               Untagged Learn        Down     

  23               Untagged Learn        Down    

 

So, it doesn’t look like a broadcast storm or something like that. Broadcasts should stay in the same VLAN right?