One port failure mass up hole network

[AkosR]

Hi,

I have a network with 23 managed switch (J9539A, J9625A, J9565A) connected via optical cables. There are 6 loops and I use spanning tree. It's a CCTV netwok so on access port there are PoE cameras which cameras communicating with servers with multicast traffic.

I experienced that video streams messed up, live streams arrived but sometimes they wasn't full (because of packet loss). After a day I found one J9565A switch had 100% CPU process, and it was only because of 1!! camera. After I disabled the port the CPU went back to 3-4% and every problem disappeared from the network.

My question is: How can a single access device confuse a switch to increase it's CPU to mess up my hole network?

[Scott_Koster]

Hello,

I would suggest contacting support to help troubleshoot the problem. https://www.hpe.com/us/en/support.html

On a switch, the CPU should not be at 100% and this is typically indicitave of a problem in your network.  Items to check are firmware updates, rebooting devices (switch or camera).  Based on this limited information, something may have caused the camera to send packets destined to the switch CPU (broadcast, malformed packets, etc) since the problem went away after shutting down the port.

You can locate your switch modes and obtain firmware updates here.  If the problem comes back when you re-enable the camera port, updating the firmware is a good option.  (Also, check the firmware/software on your camera too)

https://h10145.www1.hpe.com/support/SupportLookUp.aspx

If you can't easily isolate this, please reach out to HPE Networking support for assistance. https://www.hpe.com/us/en/support.html

Best Regards,

-Scott

[AkosR]

Hello,

Thank you for your feedback.

If I reenable the port the CPU goes back rapidly to 99%. The firwmares are the same on all switches and cameras, and there are a lot of same type of cameras and switches.

I think it's a switchport failure of camera failure. I asked it because it's very strange for me how can a single switch port problem mess up a whole network. If a switch gets malformed packets, why does it forwards them? If other switches recievs malformed packets why doesn't they drop such packets to protect the network?

[Scott_Koster]

Sorry about the confusion on malformed.  I refer to malformed packet as any packet that doesn't reach it's intended destination, corrupt or wrong.  One example of a valid packet that is malformed that could cause something like this is large amounts of packets being sent with a broadcast MAC, not the Next Hop GW.

This could be a lot of things. Bad port, bad cable, etc. 

Things that might help isolate this:

Turn "fault-finder" feature on or increase sensitivity.  Switch(config)# fault-finder all sensitivity high

Check the log for events on the port.  "show log -r"

Look for excessive broadcast, or other errors.

Can I suggest that you please re-post your question on the ProCurve Forum in the HPE Networking community?  Or contact HPE Support directly.  The other forum is monitored by switching experts and you'll recive more responses and likely more helpful responses than my own. This forum is focused on our Software Defined Networking products and is less monitored for switching queries.

https://community.hpe.com/t5/ProCurve-ProVision-Based/bd-p/switching-e-series-forum#.WG0nb-R4cy8

[Vince-Whirlwind]

Isolate the problem: swap two cameras over so that a different camera is on the suspect switch and the suspect camera is on a different switch.

We want to know if the problem follows the camera, or if it follows the switch.