Excessive CRC/alignment errors

[Leon van Dongen ONVZ]

Hello,

I am barely 6 weeks working as a network infrastructure engineer and challenged with excessive CRC/alignment errors, broadcasts and STP topology changes.

I am new to HP ProCurve networking, however I have a great deal experience on Cisco networks. I used my expertise to troubleshoot the problems however to no avail.

The network exist out of ProCurve 2600, 2800 and 5300 series switches. Connected through either Cat. 6 UTP patchcords and a mix of 50 micron and 62.5 MMF running distanes between 50 and 400 meters.

My first approach was to deal with the huge amounts of topology changes that occur almost every 20 minutes. I found - throug MIB walking the interface - ports that transitioned over 500 times to the forwarding state. This is not normal operation, so I looked further.

When investigating interface statistics I noticed receive drops on several 2600s specifically on port 25 or 26 which are loaded with a GBIC SX transceiver and 62.5/125 patchcord.

This is not local to this switch but occurs on other stacks aswell. I decided to start with physically troubleshooting the problem. I replaced the cable and transceivers with new ones, however the problem persisted.

I contacted HP, sent them the show tech output. However I did not receive an official response what causes these errors.

The current software version on the 2600s is H8.88, this version is not officially released. The 5300s ran H10.04, but where recently upgraded to H10.23. As a result the amount of TCs dropped, however there occurs at least one TC every day. This is huge contrasts with the 50-300 TCs a day before the software upgrade on the backbone switches.

The Network Management software reports large amounts of errors, I am puzzled what causes them since I have replaced most cabling on ports that report CRC and FCS errors.

I recently downgraded one stack to H8.86, to comply with the supported release. I can't see why this help since the release notes of both software do not mention known physical or data-link problems.

I appreciate if any of you can share some experience on how to trace problems the "HP way".

TIA

Leon van Dongen
Network Infrastructure Engineer

[Anders_35]

We experienced massive amounts of excessive CRC errors, and RSPT TCNs that were constantly flying aound the network.
We have 34xx, 6400 and 2626 switches,

Both issues were resolved in firmware upgrades. We now run H.08.90 on our 2626s and M.08.87 on the 34xxs/6400s, and all is well...

[Anders_35]

For 2626s, I dug up this from my case with HP:

"2626s should be on H.08.85. The reason for that is:

RSTP - Under some circumstances, the Switch may allow packets to loop for an extended period of time.
RSTP - The switch repeatedly flushes its MAC address table, resulting in intermittent flooding of all traffic and corresponding communications problems, especially through a ProCurve mesh.

The last RSTP bug mentioned though was introduced in H.08.75 (fixed in .83). The first RSTP bug mentioned here is fixed in .84.
"

So your 2626s should be fine, with regards to the topology changes, if they are running on H.8.88.

[Leon van Dongen ONVZ]

Can you confirm that the excessive CRC/alignment errors, excessive broadcast disappeard or dropped drasticly after moving to H8.90?

I appreciate your input, this finally confirms my susspicion on software related problems rather than faulty hardware.

Many thanks.

Leon

[Anders_35]

Sorry, I left out an important piece of info: We only had STP problems on 2626. The CRC/alignment errors were seen only on the other switches.
HP first suspected there was a problem with the 10gbps CX-4 cards in our 34xxs, but the CRC errors disappeared after upgrade to M.08.82. From HPs fix list:
"Packets larger than 1482 bytes within a mesh will be reported as FCS Rx errors and may generate excessive CRC error messages in the Event Log."

2800 do not run the same firmware as 2600, do they? Maybe they have not fixed the problem there...

[Leon van Dongen ONVZ]

The software release from the 2800 and the 2600 series are different. Whereas the 2800 run I8.71, 2600 run H8.88.

I just spoke to our reseller. He just sent me a copy of H8.91 software release.

I have six switches in stock, so I'll check what happens when I run this release.

Thank you for your effort.

Leon

[Eirram_1]

Hello Leon,

Excessive CRCs may be caused by duplex mismatches. Do you see the CRCs only on the inter-switch links, or only on links to hosts? Or both?

The RJ45 ports on a procurve are per default on auto. Your fiber miniGIBCs are probably 1000Fdx. Correlate this to what is configured on the other end of the link(s).

If your UTP cabling is certified for Gigabit Ethernet, then you should be fine. Check for potential problems with patch panels and media converter (if there are any).

I presume your 5300 is running E.10.23 and not H.10.23 :-)

The management software is nice, but check the eventlog of the switch itself. Then you'll have the info from the horse's mouth so to speak.

Goodluck