5120-EI Performance Issue

[Pete W]

I've got a strange issue that I'm just wondering whether anyone else has seen.

Edge

4 x 5120-EI switches

IRF using CX4 local connect modules in slot 1 (1->2, 2->3, 3->4, 4->1)

The IRF stack uplinks to the Core IRF using  2 x 10GbE SFP+ SR transceivers

The uplink transceivers are located in slot 2 of switches 1 and 2

Uplinks are bonded using LACP aggregation (LACP is working)

5120-EI code = a5120ei-cmw520-r2220p02.bin

Core

2 x 10504 Chassis

IRF using 2 x 10GbE

10504 Code = 10500-cmw520-r1208p03.bin

Issue

When I connect a host to switches 1 or 2 of any 5120-EI IRF stack, network performance (copying files to/from servers connected to core switches) is very fast - as I would expect (>100MB/s copy speeds). These are the switches where the uplinks are located.

When I connect to switches 3 or 4 of any 5120-EI IRF stack, network performance is painfully slow (2MB/s copy speeds). This traffic has to pass through the IRF stack before transiting the uplink.

I've had limited time to do troubleshooting as yet - except that this issue is consistent across all of my 17 x 5120-EI IRF stacks.

Any thoughts?

Pete

[Peter_Debruyne]

hmm, very strange behavior.

I would try to isolate the issue:

* verify iperf session directly on 1 stack between

  - pc on unit1/pc on unit2

  - pc on unit2/pc on unit3

  etc.

* verify any change in behavior when 1 irf link is down, test again with other irf link down

* verify cpu states

not sure if the 5120 support interface stats of the irf links ( think not, but you can still try it).

One of my first ideas was some possible loop on units3/4, which is overloading the irf links (pc on unit1 to core traffic would not use the irf link, but the direct uplink to the core, so that would explain why that would be fast while the other flows would be slow), however, I cannot imagine this to be the case for all your stacks ...

Keep us posted on your progress !

[Pete W]

Ok, I got to the bottom of this one eventually.

When I was investigating this issue I fired up Wireshark and noticed highly unusual unicast flooding across the stack members. A quick google search later and I found the following page:

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?prodSeriesId=4174705&objectID=c03056776

I have since applied the “mac-address mac-roaming enable” command, and performance has increased dramatically.

Regards,

Pete

[Peter_Debruyne]

Hi,

Thanks for posting your findings !

I had always been wandering why exactly they had this command.

This seems to indicate that each member of the IRF stack is still doing its own hardware mac-learning/aging, and this command would force the mac-tables to be synced by the software.

In that case (and based on your issue), it could be a recommended default setting to activate on an IRF system.

If anyone can provide additional input/insight, it would be welcome.

Best regards,Peter

[Vince_Whirlwind]

Agreed, very interesting.

I will be having a close look at my IRF stacks to see how they all behave.

[manuel.bitzi]

I had a similar bug few years ago on R2202. But they fixed it quickly.

Hope the bug is not returned, because im using 2220p02 at many customer sites.

Keep us updated.

br

Manuel

[MichaelM55]

Interesting: "mac-address mac-roaming enable" is also available on A5500-EI switching series. I am asking myself whether the mentioned flooding is the true reason for the tons of discarded packages on all of my 10Gbe IRF interfaces?

[manuel.bitzi]

Hi Michael

5500 EI has the Code from 5120 EI with additional routing-features. It's not astonishing that it has the same behaviour and issues.

If you have 40-60% discarded packets on the IRF-Interfaces of the IRF-Master, then it's an "normale" behavior of the IRF-Ring-Design. I have an open FR to fade out these "errors".

Best Regards

Manuel

[Apachez-]

Isnt the "link-aggregation load-sharing mode link-first" supposed to take of this?