Trouble with 10GB trancievers on 5412R switches

[Jkane]

Hello,

We just purchased new 5400R Aruba branded procurve switches for our high school and middle school this year. We also had new SM fiber intalled to all closets as well. 

All switch closets connect back to our core switch, again a 5412R, at 10GB using HPE 10-Gbe SFP+ 10KM LR 13010 Transceiver, Mfg Part# J9151A. 

The switches are all running firmware KB.16.01.0007 and Boot ROM KB.16.01.0006.

We are experience packet loss and excessive Errors Rx anywhere from 10-100 per minute. We are not experience drops Tx, Alignment Rx, or Collision tx. 2 days ago we had one of the closet connections go completly offline. The tranceivers would not re-light. 

If we install 1GB J4895C SFP 1310 trancievers in the same fiber pathing using the same patch cables into the same ports on the switches they will run happily without issue. 

School has not started yet and these switches are under almost no load at this point... hardly any traffic passing through them. I have reached out to our vendor to see if they can help me with getting HPE support but thought i would ask here as well. 

Thanks,

~jake

[parnassus]

Hi, in which HPE Modules are installed your (Genuine, right?) HPE X132 10G SFP+ LC LR Transceivers (J9151A)?

Supported HPE Modules for the usage of J9151A SFP+ Transceivers are:

• J9536A 20-Port Gig-T PoE+ / 2-Port SFP+ v2 zl Module
• J9538A 8-Port 10-GbE SFP+ v2 zl Module
• J9548A 20-Port Gig-T / 2-Port SFP+ v2 zl Module
• J9990A 20-Port PoE+ / 4-Port SFP+ v3 zl2 Module
• J9993A 8-Port 1G/10GbE SFP+ v3 zl2 Module

Any other HP Module is not supported for the usage of J9151A.

The 10GBASE-LR (10G Long Range, up to 10 km length) standard requires single mode Fiber Optic cables (working at 1310 nm) with a core/cladding of 9/125 um, is that fulfilled by actual FO inter-Switch/Building cabling?

[Jkane]

Parnassus,

Thanks for your reply. 

Yes they are Genuine HPE trancievers. They are all installed in HPE Module J9990A in all of the switches. 

Our fiber is all 9/125 um single mode fiber as well as patch cables certified for 1310.

All runs are under under 1KM.

Thanks,

~jake

[parnassus]

Great, that's reassuring.

Were you able to isolate a particular 10G-Base LR uplink where this issue shows up or any of yours 10G-Base LR uplinks shows this issue?

Software versions (excluded the case this is a new bug, which I really don't know) should be OK if all involved Switches are using the same KB.16.01.0007 or newer (on the same Branch there is the KB.16.01.0008 recently released) even if upgrading to the latest Branch KB.16.02 should not be mandatory regarding the Transceiver's support (As example the KB.16.02.0010 <-- a side note: I don't know if it is a typo or not but I noticed that KB.16.02 software branch, exactly as happen with the KB.15.18 software branch, states to use the BootROM KB.15.01.0001. BootROM KB.16.01.0006 is used instead on the KB.16.01 software branch).

Can you provide Ports status (show interface brief or show interface config) for all involved 10G SFP+ ports and Transceivers status (show tech transceivers) for all involved Transceivers?

[Jkane]

We are seeing this issue on links to 3 or 4 switches atm. 1 is paticualrly bad so far averaging 8-10% packet loss. This is the same one that I tested putting the HPE 1GB trancievers in and then tested and it dropped no packets and had no errors. 

All of the switches are for sure running the exact same version of firmware I double checked that. I also tried testing with flow control on and off, since its about the only thing you can change on 10gb link, and it made no difference. 

Below is the port info for the really bad edge switch in question and the core switch they all connect too. I also included the detail port counters in case its of use.

Edge Switch

show interfaces brief A21

Status and Counters - Port Status

| Intrusion MDI Flow Bcast
Port Type | Alert Enabled Status Mode Mode Ctrl Limit
------------ ---------- + --------- ------- ------ ---------- ---- ---- -----
A21 SFP+LR | No Yes Up 10GigFD NA off 0

show tech transceivers

transceivers

Transceiver Technical Information:
Port # | Type | Prod # | Serial # | Part #
-------+-----------+------------+------------------+----------
A21 | SFP+LR | J9151A | Private Info Erased| 1990-3883

Status and Counters - Port Counters - Port A21

Name :
MAC Address : Private Info Erased
Link Status : Up
Totals (Since boot or last clear) :
Bytes Rx : 104,493,639 Bytes Tx : 71,316,404
Unicast Rx : 282,124 Unicast Tx : 296,783
Bcast/Mcast Rx : 35,203 Bcast/Mcast Tx : 2368
Errors (Since boot or last clear) :
FCS Rx : 77,095 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 657 Excessive Colln : 0
Total Rx Errors : 77,752 Deferred Tx : 0

Core Switch

show interfaces brief e2

Status and Counters - Port Status

| Intrusion MDI Flow Bcast
Port Type | Alert Enabled Status Mode Mode Ctrl Limit
------------ ---------- + --------- ------- ------ ---------- ---- ---- -----
E2 SFP+LR | No Yes Up 10GigFD NA off 0

show tech transceivers

transceivers

Transceiver Technical Information:
Port # | Type | Prod # | Serial # | Part #
-------+-----------+------------+------------------+----------
C2 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C3 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C4 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C5 | SFP+LR | J9151A | Private Info Erased| 1990-3883
C6 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E2 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E3 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E6 | SFP+LR | J9151A |Private Info Erased | 1990-3883
E7 | SFP+LR | J9151A | Private Info Erased| 1990-3883
E8 | SFP+LR | J9151A | Private Info Erased| 1990-3883

Status and Counters - Port Counters - Port E2

Name : FHTMS Library Trunk 2
MAC Address : Private Info Erased
Link Status : Up
Totals (Since boot or last clear) :
Bytes Rx : 80,772,325 Bytes Tx : 198,874,653
Unicast Rx : 328,852 Unicast Tx : 397,444
Bcast/Mcast Rx : 2697 Bcast/Mcast Tx : 42,067
Errors (Since boot or last clear) :
FCS Rx : 0 Drops Tx : 0
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 0 Excessive Colln : 0
Total Rx Errors : 0 Deferred Tx : 0

Thanks,

~jake

[parnassus]

Even if you didn't explicitly wrote that your network topology is a star with the "core" connected to many other "edge" switches using single link F.O. uplinks, I presume it is (I shouldn't be wrong).

Said so is...the E2<->A21 link (E2 Port on the "core" Switch connected straight to the A21 Port of one of the "edge" Switch) the problematic one?

If so how is it possible that the FCS Rx errors (and Giant Rx errors too) on the "edge" side haven't any corresponding Tx errors of the same types on the "core" side? That's strange...

What's about Fiber Optic Patch Cables? Are they all Single Mode LC (UPC) / LC (UPC) from the SFP+ port up to the Optical Patch Panel on the Rack or what? Have you tried to exchange them?

[Jkane]

Yes it is a star topology with all of the edge switches connected back using single FO uplinks using the 10GB trancievers. 

You are also right in regard to E2 is the core switch and A21 is the edge switch port. And yes that A21 is the problematic switch in question. 

In regards to why the errors are not matching up on the core side I am not certain either. That is why im reaching out for help. 

the cables we are using are these https://www.neweggbusiness.com/product/product.aspx?item=9siv00c20k8711

The fiber path is:

Core switch (E2) -> fiber panel ( from HS under parking lot to MS) -> fiber patch (jumper) -> Edge closet fiber panel -> Edge switch (A21)

We had our fiber contractor come back and test, with the excat same patch cables (all 3), from the HS to the MS and it came up fine on the fluke. Also as I said if i use the 1GB J4895C SFP 1310 trancievers in the exact same ports on both the core switch and edge switch using the same patch panels we have no packet loss and also have not tx or rx errors. 

Thanks,

~jake

[AbeAbe]

Hi,

have you tried "Diagnostic Optical Monitoring"

You can see it with "show interface tranceiver E2 detail"

More info:

http://h22208.www2.hpe.com/eginfolib/networking/docs/switches/RA/15-18/5998-8161_ra_2620_mcg/content/ch11s05.html

hth

Alex

[parnassus]

I ran out of ideas...a last question: are the Aruba 5400R zl2 Switches equipped only with v3 zl2 Modules or there is a mix of v2+v3 Modules an so you're operating all the Switches forcing the v3 Modules to work in compatibility mode to v2 Modules (and so forcing the v3 Modules to behave like v2 Modules)?

The command show module will help to understand which Module's versions each Switch is equipped with.

I ask because would be interesting to see if a configuration that impose to the 5400R zl2 to not run its v3 Modules in compatibility mode (with the no allow-v2-modules command which will concurrently disable any v2 Module installed and will enable only v3 Modules to operate) may/may not have an impact on the issue or not.

Mmmm...cables...those FCS Errors...cables, it's a pity that Time Domain Reflectometry (TDR), introduced on v3 Modules to detect cable faults, doesn't work on 10G links...

[Jkane]

Abe,

Here is the output from details

Edge Switch

show interfaces transceiver a21 detail

Transceiver in A21
Interface Index : 21
Type : SFP+LR
Model : J9151A
Connector Type : LC
Wavelength : 1310nm
Transfer Distance : 10.0km (9um),
Diagnostic Support : DOM
Serial Number :Private Info Erased

Status
Temperature : 38.691C
Voltage : 3.2729V
Tx Bias : 36.334mA
Tx Power : 0.5589mW, -2.527dBm
Rx Power : 0.3769mW, -4.238dBm

Time Stamp : Fri Aug 26 06:19:13 2016

Core Switch

show interface transceiver E2 detail

Transceiver in E2
Interface Index : 130
Type : SFP+LR
Model : J9151A
Connector Type : LC
Wavelength : 1310nm
Transfer Distance : 10.0km (9um),
Diagnostic Support : DOM
Serial Number :Private Info Erased

Status
Temperature : 36.074C
Voltage : 3.3441V
Tx Bias : 35.250mA
Tx Power : 0.5716mW, -2.429dBm
Rx Power : 0.3530mW, -4.522dBm

Time Stamp : Fri Aug 26 06:20:18 2016

thanks,

~jake