Hi, in which HPE Modules are installed your (Genuine, right?) HPE X132 10G SFP+ LC LR Transceivers (J9151A)?
Supported HPE Modules for the usage of J9151A SFP+ Transceivers are:
- J9536A 20-Port Gig-T PoE+ / 2-Port SFP+ v2 zl Module
- J9538A 8-Port 10-GbE SFP+ v2 zl Module
- J9548A 20-Port Gig-T / 2-Port SFP+ v2 zl Module
- J9990A 20-Port PoE+ / 4-Port SFP+ v3 zl2 Module
- J9993A 8-Port 1G/10GbE SFP+ v3 zl2 Module
Any other HP Module is not supported for the usage of J9151A.
The 10GBASE-LR (10G Long Range, up to 10 km length) standard requires single mode Fiber Optic cables (working at 1310 nm) with a core/cladding of 9/125 um, is that fulfilled by actual FO inter-Switch/Building cabling?
I'm not an HPE Employee
Great, that's reassuring.
Were you able to isolate a particular 10G-Base LR uplink where this issue shows up or any of yours 10G-Base LR uplinks shows this issue?
Software versions (excluded the case this is a new bug, which I really don't know) should be OK if all involved Switches are using the same KB.16.01.0007 or newer (on the same Branch there is the KB.16.01.0008 recently released) even if upgrading to the latest Branch KB.16.02 should not be mandatory regarding the Transceiver's support (As example the KB.16.02.0010 <-- a side note: I don't know if it is a typo or not but I noticed that KB.16.02 software branch, exactly as happen with the KB.15.18 software branch, states to use the BootROM KB.15.01.0001. BootROM KB.16.01.0006 is used instead on the KB.16.01 software branch).
Can you provide Ports status (show interface brief or show interface config) for all involved 10G SFP+ ports and Transceivers status (show tech transceivers) for all involved Transceivers?
I'm not an HPE Employee
Even if you didn't explicitly wrote that your network topology is a star with the "core" connected to many other "edge" switches using single link F.O. uplinks, I presume it is (I shouldn't be wrong).
Said so is...the E2<->A21 link (E2 Port on the "core" Switch connected straight to the A21 Port of one of the "edge" Switch) the problematic one?
If so how is it possible that the FCS Rx errors (and Giant Rx errors too) on the "edge" side haven't any corresponding Tx errors of the same types on the "core" side? That's strange...
What's about Fiber Optic Patch Cables? Are they all Single Mode LC (UPC) / LC (UPC) from the SFP+ port up to the Optical Patch Panel on the Rack or what? Have you tried to exchange them?
I'm not an HPE Employee
I ran out of ideas...a last question: are the Aruba 5400R zl2 Switches equipped only with v3 zl2 Modules or there is a mix of v2+v3 Modules an so you're operating all the Switches forcing the v3 Modules to work in compatibility mode to v2 Modules (and so forcing the v3 Modules to behave like v2 Modules)?
The command show module will help to understand which Module's versions each Switch is equipped with.
I ask because would be interesting to see if a configuration that impose to the 5400R zl2 to not run its v3 Modules in compatibility mode (with the no allow-v2-modules command which will concurrently disable any v2 Module installed and will enable only v3 Modules to operate) may/may not have an impact on the issue or not.
Mmmm...cables...those FCS Errors...cables, it's a pity that Time Domain Reflectometry (TDR), introduced on v3 Modules to detect cable faults, doesn't work on 10G links...
I'm not an HPE Employee
Jkane wrote: Now having said that I have not tried a different brand of cable if you strongly feel that it could still be the patch cables I am willing to buy 3 new ones of a different brand to try?
Giving that the cost of 3 new Patch Cables (LC (UPC) / LC (UPC) 1310 nm single mode) is not so expensive, I think it's worth a try.
Jkane wrote: The only issue again is do ALL of the modules in the switch need to be in V3? I can put them into the V3 J9993A module on the core but Allow V2 modules is on for that switch and cant be disabled.
Are you asking about that regarding the usage of TDR or are you asking that generally speaking?
For TDR feature usage, as you may have read yet on 5400R zl2 ArubaOS-Switch Management and Configuration Guide for KB.16.01, it works only on v3 zl2 Modules that provide 100 Base-TX and 1000 Base-T ports (like the J9990A but I suspect only on its first 20 copper ports, while the J9993A is not listed as supported by that feature since it has SFP+ ports only) so I doubt that TDR feature will be of any help when tried on Optical Transceivers...but, absolutely, generally speaking (so not TDR related) it's not necessary nor mandatory to work with either v2 zl2 Modules or v3 zl2 Modules...you can have a 5400R zl2 and work with both in a mix...it's just a matter of having the 5400R zl2 switch that works with v3 zl2 Modules' 6th generation ASICSs at their full potential (which can/can't be related to your issue) instead of made those v3 zl2 Modules working in compatibility mode (demoted to v2 zl2 Modules' feature set).
So, for convenience, you can safely try to perform a no allow-v2-modules on Edge switches (read the manual about that command for its usage guidelines) and leave the actual settings (allow-v2-modules enabled) on the Core one...but I don't want to encourage you to introduce another difference since, as written, I'm not sure it will have for sure a benefical impact on your issue specifically...was only a side note.
Are your 10G Fiber Optics uplinks all distributed from both J9993A Module C and Module E on the Core or just from the Module E only? In other terms how are distributed the uplinks from the Core switch?
If the cables tracks are really OK...probably it's a bug, but it appears really strange.
I'm not an HPE Employee
Jkane wrote: Do you have something that you would recommend for LC/LC SM cables?
Hi Jake, I thought, just as example, something similar to those ones advertised here (note that KK code stands for LC/UPC on both duplex connectors' ends).
I'm not an HPE Employee
Judging by (optical) Tx/Rx Power Levels reported by the J9151 Transceiver's DOM (Diagnostic Optical Monitoring), used on one end of your longest run (is it just about 1 km long right?), reported here:
Transceiver in A21 Interface Index : 21 Type : SFP+LR Model : J9151A Connector Type : LC Wavelength : 1310nm Transfer Distance : 10.0km (9um), Diagnostic Support : DOM Serial Number :Private Info ErasedStatus Temperature : 41.109C Voltage : 3.2729V Tx Bias : 36.716mA Tx Power : 0.5589mW, -2.527dBm Rx Power : 0.3268mW, -4.857dBm Time Stamp : Mon Aug 29 08:02:22 2016
That Transceiver's operating conditions look, AFAIK, quite normal (Temperature too).
A Tx Power of about 0,5 mW and a Rx Power of about 0,3 mW (where I suppose m stands for milli as in milli-Watt) mean respectively 500 uW and 300 uW (where u stands for micro-Watt), those values appear in the range of "normal operation" considered the lenght of your fiber optic longest run in which these transceivers are used.
A sensible difference between the optical Tx Power and the optical Rx Power measured on each end means that transceivers are not working on a very short run (let's say below 20/30 meters or more?)...that difference is the normal path loss...moreover that difference should be almost the same once measured on both links (Transceiver 1 Tx <--> Rx Transceiver 2 and Transceiver 1 Rx <--> Tx Transceiver 2).
The lower Tx/Rx Power values (expressed in Watt) - the higher (absolute) Tx/Rx Power levels values (expressed in dBm) - the worst the Transceiver working conditions are.
So if you see a Power level sensibly higher than, let's say, generally 6 or 7 dBm (>> |-7| dBm) - corresponding to a very very low Power wattage - and depending if you're evaluating the Tx or the Rx levels (those levels are differents [*]...so Rx normal operating levels can be slightly higher than Tx normal operating levels)...then that value means there is something wrong with the Tx or the Rx optic (dirty?) or with the whole fiber link (if both values are bad)...and that condition should trigger/generate a sort of Transceiver Low Power Alarm/Warning.
[*] probably Warning/Alarm's for signal Low level thresholds can be found at about -7/-10 dBm for optical Tx Power and -11/-14 dBm for optical Rx Power. Warning/Alarm's for signal High level thresholds instead can be found in the range of about +2 to -1 dBm for both Rx and Tx.
I'm not an HPE Employee
Hi SteveHP,
That's interesting...is the KB.16.02.0012 already published?
I checked few moments ago but the very latest Software Release was the KB.16.02.0010 (Published on August, 19th 2016).
Do you think that a (congested) ingress packet queue of the receiving port could be the root source of the issue?
To check the tcp-push-preserve feature status (globally) is also available the show tcp-push-preserve command.
Probably the tcp-push-preserve feature is yet enabled by default on these Switches since its introduction with the KB.15.15.0008 release (was the 2014).
Side note: interestingly - as commands - the [no] tcp-push-preserve and the show tcp-push-preserve are erroneously reported on the HPE ArubaOS-Switch Feature and Command Index 16.02 as consultable on the ArubaOS-Switch Management and Configuration Guide (MCG)...instead they are consultable exactly on the ArubaOS-Switch Advanced Traffic Management Guide (ATMG) at page 270.
I'm not an HPE Employee