Aruba & ProVision-based
1748140 Members
3530 Online
108758 Solutions
New Discussion

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

 
SOLVED
Go to solution
jondehen
Advisor

HP 5412 10GbE Module Issues / Troubleshooting Tricks

We recently upgraded the firmware on our 5412Rzl2 (J9851A) as such: KB_15_17_0007 > KB_16_02_0013.  All operations are normal except some issues with our single 8-Port 10-GbE SFP+ v2 zl module (J9538A) as follows:

  • Occasional 1-2 second disconnect/reconnects in the switch log from the 10GbE ports
  • DRASTIC, but occasional, slowdown of speeds, which seem to self-correct eventually
  • High numbers of Discard Rxs
  • High numbers of Drop Txs (but not nearly as high as Discard Rxs)

We are looking for any additional troubleshooting commands or techniques (other than show log and show interface <PORT>) which might yield insight into the issue.

Right now we're unsure if the issue is:

  • The new firmware (both primary and secondary were updated but we can still rollback one of them)
  • Failing hardware (cables, module, ports, NICs)
  • Drivers (NICs)
  • Hosts

More Info:

  • All ports in this module are in an unroutable VLAN so there shouldn't be any commuication in/out
  • Hosts have Intel X710s (rev 01) NICs (recent but not the latest firmware)
  • I cannot verify if these ports had abnormal Discard or Drop counts before the firmware upgrade to compare to

Please let me know if I can provide any other details.  Thank you!

18 REPLIES 18
Stephen A Swain
Advisor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

I'd suggest logging a case and getting them to escalate to level2/3, you may find out they already know about this and a fix is slated for release. I've seen a few problems on Kx.16, and have already escalated one issue with a POE software bug - which turned out to be a known bug with unknown fix release date.

I wouldn't use Kx.16 in production yet, maybe test it in a lab for now until those early release cycle problems are resolved.

A useful troubleshooting tool is "show tech", and also have a look at "show instrumentation"

switch# show instrumentation ?

  • cam       Show internal version-dependent counters for debugging.
  • monitor       Show latest values for monitored parameters.
  • port       Show internal version-dependent counters for debugging the specified port.
  • resptime       Show service response time data for performance sensitive operations registered for response time measurement.
  • routing       Show routing related instrumentation parameters.
  • vlan       Show internal version-dependent counters for debugging the specified VLAN.

There is also a debug mode I've used in the past, that goes really deep into the "tech support" areas, but it gets quite complicated and probably is deeper than most customers would want to go.

http://networkgeekstuff.com/networking/procurve-and-hidden-command-line/

Search for term: edomtset

 

 

parnassus
Honored Contributor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

Curious to know if (and how) those eight 10GbE ports - of the HP 8-port 10-GbE SFP+ v2 zl Module - are all used concurrently (maybe ports overcommit/oversubscription could/couldn't enter in the picture so having a role in the issue)...first of all start with collecting the status of each Transceiver used on those ports, what the command

show interfaces transceiver n detail (where n is the port number)

reports?

Supposing that nothing else changed but the Firmware then the actual Firmware could be the first culprit one think of...but to diagnose that - without being necessarily biased by the concept "bad new Firmware versus good old Firmware" (I mean without considering other possible sources of issues) - you should be sure enough that exactly nothing else had changed in your environment before you did that Firmware upgrade.


I'm not an HPE Employee
Kudos and Accepted Solution banner
jondehen
Advisor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

Thanks for the replies!  I dug through those commands and didn't find anything particularly useful, although perhaps much of it is past my understanding.  I did check the transceiver statuses but not sure what to look for.

The firmware was the only thing which changed, unless you count the loss of network connectivity for the devices.  The devices in question are some VMware hosts and a few NAS devices.  We're going to try a full reboot of everything once we can afford downtime.

Can anyone explain the differences between firmware versions? (Major vs minor vs incremental) <MAJOR>.<MINOR>.<INCREMENTAL>

Perhaps the explaination of the three parts of the version number will help explain which firmware I should choose when upgrading...

 

parnassus
Honored Contributor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

Mmm...the best document I've read is the HP ProVision Software Release Process (2015): it should be explain exactly what you're looking for...

Don't you want to post and share (first trim all possible Serial Numbers and other relevant sensible information about your products/configurations) the result of the command above run against your various 10Gb Transceivers interfaces?


I'm not an HPE Employee
Kudos and Accepted Solution banner
jondehen
Advisor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

Excellent PDF, thank you!  So the show interface transceiver n detail command is identical for all 8 ports, except for the incrementing Interface Index and the Serial Number....

Transceiver in L1
Interface Index : 353 (varies)
Type : SFP+DA7
Model : J9285B
Connector Type : Vendor specific
Wavelength : n/a
Transfer Distance : 7m (copper),
Diagnostic Support : None
Serial Number : <VARIES>

parnassus
Honored Contributor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

Are those DAC Cables installed correctly (respecting the minimum bend radius, not below 1")?


I'm not an HPE Employee
Kudos and Accepted Solution banner
jondehen
Advisor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks


@parnassus wrote:

Are those DAC Cables installed correctly (respecting the minimum bend radius, not below 1")?


It appears that they are all installed with at least this minimum.

Michael Patmon
Trusted Contributor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

We are in the process of root causing an issue for that specific module (J9538A) on ports 4, 5, and 6.  In the meantime there is a configuration option you can disable that should alleviate the symptoms:

HP-Switch-5406Rzl2(config)# no tcp-push-preserve

There is a low level issue causing head of line blocking on those ports in the presence of of large amounts ot TCP traffic with the push bit set.  

 

HP-Switch-5406Rzl2(config)# tcp-push-preserve help
Usage: [no] tcp-push-preserve

Description: Enable TCP Push Preserve mode. This mode determines the
flow of the TCP packets that have the PUSH flag set. When
this mode is enabled and the egress queue is full, TCP
packets with the PUSH flag set are queued at the head of the
ingress queue for egress queue space. This might delay
subsequent incoming packets in the same queue. When this
mode is disabled and the egress queue is full, TCP packets
with the PUSH flag set are dropped from the head of the
ingress queue.

By default, this mode is enabled. Disable this mode when a
large number of TCP packets with the PUSH flag are being
dropped due to congestion.

jondehen
Advisor

Re: HP 5412 10GbE Module Issues / Troubleshooting Tricks

Interesting, Michael.  I think we're going to boot a dormant host on that module with a live DVD, mirror an active port to it, and use wireshark to investigate the actual traffic.  We'll hopefully be able to see what is being dropped or discarded, as well as if any of the TCP packets are indeed using the PSH flag or not.

Our issues also seem low level, and we'll likely end up rolling back firmware.  First, to the previous, and then second, to some newer ones (but not the absolute latest 16.02.xxxx).

I'll update here once we find more results.