High Number of Dropped Tx packets normal?

[André Beck]

Hi Libras,

in a situation like described by the thread owner, transmit queue overruns are inevitable. There are a tenfold of systems running at 1000Base firing their traffic onto a single egress port to the server in question, running at just 1000Base itself. Without perfectly working flow control, microbursts and even normal traffic will overrun the egress queue, no matter what. That's exactly how a switch paces the data streams flowing through it when they enter congestion land. So some number of drops here is just a sign of how the switch manages to cope with the highly suboptimal topology.

> I see lot of High Collusion or Drop Rate
> error on ports where machines are connected

These, on the other hand, are usually due to manually introduced duplex mismatches or simply sub-standard cabling.

They could also be just a side effect of a broadcast storm going on. If you see broadcast storms despite *STP being set up correctly, you might either have a broken box somewhere or you observe the phenomenon of micro-storms (broadcast storms lasting just some milliseconds), something I've seen repeatedly in larger campus spanning VLANs based topologies built with ProCurve switches. Dunno where it's coming from.

But your 2min blackout sounds more like the real thing.

Andre.

[RicN]

>To maximize the amount of memory available
>for buffering, you can configure to switch
>for 2 output queues.

Is this something that should be done on a high-traffic link without QoS enabled?

Say if I use the default 8 queues on some systems, as I understand it some amount of buffer memory is allocated to the different queues, but if I do not use QoS tagging, and all frames will be priority 0 (normal), will the other memory just lay allocated (but not used) and increase the risk for drops due to lower buffer memory?

[steve baum]

I have seen this problem for a while across gigabit links with 5412/3500 routing between Chassis. TX Drops This is a newly discovered problem we help HP TAC troubleshoot - they were able to reproduce the issue in the lab.

Large file transfers routing across vlans and gig links. File transfers would fail a high percentage the higher the mb rate across the gig the higher the tx drop. Really start to see at 200mb and increases exponentially.

[Greg Seamen]

Has there been any action with this. What was the result of HP being able to reproduce this in the lab.

Greg

[Tj2013]

Hi All, did anyone find a solution for this?  I have a 5406 Switches and they report really high Drop TX.  Servers are seeing erros and drops by checking with ifconfig on physical interfaces.

[lylehm]

I know this is a very old thread, but stumbled on this while troubleshooting a high drop count between my Procurve 5412 and a tenant's SonicWall. Figured it wouldn't hurt to add my results to the pile.

After all the typical troubleshooting... checking port speed/duplex/flow config; replacing cable; etc. I found the port had a rate limiter set.

To display port rate-limits enter from the CLI:

show rate-limit all [port-list]  ...  port-list is optional, eg. B8-B10 or C1,C5

You could also use 'show running' if you don't mind seeing everything!

To remove a port rate-limit from A1, you would enter the following at the CLI:

config

no int A1 rate-limit all in

no int A1 rate-limit all out

More info on the rate-limit command can be found in the Management and Configuration Guide.

Oh, and this useful tidbit is in that guide too:

Rate-limiting is intended for use on edge ports in a network. It is not recommended for use on links to other switches, routers, or servers within a network, or for use in the network core. Doing so can interfere with applications the network requires to function properly.

[devocite]

I though I might bump this thread, to help bring it up to date a bit.

I noticed the infamous, and missunderstood "00331 FFI: ST2-CMDR: port 1/8-High collision or drop rate." message on a switch.  I used to see this on ports with PCs connected at 100M.  I just don't see it anymore, with modern 1G hardware.

In my case, I found that new 'cheaper' phones has been purchased, that only have 100M NIC and a passthrough port for a PC.   The 'Drop TX' count of the phones with 1G NIC are far lower than the 100M phones.

In liue of a bad ethernet, NIC, OS issue, etc.. Drop TX are a common occurance. for most all high PCs connected, and pulling LAN data, i.e. from a server.

The cause is quite simple.  You have a server, with a insainly fast storage, connected at 1/10/40G capable to push line speed.  PCs can, for the most part, can handle this at 1G, but occasionally the PC will have to pause, and the switch port buffer will over run, and a retransmission will occure (Drop TX).

When traffic is egressing the server at a rate of 1Gbps, and tries to egress a 100Mbps port, it is ineviable that packets will be dropped, and retransmited, until the host gets all the data.

The concept is on different that a vehical racing down a Highway.  When you merge onto another Highway, you are able, for the most part, able to continue at the same speed.
Now going down that same Highway, the road changes from 4 lanes to 2 lanes.  The traffic begins to backup, you have collisions, etc.

In general, the Drop TX isn't much to worry about, as long as it is a small percentage of the over all traffic.

[devocite]

I wanted to follow up, to my previous post, with some things I considered afterwards...

Note: Test before implimenting any changes, as they may cause a network outage!

As I mentioned, in the previous post, in cases where the Ethernet port is running 100M in a 1G+ network,
and using VoIP, as long as there arn't VoIP quality issue, I wouldn't stress over the TX drops.
If you can't upgrade the device or NIC to GigEthernet, there isn't much you can do to solve the TX drops.
Most of the drops are to PCs, and TCP has all sorts of mechnisms to help deals with lost packets.

If this assurance isn't enough for you, here some additional things to look at, and consider:

I highly recomend going to HPE, and get the command reference PDF(s) for your switch and switch release.
The 16.x guild actually does a very good job going into how QoS works, and is implimented on the ArubaOS (Procurve) switches.
Note: I have run into commands, and command output in the documentation, that is are available in the CLI, so keep that in mind.

Physical layer diagnostics
On the 2930, 3810, and many other later 'Procurve' switches you can run the `test cable-diagnostics <port>` as a quick and dirty cable test.
This will disrupt traffic on the port being tested, so use with caution.

Ethernet port QoS Queue
Look at the output priority queues using the command `show interfaces queues all`

• Q1-Q2 is the lowest output priory (small buffer)
• Q3-Q4 is where all 'regular' traffic is prioritized (largest buffer)
• Q5-Q6 is for higher priority. (small buffer)
• Q7-Q8 is the highest priority queues, i.e. for Voice. (small buffer)

The Tx Drop Bytes is the sum of all the Egress Queue Dropped Bytes, and hopefully only Q3 shows any dropped bytes.
In cases where a PC is connected through phyiscal phone, you will see the greates number of dropped Bytes in Q3: the default.
You want to see Q7 or Q8 with traffic, and no preferably no drops.

VoIP QoS
If your network has a VoIP VLAN, for physical phones, you should, at least, have the Voice VLAN setup as thus: `vlan <ID> voice`

Enabling this feature, will preserve the QoS marking. Therefore, if the packet is mark as 802.1p 5 it will be placed in high priority queue 6.
Command `show qos queue-config` will show the tag to queue mapping.
Note: enabling this command will use LLDP to negotiate VLAN tagging for the Voice VLAN, so do research before using it.

Some phones have rather low, DSCP/CoS markings, which places the traffic in lower priority queues. You can re-mark all Voice VLAN traffic, so it will be placed in a higher
priory queue.
vlan <ID> qos priority 7
This works on voice and non-voice VLANs
.
NOTE: Reclassifying the qos priority on a PC to a higher priority is a really bad idea, and will net no advantage, especially since the queues are smaller.
i.e. If everyone with tickets to a concert, have express lanes tickets, then every waits the same as if they had regular tickets.
Since the area allocated, for the express ticket processing is smaller, there would be even slower proccessing.

Flow Control (802.3x)

• Flow control is a tricky subject, and wholy depends on the client to request a pause in traffic.
• The client device has to first support flow control.
• Most 100M chipsets don't support 802.3x flow control.
• The TX Drop is most likely to be caused by input/output buffer/queue exhastion, not the client not being able to keep up with traffic.
  Enabling flow control probably wont help, and may cause issues.

tcp-push-preserve
Per the Aruba Advanced Traffic Management Guide for ArubaOS switch 16.10:

• When enabled, the tcp-push-preserve command results in TCP packets with the push bit set are being
  given priority by buffering them on the ingress packet queue of the receiving port if the output buffer of
  the packets destination port is full. That way the packets remain in the input queue when the egress
  queue is congested. In some situations, a large amount of TCP traffic with the push bit set could
  overwhelm the input queue, causing packet drops and possible network outage.
• When disabled (the default), TCP packets flow with the push bit set and are dropped normally when the
  egress queue is congested.

In 15.17 and earlier releases, the TCP push preserve feature is enabled by default.
When upgrading to the 16.01 release, the default for tcp-push-preserve will be changed from enabled to disabled.
The default in 16.01 and releases is tcp-push-preserve disable.
This change was made due to performance considerations in modern networks.

I've tried enabling and disabling, and there was no appriciable improvment to TX Drops statistics, on the 100M Ethernet ports.
On the other hand, the command didn't incure any performace issues.

Good luck!