High Number of Dropped Tx packets normal?

[Cham Dor]

Hi,

I have a ProCurve 5406zl with 72 gigabit hosts attached to it, with two ports having a large number of drops Tx packets(~4,818,078). The hosts attached to these two ports are NFS servers (DL380's) that receive a large amount of write traffic from the other 70 hosts attached to the switch.

My question is is this drop rate seem normal? As a percentage it doesn't seem high ( Unicast Tx : 2,069,990,077 ) but the folks that monitor the network traffic claim the dropped packets are way to high.

Are there other reasons other than misconfiguration that the packet drop would be large ( e.g. I assume there is only so much buffer space on the 5406zl, would it drop after the buffers fill? Is there a way to detect that condition? )

Using "ifconfig" on the hosts in question I don't see any drops,overruns, errors, etc.

Thanks for any assistance.

-Cham

Here is the output from show int on one of the ports in question:


Port Counters for port C18

Name : stor

Link Status : Up

Bytes Rx : 2,820,671,298 Bytes Tx : 3,370,764,929
Unicast Rx : 4,028,570,168 Unicast Tx : 2,069,990,077
Bcast/Mcast Rx : 658,465 Bcast/Mcast Tx : 37,823,169

FCS Rx : 0 Drops Tx : 4,818,078
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 369 Excessive Colln : 0
Total Rx Errors : 369 Deferred Tx : 0

[Mohieddin Kharnoub]

Hi

Drops every where could happen, but maybe with number its not normal.

I wonder if had some best practices in your network like segregating these 2 Servers in a different Vlan with a high QoS enabled.

I also wonder if you need to enable Jumbo frames for your NFS, or its using a normal packet size.

You may also have a look at the following useful link from ProCurve to troubleshoot such situations:
https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c01830354

Good Luck !!!

[Cham Dor]

Hi,

Thanks for the ideas. I will look at the document.

The 70 nodes are DL140's that don't support jumbo frames.

I am not sure how does the QoS/VLAN remedy buffer limitations? This is private switch and the only traffic on it is NFS.

Also on this switch 5406zl is the buffer space "per module", e.g. moving each server to its own module improve the situation ( if it is a buffer limitation) or are the internal buffers not related to the module itself. This is fully populated switch with plenty of empty ports I could move around.

Thanks!

[Case Van Horsen]

Some ideas.

1) Do you have broadcast-limit enabled? If so, try disabling that.

2) To maximize the amount of memory available for buffering, you can configure to switch for 2 output queues. This does require a reboot.

3) What is the utilization on the links to the NFS servers? If they are very busy, you may need to increase the bandwidth to the servers by using multiple links. (trunking)

4) Does the behavior change if you enable flow-control?

casevh

[rick jones]

Ah, you did ask over here in switch land :) Probably should have deleted your HP-UX->networking post then...

Anyhow, I'll repeat that if those other things don't work-out, you could try trunking/bonding/aggregating two links from the servers to the switch(es) to increase the aggregate bandwidth and address the many-to-one bit. That or upgrade those connections to 10Gig :)

[Cham Dor]

Thanks for the suggestions. I will make sure tomorrow to disable broadcast limit and see what enabling flow control does. How does one configure only the two output ports?

Thanks!

[Case Van Horsen]

To change the configuration for port C18:

config
interface C18
broadcast-limit 0
flow-control


casevh

[rick jones]

If the issue is that a bunch of writes "all at the same time" from the clients hits the one link into the server, is flow-control on the link to the server going to be sufficient? It would have to put back-pressure on the clients, and if it isn't enabled on the client ports I'd wonder if it would happen.

If going to 10G is not an option, I'd look into bonding two links between the swtich and the server in this case to increase aggregate bandwidth into the server.

[Christian Borén]

Did you ever find a solution to this problem ? I have the same problem on a 5400zl switch and can't seem to find why it's dropping packets.

During peak we are having about 500 mbit of traffic and there are a lot of drops.

Could it be that HP/Procurve has other errors included in the Drops Tx counter ? Such as CRC ?

Regards,
Chris

[Libras]

I also got lot of these error and for 2 minutes complete network is down. i have 2 core switches and 6 edge switches . core switches with vrrp configuration. each edge switch is connected to cores with 20gb trunk ( 20 gb to core1 and 20 gb to core2 ). I see lot of High Collusion or Drop Rate error on ports where machines are connected. I dont see error on 10 GB ports. What could be the problem. Iam running all 5412 switches and core with 12:43 firmware and edges with 12:57.

Libras

[André Beck]

Hi Libras,

in a situation like described by the thread owner, transmit queue overruns are inevitable. There are a tenfold of systems running at 1000Base firing their traffic onto a single egress port to the server in question, running at just 1000Base itself. Without perfectly working flow control, microbursts and even normal traffic will overrun the egress queue, no matter what. That's exactly how a switch paces the data streams flowing through it when they enter congestion land. So some number of drops here is just a sign of how the switch manages to cope with the highly suboptimal topology.

> I see lot of High Collusion or Drop Rate
> error on ports where machines are connected

These, on the other hand, are usually due to manually introduced duplex mismatches or simply sub-standard cabling.

They could also be just a side effect of a broadcast storm going on. If you see broadcast storms despite *STP being set up correctly, you might either have a broken box somewhere or you observe the phenomenon of micro-storms (broadcast storms lasting just some milliseconds), something I've seen repeatedly in larger campus spanning VLANs based topologies built with ProCurve switches. Dunno where it's coming from.

But your 2min blackout sounds more like the real thing.

Andre.

[RicN]

>To maximize the amount of memory available
>for buffering, you can configure to switch
>for 2 output queues.

Is this something that should be done on a high-traffic link without QoS enabled?

Say if I use the default 8 queues on some systems, as I understand it some amount of buffer memory is allocated to the different queues, but if I do not use QoS tagging, and all frames will be priority 0 (normal), will the other memory just lay allocated (but not used) and increase the risk for drops due to lower buffer memory?

[steve baum]

I have seen this problem for a while across gigabit links with 5412/3500 routing between Chassis. TX Drops This is a newly discovered problem we help HP TAC troubleshoot - they were able to reproduce the issue in the lab.

Large file transfers routing across vlans and gig links. File transfers would fail a high percentage the higher the mb rate across the gig the higher the tx drop. Really start to see at 200mb and increases exponentially.

[Greg Seamen]

Has there been any action with this. What was the result of HP being able to reproduce this in the lab.

Greg

[Tj2013]

Hi All, did anyone find a solution for this?  I have a 5406 Switches and they report really high Drop TX.  Servers are seeing erros and drops by checking with ifconfig on physical interfaces.

[lylehm]

I know this is a very old thread, but stumbled on this while troubleshooting a high drop count between my Procurve 5412 and a tenant's SonicWall. Figured it wouldn't hurt to add my results to the pile.

After all the typical troubleshooting... checking port speed/duplex/flow config; replacing cable; etc. I found the port had a rate limiter set.

To display port rate-limits enter from the CLI:

show rate-limit all [port-list]  ...  port-list is optional, eg. B8-B10 or C1,C5

You could also use 'show running' if you don't mind seeing everything!

To remove a port rate-limit from A1, you would enter the following at the CLI:

config

no int A1 rate-limit all in

no int A1 rate-limit all out

More info on the rate-limit command can be found in the Management and Configuration Guide.

Oh, and this useful tidbit is in that guide too:

Rate-limiting is intended for use on edge ports in a network. It is not recommended for use on links to other switches, routers, or servers within a network, or for use in the network core. Doing so can interfere with applications the network requires to function properly.

[devocite]

I though I might bump this thread, to help bring it up to date a bit.

I noticed the infamous, and missunderstood "00331 FFI: ST2-CMDR: port 1/8-High collision or drop rate." message on a switch.  I used to see this on ports with PCs connected at 100M.  I just don't see it anymore, with modern 1G hardware.

In my case, I found that new 'cheaper' phones has been purchased, that only have 100M NIC and a passthrough port for a PC.   The 'Drop TX' count of the phones with 1G NIC are far lower than the 100M phones.

In liue of a bad ethernet, NIC, OS issue, etc.. Drop TX are a common occurance. for most all high PCs connected, and pulling LAN data, i.e. from a server.

The cause is quite simple.  You have a server, with a insainly fast storage, connected at 1/10/40G capable to push line speed.  PCs can, for the most part, can handle this at 1G, but occasionally the PC will have to pause, and the switch port buffer will over run, and a retransmission will occure (Drop TX).

When traffic is egressing the server at a rate of 1Gbps, and tries to egress a 100Mbps port, it is ineviable that packets will be dropped, and retransmited, until the host gets all the data.

The concept is on different that a vehical racing down a Highway.  When you merge onto another Highway, you are able, for the most part, able to continue at the same speed.
Now going down that same Highway, the road changes from 4 lanes to 2 lanes.  The traffic begins to backup, you have collisions, etc.

In general, the Drop TX isn't much to worry about, as long as it is a small percentage of the over all traffic.

[devocite]

I wanted to follow up, to my previous post, with some things I considered afterwards...

Note: Test before implimenting any changes, as they may cause a network outage!

As I mentioned, in the previous post, in cases where the Ethernet port is running 100M in a 1G+ network,
and using VoIP, as long as there arn't VoIP quality issue, I wouldn't stress over the TX drops.
If you can't upgrade the device or NIC to GigEthernet, there isn't much you can do to solve the TX drops.
Most of the drops are to PCs, and TCP has all sorts of mechnisms to help deals with lost packets.

If this assurance isn't enough for you, here some additional things to look at, and consider:

I highly recomend going to HPE, and get the command reference PDF(s) for your switch and switch release.
The 16.x guild actually does a very good job going into how QoS works, and is implimented on the ArubaOS (Procurve) switches.
Note: I have run into commands, and command output in the documentation, that is are available in the CLI, so keep that in mind.

Physical layer diagnostics
On the 2930, 3810, and many other later 'Procurve' switches you can run the `test cable-diagnostics <port>` as a quick and dirty cable test.
This will disrupt traffic on the port being tested, so use with caution.

Ethernet port QoS Queue
Look at the output priority queues using the command `show interfaces queues all`

• Q1-Q2 is the lowest output priory (small buffer)
• Q3-Q4 is where all 'regular' traffic is prioritized (largest buffer)
• Q5-Q6 is for higher priority. (small buffer)
• Q7-Q8 is the highest priority queues, i.e. for Voice. (small buffer)

The Tx Drop Bytes is the sum of all the Egress Queue Dropped Bytes, and hopefully only Q3 shows any dropped bytes.
In cases where a PC is connected through phyiscal phone, you will see the greates number of dropped Bytes in Q3: the default.
You want to see Q7 or Q8 with traffic, and no preferably no drops.

VoIP QoS
If your network has a VoIP VLAN, for physical phones, you should, at least, have the Voice VLAN setup as thus: `vlan <ID> voice`

Enabling this feature, will preserve the QoS marking. Therefore, if the packet is mark as 802.1p 5 it will be placed in high priority queue 6.
Command `show qos queue-config` will show the tag to queue mapping.
Note: enabling this command will use LLDP to negotiate VLAN tagging for the Voice VLAN, so do research before using it.

Some phones have rather low, DSCP/CoS markings, which places the traffic in lower priority queues. You can re-mark all Voice VLAN traffic, so it will be placed in a higher
priory queue.
vlan <ID> qos priority 7
This works on voice and non-voice VLANs
.
NOTE: Reclassifying the qos priority on a PC to a higher priority is a really bad idea, and will net no advantage, especially since the queues are smaller.
i.e. If everyone with tickets to a concert, have express lanes tickets, then every waits the same as if they had regular tickets.
Since the area allocated, for the express ticket processing is smaller, there would be even slower proccessing.

Flow Control (802.3x)

• Flow control is a tricky subject, and wholy depends on the client to request a pause in traffic.
• The client device has to first support flow control.
• Most 100M chipsets don't support 802.3x flow control.
• The TX Drop is most likely to be caused by input/output buffer/queue exhastion, not the client not being able to keep up with traffic.
  Enabling flow control probably wont help, and may cause issues.

tcp-push-preserve
Per the Aruba Advanced Traffic Management Guide for ArubaOS switch 16.10:

• When enabled, the tcp-push-preserve command results in TCP packets with the push bit set are being
  given priority by buffering them on the ingress packet queue of the receiving port if the output buffer of
  the packets destination port is full. That way the packets remain in the input queue when the egress
  queue is congested. In some situations, a large amount of TCP traffic with the push bit set could
  overwhelm the input queue, causing packet drops and possible network outage.
• When disabled (the default), TCP packets flow with the push bit set and are dropped normally when the
  egress queue is congested.

In 15.17 and earlier releases, the TCP push preserve feature is enabled by default.
When upgrading to the 16.01 release, the default for tcp-push-preserve will be changed from enabled to disabled.
The default in 16.01 and releases is tcp-push-preserve disable.
This change was made due to performance considerations in modern networks.

I've tried enabling and disabling, and there was no appriciable improvment to TX Drops statistics, on the 100M Ethernet ports.
On the other hand, the command didn't incure any performace issues.

Good luck!