Switches, Hubs, and Modems
cancel
Showing results for 
Search instead for 
Did you mean: 

High Number of Dropped Tx packets normal?

Cham Dor
Occasional Contributor

High Number of Dropped Tx packets normal?

Hi,

I have a ProCurve 5406zl with 72 gigabit hosts attached to it, with two ports having a large number of drops Tx packets(~4,818,078). The hosts attached to these two ports are NFS servers (DL380's) that receive a large amount of write traffic from the other 70 hosts attached to the switch.

My question is is this drop rate seem normal? As a percentage it doesn't seem high ( Unicast Tx : 2,069,990,077 ) but the folks that monitor the network traffic claim the dropped packets are way to high.

Are there other reasons other than misconfiguration that the packet drop would be large ( e.g. I assume there is only so much buffer space on the 5406zl, would it drop after the buffers fill? Is there a way to detect that condition? )

Using "ifconfig" on the hosts in question I don't see any drops,overruns, errors, etc.

Thanks for any assistance.

-Cham

Here is the output from show int on one of the ports in question:


Port Counters for port C18

Name : stor

Link Status : Up

Bytes Rx : 2,820,671,298 Bytes Tx : 3,370,764,929
Unicast Rx : 4,028,570,168 Unicast Tx : 2,069,990,077
Bcast/Mcast Rx : 658,465 Bcast/Mcast Tx : 37,823,169

FCS Rx : 0 Drops Tx : 4,818,078
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 369 Excessive Colln : 0
Total Rx Errors : 369 Deferred Tx : 0
15 REPLIES
Mohieddin Kharnoub
Honored Contributor

Re: High Number of Dropped Tx packets normal?

Hi

Drops every where could happen, but maybe with number its not normal.

I wonder if had some best practices in your network like segregating these 2 Servers in a different Vlan with a high QoS enabled.

I also wonder if you need to enable Jumbo frames for your NFS, or its using a normal packet size.

You may also have a look at the following useful link from ProCurve to troubleshoot such situations:

http://www.hp.com/rnd/library/troubleshoot_lan.htm

Good Luck !!!
Science for Everyone
Cham Dor
Occasional Contributor

Re: High Number of Dropped Tx packets normal?

Hi,

Thanks for the ideas. I will look at the document.

The 70 nodes are DL140's that don't support jumbo frames.

I am not sure how does the QoS/VLAN remedy buffer limitations? This is private switch and the only traffic on it is NFS.

Also on this switch 5406zl is the buffer space "per module", e.g. moving each server to its own module improve the situation ( if it is a buffer limitation) or are the internal buffers not related to the module itself. This is fully populated switch with plenty of empty ports I could move around.

Thanks!
Case Van Horsen
Frequent Advisor

Re: High Number of Dropped Tx packets normal?

Some ideas.

1) Do you have broadcast-limit enabled? If so, try disabling that.

2) To maximize the amount of memory available for buffering, you can configure to switch for 2 output queues. This does require a reboot.

3) What is the utilization on the links to the NFS servers? If they are very busy, you may need to increase the bandwidth to the servers by using multiple links. (trunking)

4) Does the behavior change if you enable flow-control?

casevh
rick jones
Honored Contributor

Re: High Number of Dropped Tx packets normal?

Ah, you did ask over here in switch land :) Probably should have deleted your HP-UX->networking post then...

Anyhow, I'll repeat that if those other things don't work-out, you could try trunking/bonding/aggregating two links from the servers to the switch(es) to increase the aggregate bandwidth and address the many-to-one bit. That or upgrade those connections to 10Gig :)
there is no rest for the wicked yet the virtuous have no pillows
Cham Dor
Occasional Contributor

Re: High Number of Dropped Tx packets normal?

Thanks for the suggestions. I will make sure tomorrow to disable broadcast limit and see what enabling flow control does. How does one configure only the two output ports?

Thanks!
Case Van Horsen
Frequent Advisor

Re: High Number of Dropped Tx packets normal?

To change the configuration for port C18:

config
interface C18
broadcast-limit 0
flow-control


casevh
rick jones
Honored Contributor

Re: High Number of Dropped Tx packets normal?

If the issue is that a bunch of writes "all at the same time" from the clients hits the one link into the server, is flow-control on the link to the server going to be sufficient? It would have to put back-pressure on the clients, and if it isn't enabled on the client ports I'd wonder if it would happen.

If going to 10G is not an option, I'd look into bonding two links between the swtich and the server in this case to increase aggregate bandwidth into the server.
there is no rest for the wicked yet the virtuous have no pillows
Christian Borén
Occasional Visitor

Re: High Number of Dropped Tx packets normal?

Did you ever find a solution to this problem ? I have the same problem on a 5400zl switch and can't seem to find why it's dropping packets.

During peak we are having about 500 mbit of traffic and there are a lot of drops.

Could it be that HP/Procurve has other errors included in the Drops Tx counter ? Such as CRC ?

Regards,
Chris
Libras
Occasional Advisor

Re: High Number of Dropped Tx packets normal?

I also got lot of these error and for 2 minutes complete network is down. i have 2 core switches and 6 edge switches . core switches with vrrp configuration. each edge switch is connected to cores with 20gb trunk ( 20 gb to core1 and 20 gb to core2 ). I see lot of High Collusion or Drop Rate error on ports where machines are connected. I dont see error on 10 GB ports. What could be the problem. Iam running all 5412 switches and core with 12:43 firmware and edges with 12:57.

Libras
André Beck
Honored Contributor

Re: High Number of Dropped Tx packets normal?

Hi Libras,

in a situation like described by the thread owner, transmit queue overruns are inevitable. There are a tenfold of systems running at 1000Base firing their traffic onto a single egress port to the server in question, running at just 1000Base itself. Without perfectly working flow control, microbursts and even normal traffic will overrun the egress queue, no matter what. That's exactly how a switch paces the data streams flowing through it when they enter congestion land. So some number of drops here is just a sign of how the switch manages to cope with the highly suboptimal topology.

> I see lot of High Collusion or Drop Rate
> error on ports where machines are connected

These, on the other hand, are usually due to manually introduced duplex mismatches or simply sub-standard cabling.

They could also be just a side effect of a broadcast storm going on. If you see broadcast storms despite *STP being set up correctly, you might either have a broken box somewhere or you observe the phenomenon of micro-storms (broadcast storms lasting just some milliseconds), something I've seen repeatedly in larger campus spanning VLANs based topologies built with ProCurve switches. Dunno where it's coming from.

But your 2min blackout sounds more like the real thing.

Andre.
RicN
Valued Contributor

Re: High Number of Dropped Tx packets normal?


>To maximize the amount of memory available
>for buffering, you can configure to switch
>for 2 output queues.

Is this something that should be done on a high-traffic link without QoS enabled?

Say if I use the default 8 queues on some systems, as I understand it some amount of buffer memory is allocated to the different queues, but if I do not use QoS tagging, and all frames will be priority 0 (normal), will the other memory just lay allocated (but not used) and increase the risk for drops due to lower buffer memory?


steve baum
Occasional Visitor

Re: High Number of Dropped Tx packets normal?

I have seen this problem for a while across gigabit links with 5412/3500 routing between Chassis. TX Drops This is a newly discovered problem we help HP TAC troubleshoot - they were able to reproduce the issue in the lab.

Large file transfers routing across vlans and gig links. File transfers would fail a high percentage the higher the mb rate across the gig the higher the tx drop. Really start to see at 200mb and increases exponentially.

Greg Seamen
Occasional Visitor

Re: High Number of Dropped Tx packets normal?

Has there been any action with this. What was the result of HP being able to reproduce this in the lab.

Greg
Tj2013
Occasional Visitor

Re: High Number of Dropped Tx packets normal?

Hi All, did anyone find a solution for this?  I have a 5406 Switches and they report really high Drop TX.  Servers are seeing erros and drops by checking with ifconfig on physical interfaces.

lylehm
Occasional Visitor

Re: High Number of Dropped Tx packets normal?

I know this is a very old thread, but stumbled on this while troubleshooting a high drop count between my Procurve 5412 and a tenant's SonicWall. Figured it wouldn't hurt to add my results to the pile.

After all the typical troubleshooting... checking port speed/duplex/flow config; replacing cable; etc. I found the port had a rate limiter set.

To display port rate-limits enter from the CLI:

show rate-limit all [port-list]  ...  port-list is optional, eg. B8-B10 or C1,C5

You could also use 'show running' if you don't mind seeing everything!

To remove a port rate-limit from A1, you would enter the following at the CLI:

config

no int A1 rate-limit all in

no int A1 rate-limit all out

More info on the rate-limit command can be found in the Management and Configuration Guide.

Oh, and this useful tidbit is in that guide too:

Rate-limiting is intended for use on edge ports in a network. It is not recommended for use on links to other switches, routers, or servers within a network, or for use in the network core. Doing so can interfere with applications the network requires to function properly.