High Number of Dropped Tx packets normal?

[Cham Dor]

Hi,

I have a ProCurve 5406zl with 72 gigabit hosts attached to it, with two ports having a large number of drops Tx packets(~4,818,078). The hosts attached to these two ports are NFS servers (DL380's) that receive a large amount of write traffic from the other 70 hosts attached to the switch.

My question is is this drop rate seem normal? As a percentage it doesn't seem high ( Unicast Tx : 2,069,990,077 ) but the folks that monitor the network traffic claim the dropped packets are way to high.

Are there other reasons other than misconfiguration that the packet drop would be large ( e.g. I assume there is only so much buffer space on the 5406zl, would it drop after the buffers fill? Is there a way to detect that condition? )

Using "ifconfig" on the hosts in question I don't see any drops,overruns, errors, etc.

Thanks for any assistance.

-Cham

Here is the output from show int on one of the ports in question:


Port Counters for port C18

Name : stor

Link Status : Up

Bytes Rx : 2,820,671,298 Bytes Tx : 3,370,764,929
Unicast Rx : 4,028,570,168 Unicast Tx : 2,069,990,077
Bcast/Mcast Rx : 658,465 Bcast/Mcast Tx : 37,823,169

FCS Rx : 0 Drops Tx : 4,818,078
Alignment Rx : 0 Collisions Tx : 0
Runts Rx : 0 Late Colln Tx : 0
Giants Rx : 369 Excessive Colln : 0
Total Rx Errors : 369 Deferred Tx : 0

[Mohieddin Kharnoub]

Hi

Drops every where could happen, but maybe with number its not normal.

I wonder if had some best practices in your network like segregating these 2 Servers in a different Vlan with a high QoS enabled.

I also wonder if you need to enable Jumbo frames for your NFS, or its using a normal packet size.

You may also have a look at the following useful link from ProCurve to troubleshoot such situations:
https://support.hpe.com/hpesc/public/docDisplay?docId=emr_na-c01830354

Good Luck !!!

[Cham Dor]

Hi,

Thanks for the ideas. I will look at the document.

The 70 nodes are DL140's that don't support jumbo frames.

I am not sure how does the QoS/VLAN remedy buffer limitations? This is private switch and the only traffic on it is NFS.

Also on this switch 5406zl is the buffer space "per module", e.g. moving each server to its own module improve the situation ( if it is a buffer limitation) or are the internal buffers not related to the module itself. This is fully populated switch with plenty of empty ports I could move around.

Thanks!

[Case Van Horsen]

Some ideas.

1) Do you have broadcast-limit enabled? If so, try disabling that.

2) To maximize the amount of memory available for buffering, you can configure to switch for 2 output queues. This does require a reboot.

3) What is the utilization on the links to the NFS servers? If they are very busy, you may need to increase the bandwidth to the servers by using multiple links. (trunking)

4) Does the behavior change if you enable flow-control?

casevh

[rick jones]

Ah, you did ask over here in switch land :) Probably should have deleted your HP-UX->networking post then...

Anyhow, I'll repeat that if those other things don't work-out, you could try trunking/bonding/aggregating two links from the servers to the switch(es) to increase the aggregate bandwidth and address the many-to-one bit. That or upgrade those connections to 10Gig :)

[Cham Dor]

Thanks for the suggestions. I will make sure tomorrow to disable broadcast limit and see what enabling flow control does. How does one configure only the two output ports?

Thanks!

[Case Van Horsen]

To change the configuration for port C18:

config
interface C18
broadcast-limit 0
flow-control


casevh

[rick jones]

If the issue is that a bunch of writes "all at the same time" from the clients hits the one link into the server, is flow-control on the link to the server going to be sufficient? It would have to put back-pressure on the clients, and if it isn't enabled on the client ports I'd wonder if it would happen.

If going to 10G is not an option, I'd look into bonding two links between the swtich and the server in this case to increase aggregate bandwidth into the server.

[Christian Borén]

Did you ever find a solution to this problem ? I have the same problem on a 5400zl switch and can't seem to find why it's dropping packets.

During peak we are having about 500 mbit of traffic and there are a lot of drops.

Could it be that HP/Procurve has other errors included in the Drops Tx counter ? Such as CRC ?

Regards,
Chris

[Libras]

I also got lot of these error and for 2 minutes complete network is down. i have 2 core switches and 6 edge switches . core switches with vrrp configuration. each edge switch is connected to cores with 20gb trunk ( 20 gb to core1 and 20 gb to core2 ). I see lot of High Collusion or Drop Rate error on ports where machines are connected. I dont see error on 10 GB ports. What could be the problem. Iam running all 5412 switches and core with 12:43 firmware and edges with 12:57.

Libras