Flex-10 VC Drops Packets @ 5 Mega Packet Per Second

[Nima Gh.]

We have a c7000 blade system.
We are sending packets on 5Mpps from external server (with ixgbe card) through UPlink X2 to a server blade with BroadCom 10G LOM through internal port d15.
We saw the received amount of 1.8Mpps!! then we monitored the UPlink X2 and d15 internal port.
The IN traffic to X2 was 5Mpps but the OUT traffic of d15 port was 1.8 Mpps!!!
It seems that flex-10 itself drops the packets!!!
Is it possible???

[Frank Gessner]

Hi Nima,

whats the speed of the assigned network in the pofile of the blade where you are sending the packets? >= 5MB ?
Which Version of bnx2x-driver are you using.
>= 1.52-14 ?
Which OS?

regards, Frank

[rick jones]

What was the CPU uiltization of each core in the blade? It is entirely possible that you were saturating one of the cores taking interrupts from the NC532i. Particularly if stream of packets was considered a single "flow" by the NIC (ie if they are all to/from the same IP address) and so were not spread-out across cores.

[Nima Gh.]

Is it possible for Flex-10 to negotiate with the NIC Cards and perform kind of an overflow control? Is there any way to disable this overflow mechanism for VC Flex-10?

[rick jones]

10GbE standards offer the prospect of flow-control via things called pause frames. I believe it is possible for a LOM to send pause frames to the Flex-10.

While you are checking the per-core CPU utilization on the blade, you can also look at the ethtool -S statistics and see if any of them talk about sending pause frames.

[Nima Gh.]

This is the point: It seems that the Flow Control is already enabled and we want to disable it!
I clarified my purpose here:
http://forums11.itrc.hp.com/service/forums/questionanswer.do?threadId=1447807

Anyway, it would be great if there could be anyway to turn off the flow control of either the FlexNIC or Flex-10 VC.

[HEM_2 · Accepted solution]

you can turn off flow control globally in VC after firmware 2.30 or later.

To turn it off log in to the CLI and issue the following command:

set advanced-networking FlowControl=off

*** IMPORTANT - you need to do this in a scheduled downtime or be aware that when you make this setting the VC downlink ports will be temporarily disconnected to affect the change.

[Nima Gh.]

Thank you very much!
And a point! what is PacketBufferOverallocationRatio Parameter?

[Nima Gh.]

I even changed the over allocation parameter but there is no change!

[Nima Gh.]

A point! it seems that the 2mpps is the maximum packet rate of the bnx2x!

Are embedded NICs of Proliant G6 Blade Servers assumed as ports of a single dual-port ethernet card or two different single-port cards?

[rick jones]

2mpps might be the pps limit for the HW driven by the bnx2x driver, but it is not necessarily the limit of the bnx2x driver itself. On the half-height blades, there is a single, dual-port LOM. The full-height "i2" Integrity blades have a pair of dual-port LOMs. I don't recall how many are on the full-height ProLiant blades.

I am not sure that all G6 blades have the same NC532m (driven by the BCM57711E chip) LOM. Some may have a different chip for their LOM. Best check the quickspecs etc of each blade.

Also, I am informed the PPS limit of the BCM57711E is 2Mpps in each direction.

[Nima Gh.]

Thanks for your reply! As I got informed, This is the Limit of Firmware! Is there a single firmware for both NICs or there are separate firmwares?

[rick jones]

The limit applies to the entire chip. As far as the *NIC* limits go, it does not matter whether one is using one port or both.

[Nima Gh.]

OMG!!! Then, who enforces these limits? and Where? Is this limit is hard-wired or is matter of firmware? Ann, how could we override these limits?

[rick jones]

The limit is as fast as the chip can go. It isn't "enforced" per se.

What is the end-goal of this testing? What sort of task do you wish to perform with the system?

[Nima Gh.]

You can consider it as a load balancer with input rate of (3mpps,10Gbps).

[rick jones]

If you do indeed need a 3Mpps input rate, and presumably a 3Mpps output rate to the "back ends" then I would suggest getting in touch with your HP sales rep and/or reseller and see about getting some of the of the 10GbE Mezz cards to try-out. While I suspect it is probably clear from the name, keep in mind that the NC532m uses the same BCM57711E chip as the NC532i LOM. Were I not one of the cobbler's children (*) :) I would be inclined to try an NC550m or NC542m.


(*) from the old saying "The cobbler's children have no shoes."

BTW, just to get the number out there, "link-rate" PPS for minimum sized frames is something like 14.7 Mpps for 10GbE. Which then leaves me wondering, so I will ask, how do you know your input pps rate will be limited to 3Mpps?

[Nima Gh.]

Thanks, Then I have to give up the on-board FlexNIC option and look for a mezzanine card!