Flow Control question for Virtual Connect (VC)

[chuckk281]

Laura had customer question on Flow Control settings in VC:

*************

I have a customer that has c7000 enclosures, each with four VC Flex10-10D modules – two are leveraged for NetApp connectivity. NetApp and VMware are recommending that Flow Control be disabled.  Because HP enables Flow Control by default & our documentation indicates this is standard practice, they are concerned about the ramifications of disabling it.

Can anyone tell me what HP’s recommended Flow Control setting in this scenario?

***********

From Brian:

I would go with the storage vendors recommendation typically.  For example StoreVirtual recommends flow control be enabled….

And input from Pedrag:

I doubt you ll find recommended setup on this. But what I can share is my latest experience with flow control where we lost all traffic within c7000. We hit into buffer limit due to enabled flow control on downlinks. And the only recommendation was to disable flow control on downlinks. As soon as we done that, we got traffics and rx dropped packets gone. Flow control is OS dependent mechanizm, so you should be careful. Also, it should be followed by firmware/driver compatibility guides. There is also advisory regarding the flow control and pause frames.

Info from Kelly:

I worked with a customer who had pre-4.xx VC firmware, we hit some pause frame flood issues (enclosure outages), and due to schedule and timing, we tried as a gap-fix to turn off flow control for a couple weeks until we could get a maintenance window.  We were initially advised by support the command (to change flow control) was non-disruptive.  After we issued the command (live), we had multiple ping drops and lost one Windows server (had to e-fuse it).  After – I recall we determined all links are dropped and re-negotiated with the change.  Someone may need to confirm that….

Also - I recall a difference of where flow control is enabled by default, I recall learning the downstream ports are on, but the upstream (uplinks) are disabled by default....  someone may need to confirm that too.  Via CLI, it does have THREE settings: auto|on|off

->set advanced-networking -help

Description:

  This command sets the Ethernet advanced network tunable parameters.

  Any changes may cause a network outage.

Syntax:

  set advanced-networking [-quiet] [PacketBufferOverallocationRatio=<ratio>]

  [FlowControl=<auto|on|off>]

Options:

  quiet : This option suppresses the user confirmation prompt. This

          option is useful for scripting operations.

Parameters:

  PacketBufferOverallocationRatio : (Optional) This sets the packet buffer

     over-allocation ratio for all VC-Enet modules in the VC domain. Setting

     the parameter adjusts the capacity of the hardware to absorb traffic

     bursts without data loss and thereby potentially increases data

     throughput. Valid ratios include 1, 2, 3, & 4, where 1 is the default.

     NOTE: PacketBufferOverallocationRatio is applicable only for

     HP 1/10Gb VC-Enet and HP 1/10Gb-F VC-Enet

     modules and deprecated for all other module types.

  FlowControl : (Optional) "auto" means VC Ethernet modules will

      decide automatically based on port type if flow control should be

      enabled or disabled. "on" means flow control is enabled for both

      RX/TX directions for all ports. "off" means flow control is disabled

      for both RX/TX directions for all ports.

      NOTE: Port-level Flow Control does not apply to ports on which FCoE is

      configured. FCoE utilizes priority-level FlowControl only when

      configured.

Examples:

  - Modify the packet buffer overallocation ratio to 2:

    ->set advanced-networking PacketBufferOverallocationRatio=2

  - Turn off flow control on all ports in the VC domain:

    ->set advanced-networking FlowControl=off

And last input from Mark:

From what I remember the CLI means this.

Auto is how VC is by default. Only the downlinks advertise Tx/Rx flow control. Uplinks/Stacking Links off

On is no change for the downlinks --- they still advertise Tx/Rx. The uplinks depending on the speed --- may be “ON” Tx/Rx (10Gb) or “advertise” Tx/Rx (1Gb)

Off is no Tx/Rx anywhere

************

Comments?