VirtualConnect and bandwidth allocation in operation?

[PatrickLong]

My question is regarding the dynamic bandwidth capabilites in VirtualConnect as defined by OneView setttings for "Preferred" and "Maximum" bandwidth. As I understand it there are three components that play some role in how much bandwidth is allocated to a particular network:

The first is in the given Network's configured Preferred bandwidth (default 2.5 Gb/s) and Maximum bandwidth (I'm seeing a default of 50 Gb/s in my OV Composer 5.40) - I'm unsure if this Network Maximum bandwidth this should be changed from default 50Gb/s to either the maximum supported by the installed VirtualConnect (40 GB for VC SE 40Gb F8 Module) or set to the maximum supported by the installed compute module CNA adapter (20 GB for Synergy3820c adapter) or just left at the default 50Gb/s despite none the equipment in my Synergy environment being able to handle 50GB - I only have 20GB Interconnects?

The second is in the Network Set's (to which the above-referenced network is assigned) Preferred bandwidth (default 2.5 Gb/s) setting and Maximum bandwidth setting of 50Gb/s - again I'm unsure if this Network Set Maximum bandwidth this should be changed to either the maximum supported by the installed VirtualConnect 40GB, the maximum supported by the installed compute module CNA adapter 20GB, or left at the default 50Gb/s?

The third is the configuration of the Connections in the Profile/Profile Template option for Requested bandwidth - which appears to pull in the Preferred Bandwidth value setting or either the Network or Network Set you selected for this connection.

I do understand that the *combined* Requested bandwidth allocations of all my Connections assigned to the interfaces on my CNA device (including the 8GB bandwidth reserved for FC) cannot exceed the total bandwidth of that device. So for my 20Gb 3820c adapters, the Requested Bandwidth of my three Connections, each configured for one of my three unique Network Sets cannot exceed 12Gb in aggregate, because the other 8Gb being reserved for FC connections. My real question is: exactly how DYNAMIC is the bandwidth allocation to these connections during normal operation and by what mechanism is it changed on the fly by the Virtualconnect? For example, Let's say I configure a Connection for management Network/Netork Set with Preferred Bandwidth of 1Gb, a Connection for vmotion Network and Network Set also with Preferred Bandwidth of 1Gb, and connection for a number of vm Networks in a single vm Network Set with Preferred Bandwidth of 10Gb. Assuming very little VM network traffic is occurring and I initiate a large-scale vmotion, I understand that since there is a large amount of bandwidth available beyond the 1Gb Preferred Bandwidth guarantee I have set for the connection carrying my vMotion Network/Network Set, that connection will burst or scale up beyond 1Gb/s to use any and all available overhead - which is what I want it to do. But what if during the vmotion operation a large amount of network traffic starts to occur within my vm Networks/Network set? How quickly is the need for reallocation of bandwidth detected, and how quickly does the reallocation occur? Does the VC detect this immediately and change the allocations on the fly to drop the vmotion Connection back to 1 GB and allow the vm Network Connection to grab back it's preferred bandwidth guarantee of 10GB, or does that evaluation only occur on a polling interval (every 30/60/120 secs, etc.) where samples are taken in order to calculate any needed bandwidth adjustments? It's important beacuse if it's based on periodic sampling, there would be some period of time that my vm network is in dire need of additional currently-consumed-by-vmotion bandwidth to get back up to it's guaranteed minimum 10Gb. So by what mechanism / interval / etc. are these Preferred Bandwidth guarantees evaluated and any needed reallocations enforced?

I have looked at the OneView Server profile connections settings Help as well as the Networks > Preferred and Maximum Bandwidth and don't see anything explaining how this occurs during normal operation. 

[ChrisLynch]

These terms can be viewed as a bit confusing.

• Preferred - Only set within a Network or Network Set resource, which can be viewed as the "minimum" or "default" bandwidth to set for a connection.
• Maximum - Only set within a Network or Network Set resource, which is the maximum bandwidth to be allowed for a connection.
• Requested - Only set within a Connection of a Server Profile Template or Server Profile resoure, and defaults to the Preferred value set within the Network or Network Set.

c-Class Virtual Connect implemented a Minimum and Maximum feature, to where if a FlexNIC wasn't transmitting, then other FlexNICs could consume more bandwidth, up to the maximum allowed by the physical port (or Maximum value was set to).  HPE OneView doesn't allow one to set the Maximum within the Server Profile, but the Network/Network Set Maximum is configured when the FlexNIC is instantiated.  Keep in mind that you will not "see" the maxmimum "link speed" value in the OS.  You will  always see the maximum port speed allowed by the adapter.  Also, as you pointed out, you cannot oversubscribe the Connections.  Meaning, that the total requested bandwidth must not exceed that of the parent port.  So if the adpater supports maximum of 20Gb, then all of the requested bandwidth cannot equal more than 20Gb.  You would also have to take into account the FlexHBA, as that is always guaranteed the bandwidth, and will never "flutuate" or "dynamically change".

So how does this work?  Well, this is time slicing within the NIC ASIC.  It is very similar to "shares" within a hypervisor.  The "Requested" bandwidth is the minimum allocated, and a "share" is calculated from the maximum bandwidth of the parent port.  So if all connections are set to 2.5, then all connnections/FlexNICs will get equal time slice to the NIC ASIC.

What do you set the maximum bandwidth to?  You can leave it at 50Gb, even though your adapters only support 20Gb, and are connected to 20Gb capable fabric modules.  OneView defaults to that value because of the fabric modules it supports.

[CSKA]

@ChrisLynch Thank you very much for your explanation and sorry for bringing up a topic which is 2 years old. I have misunderstanding and need some help here. 
In my setup we are using 2 x Virtual Connect SE 100Gb F32 and Synergy 6820C 25/50Gb CNA. The downlink ports between VCs and servers are set to 50 GB/s. Based on the documentation I assume that each server has 2 ports, each capable for 50GB/s to the VCs. We use these VCs for Network and SAN with license upgrade for FC32 SAN uplinks. Now I must specify networks and then server profiles on that way that SAN traffic is not consuming the whole bandwidth which the CNA is supporting and vice versa - network traffic is not limiting the SAN. I'm curious how to do the setup.

My idea is to set preferred bandwidth to 10 GB/s and maximum to 25 GB/s for both networks - Ethernet and SAN. So, at the end at profile settings, I will add Ethernet (network set or tunnel) 2 times for redundancy with M-LAG and aggregated maximum bandwidth for both of them will be 50 GB/s. For network 3 will add SAN_A and for network 4 will add SAN_B and for both will have 10/25GB in total 50GB/s maximum for SAN.

I'm I correct, or I must modify my setup. If it is interesting my uplink setup is following:

VC1: (bay3)  Virtual Connect SE 100Gb F32 : Q1 - 40GB/s Ethernet uplink / Q3 and Q4 - 32GB/s (QSFP28 to SFP28 adapter + FC32SFP)
VC2 (bay6) - the same as above

Thanks!

[PatrickLong]

@ChrisLynch I'm back to this old chestnut, and still finding myself not quite understanding the Preferred bandwidth setting.  I have a new Synergy environment with Virtual Connect SE 100G F32 and all hosts with 6820c CNA adapters.  If I am reading your prior post correctly the FlexHBA portion of this connection should be set to 32GB (even though it is riding a 100G QSFP28 to 4*25G-capable SFP28 connectoion to my 32G FC switch and will never transfer >25G per FC switchport?)  so that would in effect "reserve" 32G of my total 50G CNA bandwidth for FC traffic since you indicated that the HBA portion never fluctuates or dynamically changes speed - is that right?  And if so, is that 32G best set as Prefered bandwidth at the Network level when I create the Type: Fibre Channel Fabric Attach network(s)?  Or at the Profile/ProfileTemplate level when I define the Function: Fibre Channel connections' Requested bandwidth?

Doing so would leave me with only 18G of my 50G CNA throughput on which to set the Preferred values for all other Networks and/or Profile Type: Ethernet connections - and the sum Requested bandwidth of all those ethernet connections cannot exceed the remaining 18G, correct?  All of the Ethernet connections would then be variable between 0G (if no traffic is passing on them) up to a maximum of 18G of remaining throughput not already reserved for FC., and if all ethernet connections are very active then each will operate at their Requested/Preferred values distribution.

Maybe the simplest configuration is to create all Networks for ethernet at the default values of Preferred bandwidth 2.5GB/s and Maximum bandwidth 50Gb/s except for the FC "Networks" to be created at Preferred bandwidth 32Gb/s and Maximum bandwidth 32Gb/s, and then leave all Profile/Profile Template Ethernet connections at default Requested bandwidth of 2.5Gb/s.  Do I have that correct?

Right now I have ALL Networks (including FC) set at Preferred bandwidth 2.5GB/s and Maximum bandwidth 50Gb/s - maybe I need to change that for the FC Networks to match the static, non-dynamic speed of the FC environment at 32G - or should that be 25G since that's the REAL speed at which they will communicate to the 32G FC switch.?

[CSKA]

Hi Patrick,

Meanwhile I had support call with HPE representative, some points are clear on my mind, so let me give you some details.

If I am reading your prior post correctly the FlexHBA portion of this connection should be set to 32GB (even though it is riding a 100G QSFP28 to 4*25G-capable SFP28 connection to my 32G FC switch and will never transfer >25G per FC switchport?)

To get real 32GB FC on the blades, you have to check your LI configuration. If for example your QSFP28 adapter is plugged on Q3 port of VC you have to configure your uplink with the whole Q3 port, not by the 25 sub-sets which are running on subports Q3.1 /Q3.2 / Q3.3 / Q3.4. Put the Q3, instead of sub-ports in your configuration. In my scenario, I'm using QSFP28 to SFP28 (845970-B21) adapter and next FC-SFP28(32GB) P9H30A. On the switch side FC-SFP28 is in place also.  I'm not sure that you can use 25Gig cables on FC Switches.
Another advice which I receive from HPE is to set Minimum and Preferred bandwidth for FC exactly the same. Both on 16GB or 32GB, based on your preference. Basically, I've did this on the Profile template, and do not overwrite it on the profiles.

Please be sure also that when you create your LI, you configure your downlinks to 50GB. 

Regarding the question for bandwidth allocation, if you allocate 32 for FC (which is reservation) you will have maximum 18GB per port for NW traffic (2*18 = 36GB at total per CNA adapter). in this configuration, you will not be allowed to set Requested bandwidth more than the 18GB, even the Max bandwidth of connection object is set for example to 25GB. The bandwidth you configure on Max of each network is what you see at the end on the OS level, but this is shaped to what is in the Requested bandwidth of your profile.
At the end as you must check what is the total bandwidth of all your FC uplinks and calculate proper is it really beneficial to set server profiles FC to 32GB. If you have 4 FC uplinks each on 32GB (separated on 2 VC), you have in total 128GB for all the blades, which means:
128GB  /  12 blades =  10.67 GB / 2 ports - 5.3GB per CNA port.
So to be really benefical to have 32GB FC on server profiles you need to have the same bandwidth as uplinks in sum.  Which I don't believe is possible because some of the Q ports will be used for network uplinks, and finally you will have 2 to 4 Q ports assigned as FC uplinks. Let's say Q1&2 are for NW, Q3,4,5,6 are for FC. The maximum FC bandwidth will be 128GB per VC.