RDMA / PFC / DCBX on 5700

[Patrik_Hansson]

Hi

Trying to configure QoS, PFC and DCBX for use with RoCE on a 5700.

Can't find an example anyware. Anybody done it and would like to share ?

Internet is full of examples for cisco but none for HP.

The HP FlexFabric 5700 Switch Series  Layer 2—LAN Switching Configuration Guide have one but that is for Fc.

[Patrik_Hansson · Accepted solution]

With some help från HPE L2:

#
 qos map-table dot1p-lp
 import 0 export 0
 import 2 export 0
 import 3 export 0
 import 4 export 0
 import 6 export 0
 import 7 export 0
 #
 traffic classifier DCBX_class operator or
 if-match acl 3001
 if-match acl 3002
 #
 traffic behavior DCBX
 remark dot1p 5
 #
 qos policy DCBX_policy
 classifier DCBX_class behavior DCBX mode dcbx
 #
 interface Ten-GigabitEthernet1/0/1
 port link-type trunk
 port trunk permit vlan 1 10 to 11
 priority-flow-control auto
 priority-flow-control no-drop dot1p 5
 jumboframe enable 5000
 lldp tlv-enable dot1-tlv dcbx
 qos trust dot1p
 qos apply policy DCBX_policy inbound
 #
acl number 3001
 rule 0 permit tcp destination-port eq 445 counting
#
acl number 3002
 rule 0 permit tcp source-port eq 445
#

[Aurock]

Would you, or anyone else, mind explaining what a few pieces of your solution do? I'm a server admin, with some basic understanding of networking terms & concepts, but only very limited experience configuring switches, and none at this level. I'll comment on sections or even line by line below. Hopefully someone can educate me on the steps I don't understand, and maybe even correct anything that I only think I understand...

#
qos map-table dot1p-lp
import 0 export 0
import 2 export 0
import 3 export 0
import 4 export 0
import 6 export 0
import 7 export 0

The above is is remapping qos priorities.. It looks like it's saying anything tagged with a priority other than 2 or 5 will be treated as (or re-tagged to?) lowest priority. Is that right? If so, why is that necessary, as opposed to respecting the original priority?

#
traffic classifier DCBX_class operator or
if-match acl 3001
if-match acl 3002

This defines a traffic class "DCBX_Class" and includes any traffic matching either acl 3001 or 3002 in the class.

#
traffic behavior DCBX
remark dot1p 5
#
qos policy DCBX_policy
classifier DCBX_class behavior DCBX mode dcbx

The first part above defines a behavior "DCBX" and sets that behavior to remark traffic with qos priority 5.

The second part devices a new qos policy "DCBX_Policy", which affects traffic included in DCBX_class by applying behavior DCBX. 

#
interface Ten-GigabitEthernet1/0/1
port link-type trunk
port trunk permit vlan 1 10 to 11
priority-flow-control auto
priority-flow-control no-drop dot1p 5
jumboframe enable 5000
lldp tlv-enable dot1-tlv dcbx
qos trust dot1p
qos apply policy DCBX_policy inbound

This is configuring the switch port as a trunk port, setting the vlans allowed on the port, enabling PFC and setting it to treat dot1p priority 5 traffic as 'no-drop', cutting the jumboframe size from 10000 to 5000, setting the port to respect dot1p tags, and applying the DCBX_policy to all inbound traffic on this port.

Why did you cut the jumbo frame size in half?  

I assume that the above section needs to be repeated for each other port on the switch intended for similar use?

Is there anything in my comments above that I significantly misunderstood or overlooked?

Thanks,
Aurock

[bakerbas]

"The above is is remapping qos priorities.. It looks like it's saying anything tagged with a priority other than 2 or 5 will be treated as (or re-tagged to?) lowest priority. Is that right? If so, why is that necessary, as opposed to respecting the original priority?"

Modifying the qos map-table dot1p-lp ensures that certain traffic gets assigned to 1 queue, while other, non critical traffic, gets assigned to another queue. This can be leveraged later in the config by applied ETS configurations using WRR on the physical interface (not shown in your examples). With this we can tell the physical interface that a certain percentage of the bandwidth on the interface can be used for traffic in one queue, while the remaining bandwidth on that interface can be used for traffic from another queue. You can use the 'display qos map-table dot1p-lp' command (use ? as the command may be slightly different). With this you can be sure what 802.1p values are being mapped to which of the 8 queues.

The QoS policies use ACLs and classifiers to select traffic - the behaviour is used to modify the selected traffic, and then apply to interface enables the policy. The policy is there to ensure the traffic you want as critical no drop traffic gets marked with a specific 802.1p value. This 802.1p value is used to map traffic to a specific queue (described above), and to also use PFC to tell the switch to not drop that traffic.

"Why did you cut the jumbo frame size in half?"

There could be a number of reasons why they did this. For jumbo frames you want to ensure that each device in the path is set to support the correct and same jumbo frame size. It is possible that a device in the path only support x size jumbo frames. There are also performance issues that need to be considered. Sometimes iSCSI jumbo frames may improve throughput but at the expense of I/Os per second and I/O completion time, because larger frames are being transferred. Please refer to the HPE iSCSI Best Practices doc which may help a bit more on this.

http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=emr_na-c04873990

"I assume that the above section needs to be repeated for each other port on the switch intended for similar use?"

The configurations need to be applied to any interface in which you want certain traffic to be treated as no drop critical traffic. If the link is a converged link with both critical and non critical traffic then you should also add ETS to the interface to ensure that non-critical traffic does not drown out and pause to death the critical traffic.

I hope that helps!!

[Aurock]

It does, somewhat.  Thanks!

[sundu]

I stumbled upon this website after lot of googling. Kindly help me understand DCBX/RDMA/PFC and configure the below requirement. 

1) Traffic classes : 2 ( TC0 and TC1)

2) TC0 should have priorities configured to 2 to 7

3) TC1 should have priorities configured to 0 and 1

4) Assign TC0, bandwidth percentage of 80% and TC1 with bandwidth percentage of 20%

5) TC0 should be of strict priority whild TC1 should be WRR.

Kindly help.