SDN HP3800 switch broadcast fault

[Dave-B]

I have HP3800 switches configured as OpenLFow 1.3 SDN switches (firmware version KA.15.15.0006).

In OpenFlow SDN mode, If the switches reveive mutlicast packets, they broadcast the packets as per a traditional switch, without the SDN controller adding any rules in the flow tables.

If I direct the mutlciast stream to mulitple ports on the same switch (using flow rules) then remove a port, packets are broadcast on other ports for approx 0.1ms, even when there is a lower priority drop packet flow rule.

In SDN OpenFlow mode, this shoudl not happen.

Has this been seen before? Has it been fixed in newer firmware versions?

[Scott_Koster]

First off, that switch is running a very old firmware version (~ 2 years).  Please try updating the 3800 to a newer firmware.  KA.16.01.0007 would be a good target, download link below:

https://h10145.www1.hpe.com/downloads/SoftwareReleases.aspx?ProductNumber=J9573A

What SDN Controller are you using?  Is it running in Hybrid Mode or Full Openflow Mode?  Is the Switch to Controller communication using OpenFlow version 1.0 or 1.3?

Hybrid - Packets that do not match OpenFlow pipeline are forwarded normally through the swtich. This is using the OpenFlow forward normal rule.

Full OpenFlow - Packets that do not match and OpenFlow flow in the OpenFlow pipeline are forward to controller for a decision if the flow is allowed / disallowed. 

[Dave-B]

I'm using HP VAN 2.0 in full OpenFlow mode. The switches are setup for OF 1.3, but the not using any of the OF 1.3 functionality. To implement multciast I am just adding or removing portd from the actio output port list.

I'm not using hybrid mode / any forward normal commands.

[ShaunWackerly]

Hi Dave,

In this situation, it would help if you were able to post the following :

1. Before you removed the port (screenshot from VAN GUI, or "show openflow instance X flows" on 3800)
2. Example JSON of the flow you're pushing to remove the port (or the java flowmod captured in OpenFlow Trace)
3. After you removed the port (screenshot from VAN GUI, or "show openflow instance X flows" on 3800)

With this additional data, I think we'll be able to narrow in on what's going wrong. From your description above I'm confused as to whether you're saying data goes out the original port that was removed, or whether it goes out ports which were never in the multicast group to begin with.

[Dave-B]

I can send the following as .pdf if needed>

UDP Broadcast Using HP3800 Switch in OpenFlow Mode
D Butler
19 July 2016

1 Introduction
This work is part of investigation into IP Production on a SDN using Source Specific Multicast for live TV production.

2 Test Setup
The HP3800 switches with version KA.15.15.0006 firmware, configured in OF 1.3 mode, are connected in 4 switch mesh, as shown in Figure 2.1. A second, traditional, network using a separate IP range is employed to connect the SDN controller and SDN switches. The traditional network is also employed for SSH access to the source/ destination servers.

Figure 2.1: Test Network Setup

A source on one server streams UDP test packets or SSM video, using IPERF or VLC, which are received by one or more destination servers.
e.g. Using IPERF on the source server 10.0.0.100, to generate a 800Mb/s UDP stream:
$ iperf –u –p 5004 –c 232.13.1 –b 800M –t 86400
------------------------------------------------------------------
Client connecting to 232.1.3.1, UDP port 5004
Sending 1470 byte datagrams
Sending multicast TTL to 1
UDP buffer size: 208 KByte (default)
------------------------------------------------------------------
[ 3] local 10.0.0.100 port 3745 connected with 232.1.3.1 port 5004
[ID] Interval Transfer Bandwidth
[ 3] 0.0–24.8 sec 2.32 GBytes 803 Mbits/sec
[ 3] Sent 1692709 datagrams
$
e.g. Using VLC to stream a video file:
$ cvlc -vvv big.mov --sout "#rtp{proto=udp,mux=ts,name=big,sdp=sap,dst=232.1.6.1,port=5004}" --sout-keep --ttl 5 --loop

Destinations join and leave SSM streams using IGMPv3 messages which are processed by the 10.0.0.110 server, which in programmes flows using the REST interface on the HP VAN 2.0 SDN controller.

At the destinations, a simple multicast client counts dropped / repeated / out-of-order packets using the sequence number in the IPERF UDP packet payload:
$ ./ssmmeter 10.0.0140 10.0.0.100 232.1.3.1

Video streams are received and played using VLC:
$ vlc rtp://10.0.0.100@232.1.3.1:5004

A network tap and Wireshark are employed for investigating anomalous behaviour.

3 Anomalous Switch Behaviour
The HP3800 switches with version KA.15.15.0006 firmware have shown anomalous behaviour when modifying flows. The specific test setup is as show in Figure 3.1.

Figure 3.1: Test Setup Showing Anomalous Network Behaviour

If servers Dest1 (on SW2) and Dest2 (on SW4) have joined a multicast stream from Source1 (on SW1), when Dest2 leaves the stream, there are no repeat packets received by Dest1.
However, if servers Dest1, Dest2 (on SW4) and Dest3 (SW4) have all joined a multicast stream from Source1 (on SW1), when only Dest2 leaves the stream, repeat packets are received by Dest1.

The equivalent curl commands for the switches are to receive the stream at Dest1, 2 and 3 are:
SW1> $ curl --header "X-Auth-Token:$AUTH_TOKEN" -H "Content-Type:application/json" -ksS --url "https://192.168.40.200:8443/sdn/v2.0/of/datapaths/00:02:c8:cb:b8:3e:fe:40/flows" -d "{\"flow\": {\"priority\": 20000,\"table_id\":100,\"idle_timeout\": 60000,\"match\": [{\"ipv4_src\":\"10.0.0.100\"},{\"ipv4_dst\":\"224.1.3.1\"},{\"eth_type\": \"ipv4\"}],\"instructions\": [{\"apply_actions\": [{\"output\": 25}, {\"output\": 26}]}]}}" --request POST
SW2> $ curl --header "X-Auth-Token:$AUTH_TOKEN" -H "Content-Type:application/json" -ksS --url "https://192.168.40.200:8443/sdn/v2.0/of/datapaths/00:02:6c:3b:e5:62:b2:80/flows" -d "{\"flow\": {\"priority\": 20000,\"table_id\":100,\"idle_timeout\": 60000,\"match\": [{\"ipv4_src\":\"10.0.0.100\"},{\"ipv4_dst\":\"224.1.3.1\"},{\"eth_type\": \"ipv4\"}],\"instructions\": [{\"apply_actions\": [{\"output\": 17}]}]}}" --request POST
SW3> $ curl --header "X-Auth-Token:$AUTH_TOKEN" -H "Content-Type:application/json" -ksS --url "https://192.168.40.200:8443/sdn/v2.0/of/datapaths/00:02:34:64:a9:59:8a:00/flows" -d "{\"flow\": {\"priority\": 20000,\"table_id\":100,\"idle_timeout\": 60000,\"match\": [{\"ipv4_src\":\"10.0.0.100\"},{\"ipv4_dst\":\"224.1.3.1\"},{\"eth_type\": \"ipv4\"}],\"instructions\": [{\"apply_actions\": [{\"output\": 11}, {\"output\": 13}]}]}}" --request POST

Screen shots of the OpenFlow Monitor in the SDN Controller are shown in Figures 3.2, 3.3 and 3.4. The flows for 224.0.0.1 and 242.0.0.2 are for IGMP messages. The lower priority flow for 232.0.0.0/24 is to drop unjointed streams.

Figure 3.2: OF Monitor View for SW1

Figure 3.3: OF Monitor View for SW2

Figure 3.4: OF Monitor View for SW4

When Dest2 leaves the stream, port 13 is removed from the flow in SW4. All other flows are unchanged. The equivalent curl commands for the switches are to receive the stream at Dest1 and 3 are:
SW1> $ curl --header "X-Auth-Token:$AUTH_TOKEN" -H "Content-Type:application/json" -ksS --url "https://192.168.40.200:8443/sdn/v2.0/of/datapaths/00:02:c8:cb:b8:3e:fe:40/flows" -d "{\"flow\": {\"priority\": 20000,\"table_id\":100,\"idle_timeout\": 60000,\"match\": [{\"ipv4_src\":\"10.0.0.100\"},{\"ipv4_dst\":\"224.1.3.1\"},{\"eth_type\": \"ipv4\"}],\"instructions\": [{\"apply_actions\": [{\"output\": 25}, {\"output\": 26}]}]}}" --request POST
SW2> $ curl --header "X-Auth-Token:$AUTH_TOKEN" -H "Content-Type:application/json" -ksS --url "https://192.168.40.200:8443/sdn/v2.0/of/datapaths/00:02:6c:3b:e5:62:b2:80/flows" -d "{\"flow\": {\"priority\": 20000,\"table_id\":100,\"idle_timeout\": 60000,\"match\": [{\"ipv4_src\":\"10.0.0.100\"},{\"ipv4_dst\":\"224.1.3.1\"},{\"eth_type\": \"ipv4\"}],\"instructions\": [{\"apply_actions\": [{\"output\": 17}]}]}}" --request POST
SW3> $ curl --header "X-Auth-Token:$AUTH_TOKEN" -H "Content-Type:application/json" -ksS --url "https://192.168.40.200:8443/sdn/v2.0/of/datapaths/00:02:34:64:a9:59:8a:00/flows" -d "{\"flow\": {\"priority\": 20000,\"table_id\":100,\"idle_timeout\": 60000,\"match\": [{\"ipv4_src\":\"10.0.0.100\"},{\"ipv4_dst\":\"224.1.3.1\"},{\"eth_type\": \"ipv4\"}],\"instructions\": [{\"apply_actions\": [{\"output\": 11}]}]}}" --request POST

The Screen shot of the OpenFlow Monitor for SW4 is shown in Figures 3.5. Others are unchanged.

Figure 3.5: OF Monitor View for SW4 (after port removal).

Using a network tap and Wireshark, as shown in Figure 3.1, repeat packets are detected on the link SW4 to SW2. The Wireshark capture results are shown in Figure 3.6.

Figure 3.6: Wireshark Capture of Repeat Packets

As shown in Figure 3.7, the number of repeated packets increases with stream bit rate (from 10Mb/s to 900Mb/s). The duration of repeated packets is around 0.1ms for all stream rates.

Figure 3.7: Repeated Packets for Live Flow Movement using Individual Priorities

In a traditional Layer 2 network switch, multicast packets are broadcasts on all ports. In a SDN switch, packets should not be broadcast without a flow table rule to implement this. Packets should be forwarded to the SDN controller, which should then programme flows.
For a period of 0.1ms, it appears that the SDN switch is directing packets as a traditional switch.
When 1 of the 2 destinations on SW4 (Dest2 or Dest3) leaves the multicast stream, a burst of packets are broadcast on the SW4 links to SW2 and SW3. These are detected using a network tap and Wireshark.
The flow table rule for the multicast stream in SW2 direct these packets to Dest1, so the destination receives a burst of repeated packets.
Since the flow table rule for the multicast stream does not specify an incoming switch source port, incoming packets from port 7 and port 5 are delivered to Dest1 via port 17.
The rule used to drop packets from un-joined multicast streams is not applied, as it is at a lower priority.

Note:

The anomalous broadcast behaviour in the switch occurs when a flow rule is modified to remove a port from one of multiple output ports for an active stream.
If no flow rules are programmed in any switch, UDP packets are broadcast.
If the IGMP Proxy Application in server 10.0.0.110 is not running and there are no flows programmed or visible in the SDN VAN controller OF Monitor, then UDP packets are broadcast.
When packets are stream using IPERF from server 10.0.0.100:
1. Broadcast packets are detected using Wireshark and the network tap.
2. No flow rules are visible in the SDN VAN controller OF Monitor (the controller does not appear to add any rules of its own).
3. If ‘drop’ packets rule is programmed in SW4, using a curl command, the broadcasts detected by Wireshark stop.

4 SW4 Configuration and Flows

HP-3800-48G-4SFPP-4# show config
Startup configuration: 7

; J9576A Configuration Editor; Created on release #KA.15.15.0006
; Ver #05:19.ff.ff.3f.ef:cc

hostname "HP-3800-48G-4SFPP-4"
module 1 type j9576y
module 2 type j9576x
interface 49
flow-control
exit
snmp-server community "public" unrestricted
openflow
controller-id 1 ip 192.168.40.200 controller-interface oobm
instance "sdnm"
member vlan 2
controller-id 1
version 1.3 only
mode passive
enable
exit
enable
exit
oobm
ip address 192.168.40.74 255.255.255.0
exit
vlan 1
name "DEFAULT_VLAN"
no untagged 3-52
untagged 1-2
ip address 192.168.40.34 255.255.255.0
exit
vlan 2
name "VLAN2"
untagged 3-52
no ip address
exit
no autorun
no dhcp config-file-update
no dhcp image-file-update

HP-3800-48G-4SFPP-4# show openflow instance sdnm flows
OpenFlow Flow Table

Flow 1
Match
Incoming Port : Any Ethernet Type : Any
Source MAC : Any Destination MAC : Any
VLAN ID : Any VLAN priority : Any
Source Protocol Address : Any
Target Protocol Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Attributes
Priority : 0 Duration : 5270018 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : 0 Packet Count : NA
Flow Table ID : 0 Controller ID : NA
Activity Count: NA Cookie : 0x0
Hardware Index : NA
Instructions
Goto Table ID : 100

Flow 2
Match
Incoming Port : Any Ethernet Type : IP
Source MAC : Any Destination MAC : Any
VLAN ID : Any VLAN priority : Any
Source Protocol Address : Any
Target Protocol Address : 224.0.0.1/32
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Attributes
Priority : 20000 Duration : 6442 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 21
Flow Table ID : 100 Controller ID : 1
Activity Count: NA Cookie : 0x0
Hardware Index : 17
Instructions
Apply Actions
Output : 11
Output : 13

Flow 3
Match
Incoming Port : Any Ethernet Type : IP
Source MAC : Any Destination MAC : Any
VLAN ID : Any VLAN priority : Any
Source Protocol Address : Any
Target Protocol Address : 224.0.0.22/32
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Attributes
Priority : 20000 Duration : 6440 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 10
Flow Table ID : 100 Controller ID : 1
Activity Count: NA Cookie : 0x0
Hardware Index : 18
Instructions
Apply Actions
Output : 5

Flow 4
Match
Incoming Port : Any Ethernet Type : IP
Source MAC : Any Destination MAC : Any
VLAN ID : Any VLAN priority : Any
Source Protocol Address : 10.0.0.100/32
Target Protocol Address : 232.1.3.1/32
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Attributes
Priority : 20000 Duration : 3234 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 26891052
Flow Table ID : 100 Controller ID : 1
Activity Count: NA Cookie : 0x0
Hardware Index : 19
Instructions
Apply Actions
Output : 11

Flow 5
Match
Incoming Port : Any Ethernet Type : IP
Source MAC : Any Destination MAC : Any
VLAN ID : Any VLAN priority : Any
Source Protocol Address : Any
Target Protocol Address : 232.0.0.0/8
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Attributes
Priority : 19999 Duration : 6444 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 8
Flow Table ID : 100 Controller ID : 1
Activity Count: NA Cookie : 0x0
Hardware Index : 0
Instructions
Drop

Flow 6
Match
Incoming Port : Any Ethernet Type : Any
Source MAC : Any Destination MAC : Any
VLAN ID : Any VLAN priority : Any
Source Protocol Address : Any
Target Protocol Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Attributes
Priority : 0 Duration : 4915674 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : NA Packet Count : 23613373
Flow Table ID : 100 Controller ID : 1
Activity Count: NA Cookie : 0x0
Hardware Index : NA
Instructions
Goto Table ID : 200

Flow 7
Match
Incoming Port : Any Ethernet Type : Any
Source MAC : Any Destination MAC : Any
VLAN ID : Any VLAN priority : Any
Source Protocol Address : Any
Target Protocol Address : Any
IP Protocol : Any
IP ECN : Any IP DSCP : Any
Source Port : Any Destination Port : Any
Attributes
Priority : 0 Duration : 4915673 seconds
Hard Timeout : 0 seconds Idle Timeout : 0 seconds
Byte Count : 31997721733 Packet Count : 21873745
Flow Table ID : 200 Controller ID : 1
Activity Count: NA Cookie : 0x0
Hardware Index : NA
Instructions
Apply Actions
Controller Port

HP-3800-48G-4SFPP-4#

[Dave-B]

I updated the switch firmwae to KA.16.01.007 and updated the controller to SDN VAN 2.7.16 and it still does the same:

1)    UDB Broadcasts when no flow table rules.

2)    0.1ms of UDP Broadcasts when removing port from flow rule.

BTW: The license transfer website at https://h10145.www1.hpe.com/license/TransferSearch.aspx does not work!

Dave.

[Scott_Koster]

Hi Dave,

Thank you for the additional information and further detail. I will check with the team on both of these items.

Cheers,

-Scott

[Scott_Koster]

Hi Dave,

There are two options for your scenario to work without the packet leaking you witnessed. First option is to use the recently released switches based on our 3rd generation SDN capable ASIC; 5400R with v3 modules, 3810M, 2930F as there is an architecture change around this specific scenario that changes the behavior. Second option is to use OpenFlow groups instead of multiple output ports in your rules, the output port changes will be made by increasing, decreasing and changing the group buckets. This will work on the current 3800 and the newer switches mentioned above.

More details on our Group table implementation is available in the OpenFlow Admin Guide (16.02 manual - http://h20566.www2.hpe.com/hpsc/doc/public/display?docLocale=en_US&docId=emr_na-c05161702 – Section 4)

Below are a few examples on VAN to create/modify Groups.

Best Regards,
-Scott

DPID in the examples below is: 00:14:88:51:fb:35:1a:40, Controller IP in the examples is 10.15.155.61

Create Group with two buckets, each with one port:

POST: https://10.15.155.61:8443/sdn/v2.0/of/datapaths/00:14:88:51:fb:35:1a:40/groups

{
"version": "1.3.0",
"group": {
"id": 1,
"type": "all",
"command": "add",
"buckets": [
{
"weight": 0,
"watch_group": 4294967295,
"watch_port": "ANY",
"actions": [
{"output": 14}
]
},
{
"weight": 0,
"watch_group": 4294967295,
"watch_port": "ANY",
"actions": [
{"output": 15}
]
}
]
}
}

Group Mod, Change buckets and ports assigned to buckets.

PUT: https://10.15.155.61:8443/sdn/v2.0/of/datapaths/00:14:88:51:fb:35:1a:40/groups/1

{
"version": "1.3.0",
"group": {
"id": 1,
"type": "all",
"buckets": [
{
"weight": 0,
"watch_group": 4294967295,
"watch_port": "ANY",
"actions": [
{
"output": 16
}
]
},
{
"weight": 0,
"watch_group": 4294967295,
"watch_port": "ANY",
"actions": [
{
"output": 17
}
]
},
{
"weight": 0,
"watch_group": 4294967295,
"watch_port": "ANY",
"actions": [
{
"output": 14
}
]
}
]
}
}

Flow Mod:

Creat a flow matching a destination IP of 231.0.4.5 and output to group 1 defined above.

POST: https://10.15.155.61:8443/sdn/v2.0/of/datapaths/00:14:88:51:fb:35:1a:40/flows

{
"flow": {
"priority": 36000,
"idle_timeout": 0,
"hard_timeout": 0,
"match": [
{
"eth_type": "ipv4"
},
{
"ipv4_dst": "231.0.4.5",
"mask": "255.255.255.255"
}
],
"instructions": [
{
"apply_actions": [
{"group": "1"}
]
}
]
}
}

[Dave-B]

Scott,

Thansk for the reply.  

I've had a go at implementing the group appraoch in my code.

Is there a limit to the number of groups and is there anything special about the watch_group value?  e.g.  "watch_group": 4294967295?

I have mutliple groups and group flows in each switch. When I try to modify the ports in group 3, I get an error, e.g.

curl -H -ksS X-Auth-Token:$AUTH_TOKEN https://192.168.40.180:8443/sdn/v2.0/of/datapaths/00:02:34:64:a9:59:8a:00/groups/3 -d {"version": "1.3.0", "group": {"id": 3,"type": "all","command": "modify","buckets": [{"weight": 0,"watch_group": 4294967295,"watch_port": "ANY","actions": [{"output": 11}]},{"weight": 0,"watch_group": 4294967295,"watch_port": "ANY","actions": [{"output": 13}]}]}} --request PUT
{"error":"java.lang.IllegalArgumentException","message":"{ofm:[V_1_3,ERROR,76,156230],GROUP_MOD_FAILED/EPERM,#dataBytes=64,OFM-cause:[V_1_3,GROUP_MOD,80,156230]}"}

I'm using the same code to modify groups 1 and 2, without problem.

I'm currently on switch S/W version KA.16.01.0007. Is this an issue?

Dave.

[ShaunWackerly]

Hi Dave,

I tried the example you posted, and got the same error under certain conditions. I don't think that the issue is caused by the group ID (3) or the watch_group value (4294967295).

After creating a group using a POST request, I was able to successfully modify the group using a PUT request as you have below. However, I also noticed that if I re-issued the same PUT request then I'd get the error you're getting below. If I changed all ports included in the group, then the PUT request was accepted.

I will need to confirm with our firmware team, but I believe the reason you're getting this error is that either port 11 or port 13 are already included in a group. You'd get this error even if 11 or 13 were already included in group 3 (the one you're trying to modify). I realize that if this is the case, it's inconvenient.

Could you please run the following command before issuing the PUT request. This will show the current state of configured groups, and which ports have already been assigned to a group:

curl -H -ksS X-Auth-Token:$AUTH_TOKEN https://192.168.40.180:8443/sdn/v2.0/of/datapaths/00:02:34:64:a9:59:8a:00/groups --request GET

If this is the behavior you're seeing, you could work around it by issuing one group-mod which puts an unused port into the group, then a second request which adds all desired ports to the group. This workaround would temporarily cause drops, but may prevent you from getting this error. If I receive any more information from our firmware team, I'll post it here.

Shaun