OF rule won't output on port unless arp table populated

[phpHavok]

Hi All,

Consider the following topology where h1 and h2 are end hosts, s1 and s2 are OF-enabled HPE L2 switches, and r1 is an OF-enabled HPE L3 switch/router.

h1 --- s1 ------- (port 1) - r1 - (port 2) ------- s2 --- h2

On r1 port 1, we have VLAN 2 with IP address 172.23.7.161/29. On r1 port 2, we have VLAN 3 with IP address 172.23.7.169/29. Routing is enabled.

We have found that traffic traveling from h1 to h2 (or vice versa) over the NORMAL rule will flow correctly. However, we added an OpenFlow rule with higher priority than NORMAL in r1. It matches a very specific TCP flow from h1 to h2. The actions for the rule are to rewrite the MAC addresses correctly, change the VLAN, and output on port 1 or port 2 (we have two flows to create a bidirectional path). We send test traffic from h1 to h2 that matches the flow. We observe two possible outcomes.

Outcome 1
Everything functions as normal. The MACs and VLAN are rewritten, and the packets are forwarded correctly.

Outcome 2
Packets match the rule in r1, but no packets leave the outgoing port. We verified this by monitoring the outgoing port for traffic. Thus, the packet is being dropped by the HPE switch *after* matching an OpenFlow rule instruction the packet to output on a specific port.

We discovered that outcome 2 happens whenever r1's ARP table loses its entries for h1 or h2. Our question is why? We understand that for NORMAL to work, the router must ARP for h1 or h2 in order to encapsulate the packet in a new L2 frame. However, at the OpenFlow layer, we are rewriting MAC addresses ourselves (i.e., doing what the router would do if it had static ARP entries). Shouldn't this bypass the router's default processing? Why does the router need to know about neighbors for a very specific OpenFlow rule to work?

Thanks!

[Abhay_B]

Hello phpHavok,

Are there any other flows on the OpenFlow switches apart from the ones you have mentioned. Will it be possible to share all the OpenFlow rules on all the 3 switches (h1, h2 and r1).

My first guess would be that there are no OpenFlow flows on the switch that are handling the ARP packets being generated post the ARP timeout. If there aren't any, can you add a rule on all 3 switches that would match ARP packets and forward them NORMAL?

Thanks!

Abhay

[phpHavok]

Hey Abhay,

In reality, we have several switches between the end hosts and r1, but we have only noticed problems occuring at r1. All of the switches (including ones not mentioned) and r1 have the default flows which are installed when hybrid mode is enabled, including the com.hp.sdn.arp.copy flow which copies ARP traffic (controller + normal).

In one test, we did the following:

1. Clear arp table on r1 with "clear arp".
2. Install a flow matching a TCP Iperf connection at r1 with an action to output on the port leading to h2.
3. Start iperf from h1 to h2. Observe TCP SYN packets matching the flow we installed (counters increasing), but observe no SYN packet leaving the port the OpenFlow rule sets as the output port.
4. While iperf is hanging, we open another terminal and ping between h1 and h2. The iperf immediately starts working.
5. We observe with "show arp" that h1 and h2 are now known to r1.
6. We again clear the arp table with "clear arp" while the iperf is going.
7. The iperf immediately stops working.

Given this test, we seem to have isolated the issue to specifically the ARP table of r1. I really don't understand why this matters. Why does it matter what is in r1's ARP table (or even the controller's ARP table for that matter), when we are rewriting MACs manually at the OpenFlow layer?

Thanks!

[Abhay_B]

Hello phpHavok,

Thank you for sharing the additional info.

I tested the use case with a similar topology and observed the same behavior as reported by you.

In traditional networking without OpenFlow, when packets to a routable IP destination arrives, the packets will be bufferred/dropped until the next hop to that IP destination is resolved. The ARP resolution is triggered by the switch itself.

When packets come to OpenFlow, it does not undo the action taken by a different control/data plane protocol to drop a packet. It would just change the output port for the packet to the specified port in the flow after executing any packet modification actions.

The behavior you currently see is as per the current design of the system.

Let me know if I can help out with anything else.

Thanks!

Abhay

[phpHavok]

Abhay,

Thank you for your help and explanation regarding this matter.

Lastly, I would like to ask your opinion of something. To circumvent this problem, we currently have configured h1 and h2 (as well as other hosts I did not mention--this example is one piece of a larger topology) to ping their default gateway every so often via a cron job. We don't have any OpenFlow rules above the NORMAL fallback rule intercepting ping, so the pings always fall through to NORMAL. This "solves" the problem of keeping the switch aware of each host's existence, but I'm curious if you have a better idea for doing this? It's unfortunate that machines which don't talk enough on the network will always experience this problem at OF gateways once the gateway's knowledge of the machines times out in the ARP table.

Thanks!

[Abhay_B]

Hello phpHavok,

At this point, I don't have a better solution for workarounding this limitation. Ensuring the ARPs to all the hosts are resolved at any point is the only way forward.

If you have a VAN based application sitting on top of the controller, I would suggest exploring the idea of sending a periodic "Packet-Out" message to the OpenFlow switch acting as a router where the payload of the packet-out message is an ARP packet for a host already discovered and the action set to OUTPUT=OFPP_NORMAL. This would trigger an ARP response from the host and the ARP table entry for the host on the router should stay intact. This assumes the host is discovered atleast once by the controller.

This is just a theory that I feel is worthy enough of a shot since if it works, the control is with the VAN application and there is no action required on each individual hosts.

Thanks!

Abhay