FlowListener and MessageListener Not Working?

[phpHavok]

I am wanting to intercept all FLOW_MODs as they are being pushed to switches. The purpose is to keep an internal cache of all flows for quick queries without having to query the switches themselves. I am using controller version 2.6.11. I have tried doing this two ways: with FlowListener, which seems like the right answer, and with MessageListener using a FLOW_MOD mask. For debugging, I also added the FLOW_REMOVED mask to my MessageListener. With Open vSwitch, I can manually delete a flow from the switch, which sends the flow removed message to the controller. I am able to intercept this message. However, whenever I add or delete a flow via VAN's REST API, no message is caught. Is this intended behavior? I would expect the REST API to still be generating these types of events whenever it adds and deletes flows from switches.

[ShaunWackerly · Accepted solution]

Based on my reading of the code (implementation), I believe you're describing the intended behavior. The notifications you'll receive by registering a FlowListener are flow-related messages received by the controller. These messages are summarized by type in FlowEventType:

• FLOW_MOD_PUSHED (the switch responded with a clean barrier reply)
• FLOW_MOD_PUSH_FAILED (the switch responded with an error prior to the barrier reply)
• FLOW_REMOVED (the switch notified the controller that a marked flow was removed)

Likewise with the MessageListener, you'll receive notifications about the following events (taken from the MessageListener javadoc):

• OpenflowEventType.MESSAGE_RX (an OpenFlow message was received by the controller)
• OpenflowEventType.DROPPED_EVENTS_CHECKPOINT (an unspecified number of Openflow events were dropped, if your listener isn't responding fast enough)

With the current controller APIs, the only way you could keep a cache of all flows on the switch would be to register a FlowListener and then guarantee that all flowmods being pushed are confirmed via a barrier reply. This will ensure that you'll either get a FLOW_MOD_PUSHED or FLOW_MOD_PUSH_FAILED notification for each flow. Assuming you have 100% control over the flows being pushed by all apps running on the controller (a BIG assumption), the way to guarantee that all flowmods are confirmed is by using:

1. The REST API, which always confirms flowmods it sends
2. ControllerService.sendConfirmedFlowMod(). If you use ControllerService.sendFlowMod(), it will not be confirmed so your FlowListener won't be notified.

If you're not able to guarantee that the controller always confirms all flowmods, then you may have to resort to some periodic polling solution where you store the latest polled set of flows in a cache that provides the quick lookup you need.

What you're wanting to do isn't unreasonable and is a good efficiency improvement to the controller as a whole. Hopefully we can get flow caching into a future controller release.

[ShaunWackerly]

If I could ask a clarifying follow-up question: could you give more detail as to why a cache of flows in the controller is something you feel is necessary? As acknowledged above, it is certainly a performance improvement for the solution. I'm curious about the reason why you'd like it. Here are some potential reasons I can think of:

• Your application wants to optimize control-plane bandwidth and switch CPU, so it doesn't want to push a flowmod if the flow already exists on the switch.
• Your application is pushing a flowmod with no table ID specified, and the flowmod gets adjusted by the VAN device drivers. In that case, you're wanting to see how the flow was adjusted when it was pushed.
• Your application is frequently executing queries against the flow table (to report statistics or predict how packets would get forwarded) and wants to avoid a control-plane request each time the application executes a query.
• ...

[phpHavok]

First, thank you for your replies and for looking into the problem.

Your third point is closest to our reasoning. One of our applications needs to be able to take a flow specification (e.g., a list of match fields and their values) and return a list of switches (and switch information like attached ports) that have a flow installed matching that specification. This query will be ran quite often, so we don't want to be querying every single switch on the topology.

Having VAN keep an internal cache by default would be a great feature, but I would like to be able to listen for all OpenFlow messages that are happening on the controller (regardless of which application triggered the message). For example, I would like to intercept FLOW_MODs as they are being pushed to switches. Is there any reason in particular why VAN doesn't support this?

[ShaunWackerly]

Thanks for the follow-up explanation about your use case, that's helpful information to receive.

In order to allow listeners to receive notifications about FLOW_MOD messages which are sent to controlled switches, VAN would be allowing applications to snoop other applications' interactions. We'd need to think through the security and performance implications before enabling that mechanism. I do see your point that it would be helpful for some uses. We do allow this listening mechanism for how a packet-in was handled (under the OBSERVER role), just not for FLOW_MOD messages.

When you say "intercept FLOW_MODs", I presume that you're just talking about a read-only notification of the FLOW_MOD that's being sent, and not the read/write ability to deny/modify the FLOW_MOD being sent by another application. 

[phpHavok]

That is correct; I am strictly speaking about read-only notifications. I think it would be a nightmare if applications could interfere with each other's abilities to install flows (or commit other actions). However, given that all applications have the power to query switches for installed flows, I don't personally see any security concerns with allowing applications to sniff flows as they are being installed. That being said, I do understand your all's concern and the need to think about this more thoroughly. It might mitigate some of the performance concerns if the callback method on the listeners was invoked in a separate thread (or threads) so that the flow could finish being installed before all the callback methods finished executing. Thank you for discussing this with me.

[phpHavok]

Is there perhaps a bug in version 2.6.11 with the FLOW_MOD_PUSHED event or the REST API? I have a module where I create an instance of a subclass of FlowListener which, upon seeing FLOW_MOD_PUSHED, sends a message to the log. However, when I add a flow over the REST API, my listener is not notified. In an earlier message on this thread, it was mentioned that the REST API always confirms its FLOW_MODs, so I should be seeing that, right?

[ShaunWackerly]

We'll need to look into this last point you brought up. The REST API should always send a barrier request, so that means that when the barrier reply is received your FlowListener should get a notification of type FLOW_MOD_PUSHED. It shouldn't matter whether you use the REST API or whether you use ControllerService.sendConfirmedFlowMod().

Could you either take a packet capture on the control-plane, or trace the OpenFlow conversation using the "OpenFlow Trace" tab in the VAN UI to verify that VAN is receiving a barrier reply from the switch?

Also, does this happen with any flow type or just certain flows?

[phpHavok]

I upgraded to 2.7 to see if the problem was fixed. It still perists.

An OpenFlow Trace reveals that a barrier request/reply pair is being generated. The flow then appears in the flow table (I used Open vSwitch commands to verify this). However, no event of type FLOW_MOD_PUSHED seems to be generated. I haven't found an example that works.

Edit: After consulting the OF specification, this seems to be the intended order of messages. I'm not sure if this is just a pecularity of the log (or a lacking in my understanding), but the barrier request/reply appears in the log *after* the FLOW_MOD message. That is, the order of messages is:

FLOW_MOD
BARRIER_REQUEST
BARRIER_REPLY

as opposed to what I would expect:

BARRIER_REQUEST
FLOW_MOD
BARRIER_REPLY

[ShaunWackerly]

You're correct ... I see the same behavior. I pushed a flow via the REST API, but didn't see my FlowListener get notified with any FlowEvent. I do, however, see my FlowListener getting notified when receiving a FLOW_REMOVED message from the switch.

I'll look into this and see what can be done to get the functionality working.

[ShaunWackerly]

Hi phpHavok,

To follow-up on this topic, I wanted to let you know that with the recently released 2.7.18 version of the HPE VAN SDN Controller, you'll now receive FLOW_MOD_PUSHED events when confirmed flowmods (barrier request/reply) are pushed.

Shaun