virtual connect sends back arp requests when using linux bridge

[Adrian_P]

Hi,

I have a HP c7000 blade enclosure with couple of BL460c Gen8 blades, and two HP VC Flex-10/10D interconnect modules. In the blades I have OpenStack Pike installed with latest CentOS 7 as host operating system and neutron is configured to use linux bridge to connect VMs (virtual machines) to the external world. The flow is like below:

external switch<-->HP virtual connect<-->[eno1 bridge tap]<-->[eth0 cirrosVM]

The eno1 is the first physical interface CentOS 7 sees in the BL460c blades, and the bridge is used to connect the cirrosVM to the outside network. The tap interface is used by the neutron to bridge toghether the eth0 interface in the cirrosVM and the eno1 interface in the CentOS.

The problem is the following:

When the cirrosVM wants to access any IP address in the external network, an arp request is sent out on the eth0 interface, and enters the linux bridge on the tap interface. In its forwarding table, the bridge assigns the eth0 mac address to the tap interface, and sends the arp request out on the eno1 interface. Now, some device on the left side of the bridge (I am sure the HP virtual connect does that), broadcasts that arp requests back, therefore the same arp request enters back the bridge on the eno1 interface, and the bridge assigns the source mac address of the arp request (which is still the eth0 mac address) to the eno1 port in the forwarding table. This behaivious, with the eth0 mac address wrongly assigned to the eno1 port in the bridge (while it should be assigned to the tap interface), causes all IP communication to stop working for that IP address.

So the question is: why does the HP virtual connect sends back the arp request broadcast? Is there a feature in the HP virtual connect that could disable this behaiviour? Or is this a bug in the HP virtual connect?

In the flow above, the external switch (HP ProCurve 2848) is used as a router as well, therefore the layer 2 broadcast domain is terminated in that switch. I am sure the external switch does not cause this behaiviour, since I have used port monitoring to trace, and I see the arp request arriving, but not going out again.

However, if I use tcpdump and trace on the eno1 interface, I see the arp request going out and comming back in. If I use the bridge monitor command, I can see the bridge assigning the eth0 mac address to the tap interface first (when the arp request goes out), then to the eno1 interface when the arp request comes back in.

Initially instead of the eno1 interface, I have bonded eno1 and eno2 toghether, therefore both Flex-10/10D switch modules where in use. I thought this arp request behaiviour might be caused by some loop in the Virtual Connect, however even now, when only eno1 is in use (therefore no loop), the same issue happens.

The workaround I have found is to convert the bridge to a hub, by setting ageing to 0 (brctl setageing br-name 0). In this way, the bridge floods all packets on all attached bridge interfaces (eno1 and tap interface), and everything starts working. However, this is not a long term solution.

I am wondering if anybody else has seen this issue, and if there is any solution for it.

Many thanks in advance,

Adrian

[Prabu_bhat]

Hello,

Does the set up has some Loadblancers?

If so, check if the details on Link below helps:

https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c02684783

When 2 machines on the same VLAN want to talk over IP, and have not talked to each other before (or recently), the sending machine will first check to see if the other machine is local (same subnet).

If it is, a broadcast will be sent on the VLAN asking for the owner of IP Y to please respond.

*IF* the owner of the IP sees the broadcast, it will respond and cache the MAC of the sender.

The server, upon receiving the response, will cache the MAC of the destination.

This IP to MAC table is known as ARP.

Each machine maintains an ARP cache to reduce the number of times the above broadcast process has to occur.

Now, in your customer situation, when Sender sends the broadcast to find destination, AND VC Private is enabled, the VC module will not allow other blades to receive that broadcast.

BUT, the broadcast WILL be sent out the uplink.

So when VC Private is enabled, the VC module will

• Stop ARP broadcast from one blade to another blade
• But allow any blade send ARP broadcast to uplink ports
• Also allow any blade receiver ARP broadcast from uplink ports

[Adrian_P]

Thanks for the reply.

There are no load balancers in this setup.

My problem is that my own arp request, sent by the centos VM through the linux bridge out on the blade physical interface, returns back on the same blade physical interface, thus confusing the linux bridge. Its okay if this arp request goes to other blades, however I do not want it to come back to the same blade.

What I would like is to somehow disable this behaiviour. The VC should not return back on the same interface an arp broadcast.

Regards,

Adrian

[Adrian_P · Accepted solution]

Maybe this helps others as well:

After the discussion with a HP support engineer, I was directed to this link https://support.hpe.com/hpsc/doc/public/display?docId=mmr_kc-0122877 which describes exactly the problem I have. I do not know why I did not reach that link with google...

Anyway, I have disabled the SRIOV for my network adapters in the blade servers, as described in that link above, and suddently the issue went away. I did not had to upgrade the network adapter firmware... I suppose I would need to update the firmware if I would ever need the SRIOV in the future.