BGP L2VPN EVPN routes not showing up in routing table

[cttripp1]

I'm new to VXLAN, so bear with me. 

I have created the following on two HPE 5940 switches:
#

tunnel global source-address 10.255.10.1 (10.255.11.1 on the 2nd switch)
#

router id 10.255.10.1

(10.255.11.1 on the 2nd switch)
#
ospf 1
import-route direct
graceful-restart ietf
area 0.0.0.0
# WAN subnets
network 10.255.1.0 0.0.0.255
network 10.255.2.0 0.0.0.255
# LAN subnet
network 10.255.255.0 0.0.0.255
#
vlan 3001

name POC-VxLAN

#

vsi vpn-3001
description Distributed IP Gateway POC (vsi)
gateway vsi-interface 3001

arp suppression enable

vxlan 3001
evpn encapsulation vxlan
route-distinguisher auto

vpn-target auto export-extcommunity
vpn-target auto import-extcommunity

#

# This trunks down to VMware
interface Bridge-Aggregation1

port link-type trunk

port trunk permit vlan 3001

port-isolate enable group 1
link-aggregation mode dynamic

#

service-instance 3001

encapsulation s-vid 3001

xconnect vsi vpn-3001

#

interface Vsi-interface3001
description Distributed IP Gateway POC (interface)

ip address 10.130.1.1 255.255.255.0
mac-address 0000-0000-0bb9
distributed-gateway local

#

bgp 65100
peer 10.255.11.1 as-number 65100
peer 10.255.11.1 connect-interface LoopBack0
(10.255.10.1 on the 2nd switch)
#

address-family l2vpn evpn

peer 10.255.11.1 enable
peer 10.255.11.1 router-mac-local

(10.255.10.1 on the 2nd switch)
#

OK, that's a lot.  I have two 5940s configured exactly as above, except the bgp peer info, tunnel source and loopback as shown.  On the bridge, I've trunked the VLAN 3001 down to a VMware host on one of the 5940s, but haven't done the same on the other. On the VMware server, I've created two guest VMs with IPs of 10.130.1.50 and 10.130.1.52, and placed them on a distributed virtual switch using VLAN 3001.

I have no problems pinging the distributed IP gateway address of 10.130.1.1 from either end of the WAN link/tunnel. I've tested taking down each distributed IP gateway address individually, and I can still ping across the WAN link/tunnel.

I can ping the two VMs that are local to their 5940. But I can't ping them across the WAN.

If I 'display bgp l2vpn evpn route-distinguisher 1:3001' on the 5940 that on the other side for the WAN, I get:

BGP local router ID is 10.255.10.1
Status codes: * - valid, > - best, d - dampened, h - history
s - suppressed, S - stale, i - internal, e - external
a - additional-path
Origin: i - IGP, e - EGP, ? - incomplete

Route distinguisher: 1:3001
Total number of routes: 5

Network NextHop MED LocPrf PrefVal Path/Ogn

* >i [2][0][48][0050-56a4-39ec][32][10.130.1.52]/136
10.255.11.1 0 100 0 i
* >i [2][0][48][0050-56a4-4320][32][10.130.1.50]/136
10.255.11.1 0 100 0 i
* > [2][0][48][bcea-fa93-a729][0][0.0.0.0]/104
0.0.0.0 0 100 32768 i
* > [3][0][32][10.255.10.1]/80
0.0.0.0 0 100 32768 i
* >i [3][0][32][10.255.11.1]/80
10.255.11.1 0 100 0 i

So the two virtual hosts' IP addresses (in bold) are being advertised by BGP across the link. But if I try to ping their IP addresses, I get nothing.

If I try to ping them using 'emulate-ping vxlan vxlan-id 3001 source-mac 0000-0000-0bb9 destination-mac 0050-56a4-xxxx' (where 0bb9 is the MAC address manually assigned to the distributed IP gateway and xxxx is the one of the two VMs), I get nothing.

If I 'display ip routing-table', I only get the subnet, gateway, and broadcast addresses:
10.130.1.0/24 Direct 0 0 10.130.1.1 Vsi3001
10.130.1.0/32 Direct 0 0 10.130.1.1 Vsi3001
10.130.1.1/32 Direct 0 0 127.0.0.1 InLoop0
10.130.1.255/32 Direct 0 0 10.130.1.1 Vsi3001

Neither of the individual host IP addresses shows up in the routing table; I don't know if that's expected behavior or not.

This is the only distributed IP gateway I've configured. Maybe I'm missing something, but I don't know what. I wouldn't think the BGP routes would need to be redistributed into OSPF, but I'm not certain that's accurate. I have only Area 0 throughout the network, but I've only these two sites and only four routers total, so spliting that up into multiple areas doesn't seem neccessary. I inherited the exising setup, and other VXLANs are being used, but they are not using distributed IP gateways; I'm limited in the changes I can make at this time because I have only production hardware to work with.

I used the 'EVI 2.0 EVPN Data Center Interconnect Deployment Guide', User Case #2, as my baseline as it most closely represented my existing environment and should allow me to add a future data center without major changes.

I've been trying to figure out what the problem is for a week now, and haven't had any success.

Any ideas would be appreciated. Thanks in advance.

[cttripp1]

Added detail: if I disable the distributed IP gateway on the 5940 where the VMs are NOT hosted (the site remote to the VMs), pings to the VMs work just fine.  It's only when the distributed IP gateway on that 5940 is enabled that pings across the WAN to the VMs work.

I could leave the distributed IP gateway diabled, but that defeats the whole purpose of having it.  And eventually I'll have VMs on both sides of the WAN link, so that's not a real solution.

[cttripp1 · Accepted solution]

Ignore this whole thread.  It's not a routing problem; it's a problem with the whole distributed-gateway local thing.  It's not working as advertised.

[HugoFS]

Just noticed this old thread in passing.  In case you're still interested...

I'm not sure it's correct to say "It's not working as advertised".  If I read your posts correctly, I think it's working exactly as advertised but you're missing one key fact.  If you have "distributed gateway local" configured (and the associated VSI interface is up and configured with the appropriate subnet), you will never get a response to traffic generated on a remote leaf if you generate the traffic from the VSI interface itself.  The forward path to the remote host works correctly, and traffic will hit the target, but the return path will hit the leaf that the endpoint is connected to directly first, and if that has an active distributed gateway interface, the response from the host will be absorbed by it, rather than forwarded back to the original source. 

If you want to check the path to a remote host across such an environment, you must use a unique source address for the remote host to reply to. 

Hope this helps.