EVPN+VXLAN with eBGP underlay

RouterOS Forwarding Protocols
1

Hi everyone,

I am trying to create a EVPN+VXLAN leaf/spine topology in a lab environment on version 7.21.2, using eBGP for the underlay, but when I add the overlay the EVPN routes become “unreachable” on the spine router, see below. When changing out the underlay for OSPF everything starts working perfectly. I tried to advertise standard IPv4 routes as well on the overlay but it’s the same and I cannot for the life of me figure this out. Any insights would be greatly appreciated!

[admin@spine1] > /routing route print where afi=evpn 
Flags: U - UNREACHABLE; b - BGP
Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE
   DST-ADDRESS                                GATEWAY       AFI   DISTANCE  SCOPE  TARGET-SCOPE
Ub [172.18.240.3:1]imet:0|172.18.240.3        172.18.240.3  evpn        20     40            30
Ub [172.18.240.4:1]macip:0|AA:C1:AB:A0:9A:6B  172.18.240.4  evpn        20     40            30
Ub [172.18.240.4:1]imet:0|172.18.240.4        172.18.240.4  evpn        20     40            30
Topology and configuration

My topology is like this:

topology
topology1002×604 26.5 KB

And my configuration, BGP underlay and OSPF underlay for comparison (only one leaf for brevity):

BGP underlay

spine1

/ip address add interface=ether2 address=10.255.240.0/31
/ip address add interface=ether3 address=10.255.240.2/31
/ip address add interface=lo address=172.18.240.1/32

/system logging add topics=route action=memory

/routing id add name=loopback select-dynamic-id=only-loopback

/routing filter rule add chain=loopbacks rule="if (dst in 172.18.240.0/24 && dst-len == 32) { accept; }"
/routing bgp instance add name=fabric as=65001 router-id=loopback
/routing bgp template add name=underlay afi=ip input.filter=loopbacks output.filter-chain=loopbacks output.redistribute=connected
/routing bgp connection add name=underlay-leaf1 instance=fabric templates=underlay local.address=10.255.240.0 local.role=ebgp remote.address=10.255.240.1 remote.as=65003
/routing bgp connection add name=underlay-leaf2 instance=fabric templates=underlay local.address=10.255.240.2 local.role=ebgp remote.address=10.255.240.3 remote.as=65004
/routing bgp template add name=overlay afi=evpn multihop=yes nexthop-choice=propagate
/routing bgp connection add name=overlay-leafs instance=fabric templates=overlay local.address=172.18.240.1 local.role=ebgp remote.address=172.18.240.0/24

leaf1

/ip address add interface=ether2 address=10.255.240.1/31
/ip address add interface=lo address=172.18.240.3/32

/system logging add topics=route action=memory

/routing id add name=loopback select-dynamic-id=only-loopback
/routing filter rule add chain=loopbacks rule="if (dst in 172.18.240.0/24 && dst-len == 32) { accept; }"
/routing bgp instance add name=fabric as=65003 router-id=loopback
/routing bgp template add name=underlay afi=ip input.filter=loopbacks output.filter-chain=loopbacks output.redistribute=connected
/routing bgp connection add name=underlay-spine1 instance=fabric templates=underlay local.address=10.255.240.1 local.role=ebgp remote.address=10.255.240.0 remote.as=65001
/routing bgp template add name=overlay afi=evpn multihop=yes
/routing bgp connection add name=overlay-spine1 instance=fabric templates=overlay local.address=172.18.240.3 local.role=ebgp remote.address=172.18.240.1 remote.as=65001

/interface bridge add name=bridge1 pvid=10 vlan-filtering=yes
/interface vxlan add bridge=bridge1 bridge-pvid=10 learning=no local-address=172.18.240.3 name=vxlan10 vni=100010
/interface bridge port add bridge=bridge1 interface=ether3 pvid=10
/ip address add interface=bridge1 address=203.0.113.1/24
/routing bgp evpn add export.route-targets=65000:100010 import.route-targets=65000:100010 instance=fabric name=evpn-100010 vni=100010

OSPF underlay

spine1

/ip address add interface=ether2 address=10.255.240.0/31
/ip address add interface=ether3 address=10.255.240.2/31
/ip address add interface=lo address=172.18.240.1/32

/system logging add topics=route action=memory

/routing id add name=loopback select-dynamic-id=only-loopback
/routing ospf instance add name=underlay router-id=loopback
/routing ospf area add area-id=0.0.0.0 instance=underlay name=area0
/routing ospf interface-template add area=area0 interfaces=ether2,ether3
/routing ospf interface-template add area=area0 interfaces=lo passive
/routing bgp instance add name=fabric as=65001 router-id=loopback
/routing bgp template add name=overlay afi=evpn multihop=yes nexthop-choice=propagate
/routing bgp connection add name=overlay-leafs instance=fabric templates=overlay local.address=172.18.240.1 local.role=ebgp remote.address=172.18.240.0/24

leaf1

/ip address add interface=ether2 address=10.255.240.1/31
/ip address add interface=lo address=172.18.240.3/32

/system logging add topics=route action=memory

/routing id add name=loopback select-dynamic-id=only-loopback
/routing ospf instance add name=underlay router-id=loopback
/routing ospf area add area-id=0.0.0.0 instance=underlay name=area0
/routing ospf interface-template add area=area0 interfaces=ether2
/routing ospf interface-template add area=area0 interfaces=lo passive
/routing bgp instance add name=fabric as=65003 router-id=loopback
/routing bgp template add name=overlay afi=evpn multihop=yes
/routing bgp connection add name=overlay-spine1 instance=fabric templates=overlay local.address=172.18.240.3 local.role=ebgp remote.address=172.18.240.1 remote.as=65001

/interface bridge add name=bridge1 pvid=10 vlan-filtering=yes
/interface vxlan add bridge=bridge1 bridge-pvid=10 learning=no local-address=172.18.240.3 name=vxlan10 vni=100010
/interface bridge port add bridge=bridge1 interface=ether3 pvid=10
/ip address add interface=bridge1 address=203.0.113.1/24
/routing bgp evpn add export.route-targets=65000:100010 import.route-targets=65000:100010 instance=fabric name=evpn-100010 vni=100010
2

With default scope, target scope settings, nexthop for BGP routes are not resolved over other BGP routes. You need to change scopes and target scopes to recursively resolve gateways over BGP routes.

1 Like
3

Thanks!

That worked great, since the underlay target-scope was 40 I added this below to get it to work.

spine1

/routing filter rule add chain=evpn rule="set scope-target 40; accept;"
/routing bgp template add name=overlay afi=evpn input.filter=evpn multihop=yes nexthop-choice=propagate

leaf1

/routing filter rule add chain=evpn rule="set scope-target 40; accept;"
/routing bgp template add name=overlay afi=evpn input.filter=evpn multihop=yes

Now the routes are active and get re-advertised to the leafs

[admin@spine1] > /routing route print where afi=evpn            
Flags: A - ACTIVE; b - BGP
Columns: DST-ADDRESS, GATEWAY, AFI, DISTANCE, SCOPE, TARGET-SCOPE, IMMEDIATE-GW
   DST-ADDRESS                                GATEWAY       AFI   DISTANCE  SCOPE  TARGET-SCOPE  IMMEDIATE-GW       
Ab [172.18.240.3:1]macip:0|AA:C1:AB:96:A7:E9  172.18.240.3  evpn        20     40            40  10.255.240.1%ether2
Ab [172.18.240.3:1]imet:0|172.18.240.3        172.18.240.3  evpn        20     40            40  10.255.240.1%ether2
Ab [172.18.240.4:1]macip:0|AA:C1:AB:7E:1A:F4  172.18.240.4  evpn        20     40            40  10.255.240.3%ether3
Ab [172.18.240.4:1]imet:0|172.18.240.4        172.18.240.4  evpn        20     40            40  10.255.240.3%ether3
4

To save you or others from headache when implementing VXLAN with BGP EVPN overlay and trying to combine it with eBGP underlay.

On Mikrotik, you can’t (yet?) specify the nexthop-choice definition per afi.
Meaning, you set either force-self or propagate for all afis of the BGP connection.

eBGP, for the underlay, as per RFC, would usually assume nexthop-choice=force-self, to make sure the next hop for all packets is the peer, not the originating router.

VXLAN via BGP EVPN, as the overlay, on the other side requires nexthop-choice=propagate, because VXLAN VTEPs require knowledge of the original VTEP IP, so in case of VXLAN based on BGP EVPN of the originating router.

This brings a conflict.

Yes, you could create routing filter rules for each afi and each BGP connection, to modify each received route and set next-hop for afi=ip/ipv6 → peer-address, but this is likely to cause additional efforts, extra care and probably errors at some point in time and larger scaling.

Therefore, at the time of this writing and the current RouterOS Version not providing different nexthop-choice per afi, i would recommend to stay with your initial approach of using OSPF or IS-IS for the underlay (basically providing nexthop-choice=self) and (e)BGP EVPN for the VXLAN overlay with nexthop-choice=propagate.

5

You do not need to set nexthop per AFI?

Run eBGP peering for underly to distriburte loopbacks.
Run another set of eBGP peering for overly (peers on loopback addresses) with different nexhop-choice.

That is how almost every other vendor is doing.

Here are some examples for inspiration:

https://www.jasonvanpatten.com/2018/11/15/cumulus-and-arista-evpn-configuration/

https://overlaid.net/2018/08/27/arista-bgp-evpn-overview-and-concepts/

6

A nice idea as well, @mrz , thanks for your feedback.

Both approaches have one thing in common:
You save headache when separating the underlay routing (via interface IPs) from the overlay routing (via loopback IPs) - either using a combination of OSPF and BGP or using two separate BGP connections with different nexthop-choice.

Indeed i was referring to FRR, which is the routing part in cumulus as well, which is able to set different nexthop-choice per AFI on the same BGP connection.

7

You would need overly BGP peers on loopbacks anyway as it is the best practice.