Hi all,

I am trying to configure a lab simulating a "3-branch" setup where two of the CE's receive a DIA/Internet [100.64.88.0/24] network with one VRF, and the third CE receives/announces just the VRF routes. I attached a diagram illustrating the physical setup. I am having trouble establishing the eBGP peering between PE03-CE03 on the VRF enabled /30 link. The eBGP on the "Internet" /30 came is up without any issues. The BGP debugs showed the following message - "Reject connection: EBGP peer is not on a shared network and multihop is not configured". I enabled multihop on both sides which established the peering, but the routes between them are still not propagated correctly and I don't believe enabling multihop is or should be the correct root fix.

PE-to-CE Overview:
I am utilizing the default BGP instance with two separate peering sessions - 100.64.88.4/30 "Internet" and 10.88.123.0/30 for "VRF-Green". I tried a variety of workarounds before enabling multihop and nothing seemed to establish the VRF peering. I was able to ping across just fine, I added route look-up / policy route rules with no luck. My "Internet" peer is enabled with "IP address family" only and the VRF peer is enabled with just VPN4. Lastly, I tried adding a separate BGP instance for VRF-Green [with the correct routing table mark] and the end result was the same. Any help would be greatly appreciated. Thanks in advance!

Could you share the output of the "/routing bgp export" command?

Hi clambert, thanks your reply.

PE03:
# RouterOS 6.48.6
# model = CCR2004-1G-12S+2XS
# serial number = D4F10DC99677
/routing bgp instance
set default as=65088 router-id=100.64.88.5
add as=65088 disabled=yes name=VRF-Green router-id=10.88.123.1 routing-table=vrf-green
/routing bgp instance vrf
add redistribute-connected=yes redistribute-other-bgp=yes routing-mark=vrf-green
/routing bgp peer
add hold-time=1m30s keepalive-time=30s name=iBGP-CORE remote-address=10.88.255.10 remote-as=65088 update-source=loop88-bridge
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=iBGP-PE01 remote-address=10.88.255.1 remote-as=65088 update-source=loop88-bridge
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=iBGP-PE02 remote-address=10.88.255.2 remote-as=65088 update-source=loop88-bridge
add address-families=vpnv4 hold-time=1m30s keepalive-time=30s multihop=yes name=eBGP-CE03-Green remote-address=10.88.123.2 remote-as=65388
add hold-time=1m30s keepalive-time=30s name=eBGP-CE03-DIA remote-address=100.64.88.6 remote-as=65388

CE03:
# model = RBD53iG-5HacD2HnD
# serial number = D96C0C9F8B01
/routing bgp instance
set default as=65388 router-id=100.64.88.6
add as=65388 disabled=yes name=VRF-Green router-id=10.88.123.2 routing-table=vrf-green
/routing bgp instance vrf
add redistribute-connected=yes redistribute-other-bgp=yes routing-mark=vrf-green
/routing bgp network
add network=100.64.88.6/32 synchronize=no
add network=100.64.88.32/28 synchronize=no
add disabled=yes network=10.88.103.1/32 synchronize=no
add disabled=yes network=10.88.103.2/32 synchronize=no
add disabled=yes network=10.88.103.3/32 synchronize=no
add disabled=yes network=10.88.103.4/32 synchronize=no
add disabled=yes network=10.88.103.5/32 synchronize=no
/routing bgp peer
add address-families=vpnv4 hold-time=1m30s keepalive-time=30s multihop=yes name=eBGP-PE03-Green remote-address=10.88.123.1 remote-as=65088
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=eBGP-PE03-DIA remote-address=100.64.88.5 remote-as=65088

I think you have the following errors in your PE config:
- the non default BGP instance in vrf-green is disabled.
- the BGP peerings between PE an CE are not using the default BGP instance.
- you are using VPNv4 address-family instead IP address family in the peering to the CE.

In the CE config:
- you should use IP address family for the two peerings.

I am using the default BGP instance in all peering at the moment with the BGP VRF enabled. I disabled the "VRF-Green" instance to simplify the design. I believe the separate instance is required only when there is no BGP VRF configured. Is this statement not valid?

I have enabled the IP/VPN4 address families on both PE-CE peerings and disabled multihop. The VRF peering is not coming up and this was the initial prompt for this post.

[atetu@mpls-lab-pe03] > ping 10.88.123.2
SEQ HOST SIZE TTL TIME STATUS
0 10.88.123.2 56 64 0ms
1 10.88.123.2 56 64 0ms
2 10.88.123.2 56 64 0ms
3 10.88.123.2 56 64 0ms
4 10.88.123.2 56 64 0ms
[atetu@mpls-lab-pe03] > ping routing-table=vrf-green 10.88.123.2
SEQ HOST SIZE TTL TIME STATUS
0 10.88.123.2 56 64 0ms
1 10.88.123.2 56 64 0ms
2 10.88.123.2 56 64 0ms
3 10.88.123.2 56 64 0ms
sent=4 received=4 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

[atetu@mpls-lab-pe03] > ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADC 10.88.123.0/30 10.88.123.1 l3vpn-green 0
1 X S 0.0.0.0/0 192.168.99.254 1
2 ADo 10.88.151.0/30 10.88.153.1 110
3 ADo 10.88.152.0/30 10.88.153.1 110
4 ADC 10.88.153.0/30 10.88.153.2 sfp-sfpplus10 0
5 ADo 10.88.255.1/32 10.88.153.1 110
6 ADo 10.88.255.2/32 10.88.153.1 110
7 ADC 10.88.255.3/32 10.88.255.3 loop88-bridge 0
8 ADo 10.88.255.10/32 10.88.153.1 110
9 ADC 100.64.88.4/30 100.64.88.5 dia-access 0
10 ADb 100.64.88.6/32 100.64.88.6 20
11 ADb 100.64.88.32/28 100.64.88.6 20

[atetu@mpls-lab-pe03] > routing bgp export
# dec/11/2023 13:07:55 by RouterOS 6.48.6
# software id = FZ7H-Z3K8
#
# model = CCR2004-1G-12S+2XS
# serial number = D4F10DC99677
/routing bgp instance
set default as=65088 router-id=100.64.88.5
add as=65088 disabled=yes name=VRF-Green router-id=10.88.123.1 routing-table=vrf-green
/routing bgp instance vrf
add redistribute-connected=yes redistribute-other-bgp=yes routing-mark=vrf-green
/routing bgp peer
add hold-time=1m30s keepalive-time=30s name=iBGP-CORE remote-address=10.88.255.10 remote-as=65088 update-source=loop88-bridge
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=iBGP-PE01 remote-address=10.88.255.1 remote-as=65088 update-source=loop88-bridge
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=iBGP-PE02 remote-address=10.88.255.2 remote-as=65088 update-source=loop88-bridge
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=eBGP-CE03-Green remote-address=10.88.123.2 remote-as=65388
add hold-time=1m30s keepalive-time=30s name=eBGP-CE03-DIA remote-address=100.64.88.6 remote-as=65388

I have disabled the following route rules and now I can only ping across with the vrf mark applied.

PE:
[atetu@mpls-lab-pe03] > ip route rule print
Flags: X - disabled, I - inactive
0 src-address=10.88.123.1/32 dst-address=10.88.123.2/32 action=lookup table=vrf-green

CE:
[atetu@mpls-lab-ce03] > ip route rule print
Flags: X - disabled, I - inactive
0 src-address=10.88.123.2/32 dst-address=10.88.123.1/32 action=lookup table=vrf-green

[atetu@mpls-lab-pe03] > ping 10.88.123.2
SEQ HOST SIZE TTL TIME STATUS
0 no route to host
1 no route to host
2 no route to host
sent=3 received=0 packet-loss=100%

[atetu@mpls-lab-pe03] > ping 10.88.123.2 routing-table=vrf-green
SEQ HOST SIZE TTL TIME STATUS
0 10.88.123.2 56 64 0ms
1 10.88.123.2 56 64 0ms
2 10.88.123.2 56 64 0ms
3 10.88.123.2 56 64 0ms
sent=4 received=4 packet-loss=0% min-rtt=0ms avg-rtt=0ms max-rtt=0ms

What is preventing the BGP to establish on the VRF enabled /30 link?

I've been working on my "PE02-CE02" setup this morning where I have two VRF peerings instead of 1 VRF and 1 DIA peering. Same result unfortunately.

Posting an update on this lab since I finally got it to work!

Key Takeaways:
I did not need to configure the "BGP-VRF" on the "CE", just the vrf-instance for vrf-marked traffic and the default-instance for "internet" or all non vrf-marked traffic. The "PE" router on the other hand, has both "vrf-green" and default instances along with the bgp-vrf configured. Lastly, I am relying on the "Redistributed Connected" routes instead of advertising vrf-marked and non-vrf routes separately with the appropriate filters in place.

PE03:
[atetu@mpls-lab-pe03] > routing bgp export
# model = CCR2004-1G-12S+2XS
# serial number = D4F10DC99677
/routing bgp instance
set default as=65088 client-to-client-reflection=no router-id=100.64.88.5
add as=65088 client-to-client-reflection=no name=VRF-Green router-id=10.88.123.1 routing-table=vrf-green
/routing bgp instance vrf
add redistribute-connected=yes redistribute-other-bgp=yes routing-mark=vrf-green
/routing bgp peer
add hold-time=1m30s keepalive-time=30s name=iBGP-CORE remote-address=10.88.255.10 remote-as=65088 update-source=loop88-bridge
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=iBGP-PE01 remote-address=10.88.255.1 remote-as=65088 update-source=loop88-bridge
add address-families=ip,vpnv4 hold-time=1m30s keepalive-time=30s name=iBGP-PE02 remote-address=10.88.255.2 remote-as=65088 update-source=loop88-bridge
add hold-time=1m30s instance=VRF-Green keepalive-time=30s name=eBGP-CE03-Green remote-address=10.88.123.2 remote-as=65388
add default-originate=always hold-time=1m30s keepalive-time=30s name=eBGP-CE03-DIA remote-address=100.64.88.6 remote-as=65388

CE03:
[atetu@mpls-lab-ce03] > routing bgp export
# model = RBD53iG-5HacD2HnD
# serial number = D96C0C9F8B01
/routing bgp instance
set default as=65388 client-to-client-reflection=no redistribute-connected=yes router-id=100.64.88.6
add as=65388 client-to-client-reflection=no name=VRF-Green redistribute-connected=yes router-id=10.88.123.2 routing-table=vrf-green
/routing bgp network
add disabled=yes network=100.64.88.6/32 synchronize=no
add disabled=yes network=100.64.88.32/28 synchronize=no
add disabled=yes network=10.88.103.1/32 synchronize=no
add disabled=yes network=10.88.103.2/32 synchronize=no
add disabled=yes network=10.88.103.3/32 synchronize=no
add disabled=yes network=10.88.103.4/32 synchronize=no
add disabled=yes network=10.88.103.5/32 synchronize=no
/routing bgp peer
add hold-time=1m30s instance=VRF-Green keepalive-time=30s name=eBGP-PE03-Green remote-address=10.88.123.1 remote-as=65088
add hold-time=1m30s in-filter=ebgp-pe03-accept keepalive-time=30s name=eBGP-PE03-DIA out-filter=ebgp-pe03-announce remote-address=100.64.88.5 remote-as=65088

[atetu@mpls-lab-pe03] > ip route print where received-from=eBGP-CE03-Green
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 ADb 10.88.103.1/32 10.88.123.2 20
1 ADb 10.88.103.2/32 10.88.123.2 20
2 ADb 10.88.103.3/32 10.88.123.2 20
3 ADb 10.88.103.4/32 10.88.123.2 20
4 ADb 10.88.103.5/32 10.88.123.2 20
5 Db 10.88.123.0/30 10.88.123.2 20
[atetu@mpls-lab-pe03] >
[atetu@mpls-lab-pe03] > ip route print where received-from=eBGP-CE03-DIA
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
# DST-ADDRESS PREF-SRC GATEWAY DISTANCE
0 Db 100.64.88.4/30 100.64.88.6 20
1 ADb 100.64.88.32/28 100.64.88.6 20
2 ADb 100.64.88.41/32 100.64.88.6 20
3 ADb 100.64.88.42/32 100.64.88.6 20
4 ADb 100.64.88.43/32 100.64.88.6 20
5 ADb 100.64.88.44/32 100.64.88.6 20
6 ADb 100.64.88.45/32 100.64.88.6 20
7 Db 192.168.99.0/24 100.64.88.6 20