7.23.1 Resolvable route is unreachable. Did I find a bug or doing something wrong?

RouterOS General
1

EDIT: Nevermind, while I still don't understand why this route was unreachable everything stops working when both sides are in an VRF. VRFs are really the biggest whack-a-mole ever. What I am actually trying to so is same as described in this question.

EDIT2: Solution to this particular question: 7.23.1 Resolvable route is unreachable. Did I find a bug or doing something wrong? - #6 by CGGXANNX . Turns out I also tried that already but what the issue is that the VRF has to be specified even if it's to be resolved in the same VRF. I was using the target VRF, not the source VRF

First off, I am setting up a maclan interface for inter-VRF communication (inspired by this post but I think in this case the detour over a non-local bogus IP isn't needed):

/interface bridge
add name=br-xvrf2 protocol-mode=none

/interface macvlan
add interface=br-xvrf2 mac-address=3E:8A:A1:C7:A6:08 name=xvrf-as
add interface=br-xvrf2 mac-address=16:B3:12:46:B8:C9 name=xvrf-main2

/ip address
add address=172.18.1.20/24 interface=xvrf-main2 network=172.18.1.0
add address=172.18.1.21/24 interface=xvrf-as network=172.18.1.0
/ip vrf
add interfaces=vlan55,vrrp55,xvrf-as name=vrf-as

In this instance, communication between VRF vrf-as and main VRF.

Let me first describe what works well and as expected: I can access a local service (wireguard and ssh) which is assigned to vrf-as from main VRF:

/ip route
add distance=1 dst-address=192.0.2.210/32 gateway=172.18.1.21
routing-table=main scope=30 target-scope=10
add disabled=no distance=1 dst-address=10.2.79.0/24 gateway=vlan3 routing-table=vrf-as scope=30 target-scope=10

In this case 192.0.2.210 is a local IP of RouterOS whose interface is in vrf-as. 10.2.79.0/24 is the LAN (vlan3) which is part of main VRF. Without these interfaces, a host in 10.2.79.0/24 is not able to reach 192.0.2.210 (because packets arrive in main VRF but 192.0.2.210 is in vrf-as). With this setup, any host from 10.2.79.0/24 can perfectly reach ssh/wireguard available on 192.0.2.210. Great!

Now I want to do the opposite as well: DNS is part of main VRF but should also be available from vlan55 (which is part of vrf-as):

/ip route
add distance=1 dst-address=10.2.79.2/32 gateway=172.18.1.20 routing-table=vrf-as scope=30 target-scope=10

Here 10.2.79.2 is RouterOS's own IP in main VRF. Note that above route command is identical to the one above (just IP flipped).

However, this route shows up as Unreachable!

/ip/route/print detail where dst-address=10.2.79.2/32 routing-table=vrf-as
Flags: D - DYNAMIC; X - DISABLED, I - INACTIVE, A - ACTIVE;
c - CONNECT, s - STATIC, r - RIP, b - BGP, o - OSPF, i - IS-IS, d - DHCP, v - VPN, m - MODEM, y - BGP-MPLS-VPN; H - HW-OFFLOADED; + - ECMP
27 Is dst-address=10.2.79.2/32 routing-table=vrf-as gateway=172.18.1.20 immediate-gw="" distance=1 scope=30 target-scope=10

This is crazy, because it's identical to the first working scenario for one and second, a connected route for 172.18.1.0/24 exists so it doesn't make sense at all that this route is unreachable! It is NOT.

For comparison, both scenarios, along with the connected routes:

/ip/route/print detail where (dst-address=10.2.79.2/32 or dst-address=172.18.1.0/24) and routing-table=vrf-as
Flags: D - DYNAMIC; X - DISABLED, I - INACTIVE, A - ACTIVE;
c - CONNECT, s - STATIC, r - RIP, b - BGP, o - OSPF, i - IS-IS, d - DHCP, v - VPN, m - MODEM, y - BGP-MPLS-VPN; H - HW-OFFLOADED; + - ECMP
27  Is   dst-address=10.2.79.2/32 routing-table=vrf-as gateway=172.18.1.20 immediate-gw="" distance=1 scope=30 target-scope=10

DAc   dst-address=172.18.1.0/24 routing-table=vrf-as gateway=xvrf-as@vrf-as immediate-gw=xvrf-as distance=0 scope=10 target-scope=5 local-address=172.18.1.21%xvrf-as@vrf-as

/ip/route/print detail where (dst-address=192.0.2.210/32 or dst-address=172.18.1.0/24) and routing-table=main
Flags: D - DYNAMIC; X - DISABLED, I - INACTIVE, A - ACTIVE;
c - CONNECT, s - STATIC, r - RIP, b - BGP, o - OSPF, i - IS-IS, d - DHCP, v - VPN, m - MODEM, y - BGP-MPLS-VPN; H - HW-OFFLOADED; + - ECMP
11  As   ;;; xVRF2: Services on 192.0.2.210 accessible from main
dst-address=192.0.2.210/32 routing-table=main gateway=172.18.1.21 immediate-gw=172.18.1.21%xvrf-main2 distance=1 scope=30 target-scope=10

DAc   dst-address=172.18.1.0/24 routing-table=main gateway=xvrf-main2 immediate-gw=xvrf-main2 distance=0 scope=10 target-scope=5 local-address=172.18.1.20%xvrf-main2

Did I hit yet another RouterOS VRF bug or is there anything wrong?

PS:

/ip/arp/print where address=172.18.1.20 or address=172.18.1.21
Flags: D - DYNAMIC; C - COMPLETE
Columns: ADDRESS, MAC-ADDRESS, INTERFACE, VRF, STATUS

ADDRESS      MAC-ADDRESS        INTERFACE   VRF     STATUS

1 DC 172.18.1.21  3E:8A:A1:C7:A6:08  xvrf-main2  main    stale
0 DC 172.18.1.20  16:B3:12:46:B8:C9  xvrf-as     vrf-as  stale
2

Much thanks for the Nevermind at the top! Saved some angst! I have not used VRFs yet, and if nothing else this thread has motivated me to "stay the course" LOL.

3

You are becoming very selective with age ...
... no CapsMan/Caps
... no VRFs

4

I still want to try for the ultimate most seemless failover on using wireguard and two ISPs, and VPS using BGP and OSPF, but I keep seeing fixes for that functionality so I will wait till its stable LOL.

5

I've posted an example configuration in the other thread, where it's possible to access the router's services in @main from other VRFs:

6

The reason why this route is unreachable is because you forgot to include @vrf-as at the end of the gateway. It should be gateway=172.18.1.20@vrf-as.

7

Nice to show it but why not add an explanation of

a. how mt allows one to add "happy endings" to things.
b. what it says if reading it, and also the functionality it applies.

8

I consider @ to mean "at" :wink: :
Address@VRF: Tells the router which routing table to look at to find the path.

and % to mean "through" : Address%Interface: Tells the router which physical/virtual pipe to go through.

9

When no @xxx is added to the gateway, it's assumed to be @main. So

means: in the routing table vrf-as destinations 10.2.79.2/32 can reach the gateway by resolving 172.18.1.20 in the main routing table/VRF.

However, in the main VRF, 172.18.1.20 is a local IP address of the router itself, as defined by:

because the xvrf-main2 interface is in the main VRF. The router's own address is not allowed as gateway (you can try even without any VRF in your set up, to add a route with gateway=192.168.88.1 for example, when your bridge has the 192.168.88.1/24 address to see the same effect) that's why the route is marked as invalid and unreachable.

After adding @vrf-as to the end of the address, so that it becomes gateway=172.18.1.20@vrf-as, when the router tries to reach the gateway, the 172.18.1.20 address will be resolved in the vrf-as VRF instead of main.

In this VRF, 172.18.1.20 is not an IP address of the router (instead 172.18.1.21 is one of the IP addresses of the router in the VRF vrf-as), but some remote IP address that can be contacted using the interface xvrf-as as outgoing interface. It's the result of this:

which automatically adds this route in the vrf-as routing table:

because 172.18.1.20 is resolvable in vrf-as as remote (non-local) address, the

/ip route
add distance=1 dst-address=10.2.79.2/32 gateway=172.18.1.20@vrf-as routing-table=vrf-as scope=30 target-scope=10

route will not be marked as invalid nor unreachable anymore.

10

EDIT: Whops for some reason I never got a notification for the other thread and did not see your answer. As I describe in this post below, I think this works much simpler, without macvlan interfaces.

I could swear I tried this (and all other combinations, including and excluding interfaces)

Having said that, I found a much nicer method that works completely without the macvlan hack. I'll document it here for everyone looking for the same.

It seems for crossing VRF, when the target is on the same router (and possibly? both not main VRF), the following is sufficient:

# ordinary static route leak from vrf-main to vrf-as (response route)
/ip route
add disabled=no distance=1 dst-address=192.0.2.208/28 gateway=\
    vlan44@vrf-as routing-table=vrf-main scope=30 target-scope=10

# Special /32 routes (forward route)
/ip route
add comment="XVRF: Access services@vrf-main from vrf-as" disabled=no distance=1 dst-address=10.2.4.2/32 gateway=vrf-main@vrf-main routing-table=vrf-as scope=30 target-scope=10
add comment="XVRF: Access services@vrf-main from vrf-as" disabled=no distance=1 dst-address=10.2.4.254/32 gateway=vrf-main@vrf-main routing-table=vrf-as scope=30 target-scope=10

In this example, 10.2.4.2 is the IP address of the router which is from an interface inside vrf-main which should be accessible from vrf-as (e.g. from network 192.0.2.0/24). 10.2.4.254 is a VRRP address.

Probably a firewall rule is required to allow traffic to flow. For example, to access Mikrotik's DNS server in vrf-main from vrf-as:

/ip firewall filter
add action=accept chain=forward comment="XVRF: Allow dns access from vrf-as" dst-port=53 out-interface=vrf-main protocol=udp src-address=192.0.2.0/24

This really is it! :slight_smile:

Now an added bonus is having the service (e.g. DNS) also available on a router owned address from vrf-as, for example, 192.0.2.210. It took me a while to get this working properly. The trivial part is to set up a DNAT rule. But the non-trivial part is to make sure that the return packets are immediately routed to the right routing table, otherwise DNAT is looked up twice and the TCP handshake succeeds but following packets all time out. Extremely nasty issue to debug!

/ip firewall nat
add action=dst-nat chain=dstnat dst-address=192.0.2.210 dst-port=53
in-interface-list=MY_PREFIX protocol=udp src-address=192.0.2.0/24 to-addresses=10.2.4.254
/ip firewall mangle
add action=mark-routing chain=output comment="XVRF PortFW: dns" new-routing-mark=vrf-as out-interface=vrf-as protocol=udp src-port=53

Now being super excited that this is working more elegantly, I am not able to get dynamic inter-VRF route leaking (via bgp vpn) working and by now I'm fairly confident this must be a regression/bug in RouterOS. Have you ever used /routing/bgp/vpn for dynamic VRF route leaking in the past and was it working?

11

Yes, the MACVLAN "tunnel" workaround was intended for the situation when one of the two sides of the communication is a router's local address in main and the other side is in a different VRF, because accessing the local address in main has always been special (in the routing lookup decision this is the local chain / local table in the routing rules).

Unlike the router's address in the other non-main VRFs: you could always reach those when you steer the traffic to that VRF. For example, if you remove or disable this route:

and add a mangle mark-routing rule to steer traffic destined to 10.2.4.2 to routing-mark=vrf-main, you'll see that the address is pingable from the vrf-as VRF. While if that address was in @main instead, then no amount of routing-mark=main would be able to allow you to reach it at all.

Also, while steering with mangle to the router's addresses on different non-main VRFs works for ping, if the router services behind that VRF only accept incoming connections from the particular VRF, then they might reject the incoming connection (with TCP reset if using TCP).