Community discussions

MikroTik App
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Unable to get routing on multi VRF setup working, routes marked as inactive

Thu Mar 31, 2022 7:16 pm

Hey all

Hoping this is a simple config issue that I'm just not seeing. I have a CRS309 that acts as my core switch with multiple VRFs. Each VRF will have unique default routes that go up to my firewall that has interfaces into each of the VRFs. Attached is a high level diagram.
Drawing1.jpg
Everything on my main routing table works fine going out through the firewall although oddly, if I look at the routing table it shows my configured default route for the MAIN routing table as entry 0 AND entry 1? with 1 being Inactive. In anycase, this is how it looks
[admin@CRS309] /ip/route> print
Flags: D - DYNAMIC; I, A - ACTIVE; c, s, y - COPY; H - HW-OFFLOADED
Columns: DST-ADDRESS, GATEWAY, DISTANCE
#      DST-ADDRESS      GATEWAY                  DISTANCE
0  AsH 0.0.0.0/0        172.19.0.9                      1
1  IsH 0.0.0.0/0                                        1
  DAcH 172.19.0.0/24    MainGateway                     0
  DAcH 10.10.10.0/24    HomeWired                       0
2  IsH 0.0.0.0/0        172.19.80.9                     1
  DAcH 172.19.80.0/24   VRF-X-Gateway@vrf-x             0
  DAcH 10.11.11.0/24    VRF-X-LAN@vrf-x                 0
3  IsH 0.0.0.0/0        172.19.81.9                     1
  DAcH 172.19.81.0/24   VRF-Y-Gateway@vrf-y             0
  DAcH 10.12.12.0/24    VRF-Y-LAN@vrf-y                 0
4  IsH 0.0.0.0/0        172.19.82.9                     1
  DAcH 172.19.82.0/24   VRF-Z-Gateway@vrf-z             0
  DAcH 10.13.13.0/24    VRF-Z-LAN@vrf-z                 0
I can ping the firewall interface from their respective VRFs no problem. Here's an example from the VRF-Z

[admin@CRS309] /tool> ping address=172.19.82.9 vrf=vrf-z src-address=172.19.82.1
SEQ HOST SIZE TTL TIME STATUS
0 172.19.82.9 56 64 1ms713us
0 172.19.82.9 56 64 1ms816us
1 172.19.82.9 56 64 1ms17us
1 172.19.82.9 56 64 1ms125us

The hosts within each VRF can talk to each other no problem. If I send traffic between VRFs, what should happen is they route out to the firewall and let the firewall route traffic. There's nothing showing on the firewall pcaps which makes sense since the default route for each VRF isn't being installed... I tried testing with check-gateway using arp or ping and neither works and the CRS still marks it inactive (that's also why you will see I have ping or arp on check-gateway of the various routes since I was testing). Is there anything I'm missing here?

EDIT: Noting here on OP post that I tested 7.3 where it apparently should only enable l3hw-offload on the main table but this does not fix the issue with VRF breaking when enabled.
You do not have the required permissions to view the files attached to this post.
Last edited by donatoroman on Sun Jun 12, 2022 5:59 pm, edited 1 time in total.
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Thu Mar 31, 2022 8:45 pm

Ok, I think I found the issue. What I noticed when I was checking ALL traffic on the firewall itself, the CRS309 was pinging the gateways of the other VRFs using an IP thats on the main routing table. I guess if you don't define a vrf-interface, the router will just use what's available on the main routing table. I explicitly defined vrf-interface for each routing entry including the main one too and now routing is working as expected.
/ip route
add check-gateway=ping dst-address=""
add check-gateway=arp disabled=no dst-address=0.0.0.0/0 gateway=172.19.0.9 routing-table=main vrf-interface=MainGateway
add check-gateway=ping dst-address=0.0.0.0/0 gateway=172.19.80.9 routing-table=vrf-x vrf-interface=VRF-X-Gateway
add check-gateway=ping dst-address=0.0.0.0/0 gateway=172.19.81.9 routing-table=vrf-y vrf-interface=VRF-Y-Gateway
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=172.19.82.9 routing-table=vrf-z vrf-interface=VRF-Z-Gateway
I'm not sure why there's an entry for this here

add check-gateway=ping dst-address=""

I can't seem to remove that...
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Thu Mar 31, 2022 10:25 pm

Spoke too soon, it stopped working altogether... This is odd. I'll have to troubleshoot this some more and then post back here of my findings. What I did notice was that a test host that I placed into one of the VRFs, like say VRF-Z, for some reason their traffic is being routed through the Main routing table. I'm unclear why the CRS would have the packet routed there.
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Fri Apr 01, 2022 12:46 am

Ok, I'm throwing in the towel and I can't figure out why multi VRF setup doesn't work here. If I just define the default route in the MAIN table, my main LAN works fine, but obviously the other vrfs do not work.

/ip route
add check-gateway=arp disabled=no dst-address=0.0.0.0/0 gateway=172.19.0.9 routing-table=main vrf-interface=MainGateway

As soon as I introduce another routing rule there for a completely DIFFERENT routing-table, the hosts that were on the main routing-table uses that instead which breaks connectivity.

/ip route
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=172.19.0.9 routing-table=main vrf-interface=MainGateway
add check-gateway=ping disabled=no dst-address=0.0.0.0/0 gateway=172.19.80.9 routing-table=vrf-x vrf-interface=VRF-X-Gateway

It's as if every subsequent rule overrules the previous ones. This is easily noticeable because my firewall would see ingress traffic from my hosts on the main routing-table, and then as soon as I add the second routing rule there, the hosts show up coming from VRF-X.

What is up with that? LOL.
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Mon Apr 04, 2022 6:27 am

*bump*

Anyone able to get multi-vrf routing with the same prefix to work? It looks to me that it can't handle true VRF, meaning it can't have multiple copies of the same prefix even if they are on separate VRFs pointing to different next hop addresses.
 
eduplant
Member Candidate
Member Candidate
Posts: 139
Joined: Tue Dec 19, 2017 9:45 am

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Mon Apr 04, 2022 7:21 am

Is this on v7.x.x or v6.x.x?

I’m not super familiar with VRF on Mikrotik specifically but I’m willing to configure this in my lab and use it as a learning experience. I’d rather do it with a matching RouterOS version since routing and feature support is so different across each right now.


Looking closer at your output it's pretty obvious its v7.x.x on second inspection.
 
eduplant
Member Candidate
Member Candidate
Posts: 139
Joined: Tue Dec 19, 2017 9:45 am

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Mon Apr 04, 2022 8:33 am

Okay, I think I got it. My test setup is using two v7.1.5 CHRs with one acting as the firewall:

donatoroman_topo.PNG

When installing your configuration at first I also had inactive default routes. The problem seems to be that the next-hops of the static routes are always looked up in the main table unless specified otherwise. Changing the next-hops from something like gateway=172.19.80.9 to gateway=172.19.80.9@vrf-x allows it to look up the next hop in the appropriate table and then install the route:

donatoroman_static.PNG

Here is a ping from vrf-z to vrf-x through the firewall (and implicitly relying on two of the working default routes in those tables):

donatoroman_ping.PNG

And :tool sniff proof that the packet is actually leaving hitting the firewall and coming in and out tagged on VLANs 982 and 980 as you might expect:

donatoroman_pcap.PNG

Let me know how this goes when applied to your situation.
You do not have the required permissions to view the files attached to this post.
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Mon Apr 04, 2022 5:44 pm

Thanks so much for following up and testing it on your end. So I tested this and inter-vlan routing works but it breaks internet connectivity again like before because my hosts that are on the main routing table are now using the default route configured on a separate VRF. So in this setup here:

add check-gateway=arp disabled=no dst-address=0.0.0.0/0 gateway=172.19.0.9 routing-table=main
add check-gateway=arp disabled=no dst-address=0.0.0.0/0 gateway=172.19.82.9@VRF-X routing-table=VRF-X

Hosts that are on the main routing table are now using VRF-X as its routing table and my firewall logs reflect that.

I am starting to feel like CRS line does NOT support true VRF.

EDIT: I also tested by defining the @main in the first entry but same issue. As soon as I add the second default route for a different VRF, the hosts in the main table lose access to internet.
 
eduplant
Member Candidate
Member Candidate
Posts: 139
Joined: Tue Dec 19, 2017 9:45 am

Re: Unable to get routing on multi VRF setup working, routes marked as inactive  [SOLVED]

Mon Apr 04, 2022 7:38 pm

Thanks so much for following up and testing it on your end. So I tested this and inter-vlan routing works but it breaks internet connectivity again like before because my hosts that are on the main routing table are now using the default route configured on a separate VRF.

The internet in this case is supposed to be pictured north of the firewall right? I added a loopback to the "firewall" in my test setup w/ 8.8.8.8/32 to represent the internet and when I attempt to ping from the router to there, it does still correctly use the main table and it takes Eth1/1 to the firewall like it's supposed to.

Can you do me a favor and post a copy of your :routing route print detail? I'm beginning to wonder whether this might be a negative interaction with L3 hardware offloading. According to the documentation [1], VRF isn't supported on L3-hardware-offloading but I just assumed that it would only install main table routes and leave the rest alone. It'd also be easy to rule out if you just disable it with :interface ethernet switch set 0 l3-hw-offloading=no and see if you get different behavior.

[1] https://help.mikrotik.com/docs/display/ ... Offloading
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Wed Apr 06, 2022 2:56 am

I'll have to wait for this weekend to troubleshoot further. I really hope that's not an issue with clashing with hardware offloading! Can you confirm the config syntax you used to add the static routes? Did you also include vrf-interface?
 
eduplant
Member Candidate
Member Candidate
Posts: 139
Joined: Tue Dec 19, 2017 9:45 am

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Wed Apr 06, 2022 3:21 am

Can you confirm the config syntax you used to add the static routes? Did you also include vrf-interface?

I didn't specify vrf-interface, I just specified the dst-address, the routing-table, and the next-hop gateway using the <address>@<table> format. I'm frankly not even sure what vrf-interface even does. The only thing the documentation even says about it is "VRF interface name" and that it is a string with a default value of "10"????? [1]


[1] https://wiki.mikrotik.com/wiki/Manual:IP/Route
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Sat Apr 09, 2022 5:48 pm

Ok, I did some more testing and that is exactly the issue. If you enable hardware offloading, it breaks VRF completely. I was able to reproduce this behavior disabling and enabling the feature. If enabled, all VRFs will utilize the last configured routes you have in your routing tables. If disabled, they will stick to their respective routing tables.

Normally this wouldnt be a big deal for networks that only utilize a single vlan to cover their entire lan but in my case, my setup utilizes multiple vlans in the main table for servers, users, etc, that has alot of inter-vlan traffic, and I also have broken off a VRF to hold guest wifi and iot devices, etc.

This sounds like I should probably look at alternatives for my core switch and leave my Mikrotiks as just L2 switches.
 
eduplant
Member Candidate
Member Candidate
Posts: 139
Joined: Tue Dec 19, 2017 9:45 am

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Sat Apr 09, 2022 7:09 pm

Ok, I did some more testing and that is exactly the issue. If you enable hardware offloading, it breaks VRF completely.

Glad we isolated it. I still think this behavior deserves some inspection by Mikrotik. It's all fine and well to not support VRF via hardware offloading, but the correct answer should be to invalidate the "hardware offloading" H flag for VRF routes so that only the main table ones are installed. Turning the hardware forwarding table into a FIFO route salad of the most recent additions from multiple VRFs just isn't sane.

I haven't exactly combed the patch notes for this either. Are you running v7.1.5?

This sounds like I should probably look at alternatives for my core switch and leave my Mikrotiks as just L2 switches.

Mikrotik doesn't really do roadmaps but they did communicate [1] that a lot more software support for hardware offloading is coming in v7.x.x, including VRF. Still, for right now a capable CPU-based router like one of the CCRs is probably your best bet and leaving the CRSs to do line-rate switching.



[1] https://youtu.be/7_uLxZYYEpQ?t=86
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Sat Apr 09, 2022 11:19 pm

Yes, I'm running 7.1.5 and I agree, if hw-offloading isnt supported on VRFs, then it really shouldn't break VRF functionality. I would expect it to work on the main table and not on custom VRFs. I opened a ticket with them so maybe they have an avenue of communication to the devs for either a feature request/bug fix?
 
glow
newbie
Posts: 29
Joined: Sun Dec 05, 2021 1:56 am

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Sun Apr 10, 2022 12:32 am

Ok, I did some more testing and that is exactly the issue. If you enable hardware offloading, it breaks VRF completely. I was able to reproduce this behavior disabling and enabling the feature. If enabled, all VRFs will utilize the last configured routes you have in your routing tables. If disabled, they will stick to their respective routing tables.

Normally this wouldnt be a big deal for networks that only utilize a single vlan to cover their entire lan but in my case, my setup utilizes multiple vlans in the main table for servers, users, etc, that has alot of inter-vlan traffic, and I also have broken off a VRF to hold guest wifi and iot devices, etc.

This sounds like I should probably look at alternatives for my core switch and leave my Mikrotiks as just L2 switches.
At 1:32 of the CCR2216 video https://www.youtube.com/watch?v=7_uLxZYYEpQ

Notes HW offloads for VRF are still being worked on.

and the table here:
https://help.mikrotik.com/docs/display/ ... ureSupport

shows it is not supported at the moment.
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Mon Apr 11, 2022 5:08 am

Yup, noted. I read it wasn't supported on VRFs but I didn't think it would break it.
 
donatoroman
newbie
Topic Author
Posts: 30
Joined: Tue Dec 07, 2021 9:03 pm

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Sun Jun 12, 2022 8:36 am

From version 7.3

*) l3hw - offload only main routing table;

!?!? Will test out this new version and see if that corrects the issue here.

EDIT: Tested, still breaks VRF. Oh well.
 
paolobyte
newbie
Posts: 28
Joined: Wed Apr 05, 2017 6:00 am

Re: Unable to get routing on multi VRF setup working, routes marked as inactive

Fri Jul 28, 2023 5:44 am

Same experience here. Using 7.10 already.
I'm going to raise a ticket about this.

Who is online

Users browsing this forum: anav, AshuGite and 39 guests