What does /ip route vrf really do?

RouterOS Forwarding Protocols
1

I am using a separate route table for a network (hamnet) and have BGP put routes in that table.
Using a couple of ip route rules, I select the proper route table. For local interfaces, I have a
fixed route in the second table. This works fine.

As far as I understand, “/ip route vrf” is a layer on top of this, that may ease the management of
such an environment. However, I can find little information on what the parameters really do.

Is there a description of what creating a VRF and setting some of the parameters really does to
the underlying system in terms of route table entries (marks), rules, etc? That would make
migration (and the decision if it is a good idea to migrate) easier.

2

Hi,
VRF means Virtual Routing and Forwarding. It is one of the main services that you can use with MPLS.
There is very good documentation about it on wiki.
https://wiki.mikrotik.com/wiki/Manual:Virtual_Routing_and_Forwarding

3

I know what it does but I don’t think the documentation is that good.
I would like to know exactly what will happen when I add a VRF and add some interfaces.

  • what route tables and entries will it create
  • what routing rules will it create
  • is it possible to see this information in the GUI
  • is it possible to add custom rules
  • what is the risk of getting locked out during the migration process

I think I will have to experiment on a CHR…

4

My first experiments indicate that:

/ip route vrf
add interfaces=ether1 routing-mark=hamnet

will only put the address of ether1 into a table named hamnet but will do nothing else that is visible.
Moving routes pointing to that interface to the hamnet table, and adding ip route rules to select that table
for traffic sourcing from that interface address will have to be done manually.

Is that correct?

5

I can’t speak 100% for MikroTik but in Cisco a VRF is a way to virtualize a router. It creates separation at layer 3 similar to what VLANs do at layer 2. This means if you create a VRF named “hamnet” and add an interface to it that is the routing table that gets the “connected” route for that interface. You can add a route to a VRF statically or by running dynamic routing protocols that support VRFs. This is visualized in BGP as the VPNv4 and VPNv6 address pool. The interface or interfaces that are added to “hamnet” can only communicate based on the “hamnet” routing table. Say you have ether1 and ether2 in “hamnet” and ether3 and ether4 in the default VRF. Traffic internal to the MikroTik could go from ether1 to ether2 and back but not from ether1 to ether3 or ether4. It is separated.

Think of it as slicing an individual router up to do multiple tasks safely. VRFs could be used to create the separation needed to have a router be an Internet edge router and an a router for the internal network.

Here is an oldy but a goody that might help:

http://packetlife.net/blog/2009/apr/30/intro-vrf-lite/

6

Maybe my question was not very clear…
What I mean is not to ask what a VRF is for, or what its principles are. I know that.
What I want to ask is what the /ip route vrf command really does in RouterOS.
When I enter it, what will the router do?
I know how to operate /ip route rule and /ip route with routing-mark= parameter, and it looks like /ip route vrf
uses the same mechanisms. So I would like to know what internal commands or actions the example command
I gave will be translated to, so I know the impact of the command and I know what other commands I should remove.

7

Hello,

ip vrf is not a macro for special commands, as in, you cannot recreate what it does using a set of other commands.

It actually doesn’t behave just like routing-marks defined within a firewall rule (for instance when you do load-balancing). The rules for processing traffic that enters via a VRF interface are different from those that are used when you mark packets (routing) via the firewall facility. Basically in two things:

1.- route selection fallback and
2.- route-recursion

When you mark routing via firewall rules you’re telling the router to lookup a specific routing table to find a way to forward said packet, in the case there’s no proper route match in the specified table the router will fallback to the main routing table. And in the case there’s a match and it requires recursion it’ll use the main routing table for it.

On a VRF this doesn’t happen, when traffic arrives an interface that belongs to a VRF if there’s no route match in the VRF’s table the traffic will be discarded, if there’s and it needs recursion it’ll do so within the same VRF table not using the main.

This is important to understand since a VRF purpose is traffic isolation and router virtualization.

Regarding whether it’ll move routes from the main table to the VRF table, it won’t. As with other brands, when you define a VRF you also need to define instances of the routing protocols for said VRF and when adding static routes you have to explicitly state this as well. When you define a VRF you’re effectively creating another router with its own interfaces and optionally its own routing protocol instances, the VRF won’t be aware of the Main table, other VRFs or other routing tables, so if, for instance, you happen to add to a VRF the interface being used to connect to the internet your LAN users will loose connectivity.

Why? again, because it’s basically two routers in one, one with the LAN interface and another with the WAN interface that doesn’t know anything about each other in the forwarding and control plane.

8

Thanks for the further description.
But when doing experiments, and also when looking at stated limitations in the documentation, it really looks like the MikroTik VRF
function (probably unlike other router software) is really doing “hidden” setup for packet markings and ip route rules to do its work.
For example, I can route traffic into another VRF by just defining an ip rule alongside the VRF.

/ip route vrf
add interfaces=ether1 routing-mark=hamnet
/ip route rule
add src-address=44.0.0.0/8 table=hamnet
add dst-address=44.0.0.0/8 table=hamnet

This just routes the traffic from/to that network into the VRF.
So, I think while the description tries to describe the intended functionality, it is not what is really happening.

As I have existing configuration with rules like that and static routes with marks in the table, and I would like to see if and how
I could maybe change that into a VRF, it would still be interesting to know what is going on.

I expect there are some hidden marking rules that mark traffic incoming on the interfaces of the VRF so that
in the routing those marks will be used. The behaviour “if no match” (stop or go back to main table) is just specified
by the rule, so that is not a decisive factor either.

9

Hello,

What you’re describing is basically policy based routing (PBR) and it’s something you can do with other brands as well, you can reference traffic and force it via an specific VRF Table (among other things).

Same thing you’re doing here could be done in Cisco with:

ip vrf hamlet
access-list 100 permit ip 44.0.0.0 0.0.0.255 any
access-list 100 permit ip any 44.0.0.0 0.0.0.255
route-map hamlet-test permit 10
 match ip address 100
 set vrf hamlet

Unlike MikroTik though, in Cisco IOS you’d need to attach this policy to a particular interface before being able to use it.

Anyway back to the topic:
The fact that you can do this doesn’t mean it’s behind stage doing some command macroing. It’s basically doing what it’s supposed to, virtualizing the router and isolating traffic, when you get to ISP architectures, as well as DC or some sort or really specialized company request, you’ll need what’s known as inter-vrf-traffic-leaking which is what you’re doing with PBR and can be done in other ways as well (hint, one involves using BGP).

Again the differences are in the way traffic is processed, in your example think what would happen with the return traffic? would it get routed without intervention from you? Say for instance a packet coming from 192.168.1.1 goes to 44.0.0.1, and the incoming interface is not within the VRF you’re specifying, 44.0.0.1 is reachable via ether1 it answer, what would happen with this answer? it gets back to 192.168.1.1?

Now do the same scenario without VRF but plain route marks, what happens this time?

I quote myself:

When you mark routing via firewall rules you’re telling the router to lookup a specific routing table to find a way to forward said packet, in the case there’s no proper route match in the specified table the router will fallback to the main routing table. And in the case there’s a match and it requires recursion it’ll use the main routing table for it.

On a VRF this doesn’t happen, when traffic arrives an interface that belongs to a VRF if there’s no route match in the VRF’s table the traffic will be discarded, if there’s and it needs recursion it’ll do so within the same VRF table not using the main.

This is important to understand since a VRF purpose is traffic isolation and router virtualization.

Also:

The behaviour “if no match” (stop or go back to main table) is just specified
by the rule, so that is not a decisive factor either

This is not the case, once there’s a match by a rule it goes to a route-lookup process, this process is where the “no match” actions and decisions I mention are taken, it doesn’t go back to the policy rules to check whether there’s another rule pointing to another table or what’s the action to be taken. Unless I’m not getting what you’re saying, policy rules processing and route-lookup processing are two separate and independent processes. Also, when you’re getting traffic from a VRF interface if you want to model it as a policy-rule, it’d be a “catch-all” one, pointing to the table specified within the VRF unless overridden by PBR.

Check this document, the first lines https://wiki.mikrotik.com/wiki/Manual:Virtual_Routing_and_Forwarding

VRF also is used mainly with BGP and MPLS to provide aggregated services to customers, and one more thing I have to point is that (in MikroTik) VRF doesn’t isolate traffic going towards the router (input), you can get traffic from a VRF interface that points to an IP of the router that’s set within the main table and the router will reply, so traffic going towards the router is not isolated within a VRF.

10

The discussion remains too much focussed on what a VRF is and how it is supposed to work on a functional level.
In the document you quote it says:

Technically VRFs are based on policy routing. There is exactly one policy route table for each active VRF. The existing policy routing support in MT RouterOS is not changed; but on the other hand, it is not possible to have policy routing within a VRF.

My question is and remains: what policy routing configurations are made by the example config I gave.
When you don’t know, no problem, I wait for (and hope for) a MikroTik employee with inside knowledge of the system to explain it.

11

I’m actually not focusing on telling you what VRF is supposed to do in any way, I’m telling you why you can’t compare it to a policy route like you’re stating and that it’s not a macroing that you can recreate with individual commands, that it’s got its specific use cases.

“based on” doesn’t mean “it is”, I even gave you an example for you to test to see -why- you can’t compare these the way you want.

  • what route tables and entries will it create
  • what routing rules will it create
  • is it possible to see this information in the GUI
  • is it possible to add custom rules
  • what is the risk of getting locked out during the migration process

Using the information I provided you, the answer to these are quite simple:
1.- It will create one routing table, when you assign an interface and add an IP to said interface it’ll add that entry to it, if you create an instance of say OSPF tied to that table and that instance “learns” X number of routes, these routes will go to said table. If you then add another interface and add another IP to this new interface, it’ll add the route to the table.
2.- It won’t create any routing rule that you can see or manipulate (just like any other brand out there)
3.- No
4.- No, read what you quoted from the page I asked you to check and what I’ve been telling you
5.- Taking into account:

and one more thing I have to point is that (in MikroTik) VRF doesn’t isolate traffic going towards the router (input), you can get traffic from a VRF interface that points to an IP of the router that’s set within the main table and the router will reply, so traffic going towards the router is not isolated within a VRF

If you understand correctly what that document says and what I’ve told you, the risks of doing it in a controlled way are few.

You’re right about it having an internal mechanism but I highly doubt MikroTik will tell you what exactly it is, and I don’t see it relevant in any way, nor understand why it’s to you, but I do have to admit that every mind is its own world. Hopefully they’ll get back to you with more information.

12

Mikrotik is Linux based and Linux has support for multiple routing tables and routing rules. A VRF attaches the connected interface route to an alternative routing table so that you can have overlapping routes. I prefer examples:
Create PPPoE client and obtain default gateway, this will automatically go in to the main routing table. Then create VPN connection and configure it to set a default gateway. Normally this then wouldn’t work as users would either continue to route out via default gateway or tunnel would collapse. If you however attach the LAN interface and VPN interfaces to a VRF you would provide the router with direct internet access whilst limiting the LAN to exclusively being and to route out via the VPN tunnel.

With regards to Mikrotik not properly isolating any local IPs for interfaces placed in their own VRFs:
This requires a later kernel, the one in RouterOS works the same way RedHat 6 does. It creates a routing table ‘255’, which has a fixed priority of 0 (first), so any local IP is accessible irrespective of what routing table the connected route is placed in. Herewith the output from a standard Linux host:

[admin@unix-01 ~]# ip rule list
0: from all lookup 255
32766: from all lookup main
32767: from all lookup default
[admin@unix-01 ~]# ip route show table 255
broadcast 192.168.1.0 dev eth0 proto kernel scope link src 192.168.1.3
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
local 192.168.1.3 dev eth0 proto kernel scope host src 192.168.1.3
broadcast 192.168.1.255 dev eth0 proto kernel scope link src 192.168.1.3
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1

Any IP, broadcast or loop back subnet is automatically present in table 255.

13

Yes, this is what I know and that is why I think it works like you describe!
The part about the multi routing tables is clear, that is in fact written on the documentation page, but what I would
like to know is what kind of ip rules or iptables -t mangle rules are added to keep the traffic of a VRF within a set of ports.
Is an incoming packet on a port associated with a VRF automatically marked to be routed via the routing table associated with that VRF?
And where is that rule placed?
That is the info I am after.

For two reasons:

  1. I want to know how it really works so I do not encounter something that is not compatible with what I want to do with it.
  2. I already have a manual setup that works. I would like to migrate it to VRF when possible, but without locking myself out in the process
    or forgetting to delete or add things that are required with the VRF solution.
14

As others have already stated, assigning an interface to a VRF should not necessitate iptable entries nor ip routing rules.

Creating ip routing rules creates ‘ip rule’ entries.

Some implementation examples:
You place certain interfaces in to a VRF, to separate routing from the router’s own requirements (VPN example earlier), but want to check reachability of various devices in the VRF from a network monitoring system (NMS) that has connectivity to the router via an interface that is not part of the VRF:
ip route rule add src-addres=<nms_ip> dst-address=10.0.0.0/24 table=vpn
ip route rule add src-address=10.0.0.0/24 dst-address=<nms_ip> table=main

You could alternatively achieve something similarly by using standard routes, although they wouldn’t restrict routing to the source IP/subnet:
ip route add dst-address=10.0.0.0/24 gateway=ether2
ip route add dst-address=<nms_ip> gateway=41.243.52.1%ether1 routing-mark=vpn

15

@pe1chl I was wondering the same. Did you ever get more clarification regarding this?

16

No, I just kept using my existing setup with multiple routing tables and “/ip route rule” which I fully understand.
Probably I want to do too much routing into and out of the VRF to be using that mechanism, it appears to be suited to things like separate customers each with their own IP space which is overlapping, while I want to use it to route a different IP space (hamnet) using different routing policy from the remainder of IP space (internet), while still being able to tunnel one over the other or to use one as a fallback for the other.
That is probably not what VRF is useful for, and “ip route rule” (and “route marking”) does a good job for me.

17

@pe1chl So do the firewall filter rules apply to all routing tables by default?

18

In the config without VRF: yes. I use interface lists where necessary to apply firewall rules to one network only.

19

pe1chl
What would be the application of VRF in the homlan setting?
Whether one wants
a. functional separation
b. device/use separation
c. any other kind of separation or hiving off to a separate network

means that normally the use of managed switches will be necessary. Then vlans make sense as the first go to available strategy.
What do VRFs bring to the table, what is the attraction??

20

The attraction is to be able to use overlapping address spaces handled by a single router and/or having an individual routing for different customers configured more easily than using the whole suite of policy routing settings.

Let’s limit the example to routing inside overlapping address spaces.

Imagine four networks - A, B, C and D. A and B use identical subnets, and so do C and D. On your router, one interface is used to connect each of the four networks, and assigned an IP address from the corresponding subnet. Therefore, the system dynamically creates routes to these connected subnets, and choses between them in an unpredictable way (it’s not random, it’s just unpredictable, so always one of the identical subnets in each pair will be used but you cannot know in advance which one it will be).

What you want is to make sure that a packet from network A with destination address matching both C and D will always go to C, while a packet from network B to the same destination address will always go to D.

With “manual” policy routing, you have to create mangle rules to assign a routing mark “AC” to packets coming in via interfaces to which network A and network C are connected, and a routing mark “BD” to packets coming in via interfaces to which network B and network D are connected. Also, you have to manually create routes to subnet A and subnet C with routing mark AC, and routes to subnet B and subnet D with routing mark BD, to override the dynamically created routes as described above.

As a result, when a packet towards a “C or D” address comes in via interface A, it gets a routing mark AC, so only the route to C becomes valid for it although the dynamically generated route to D has the same dst-address; likewise, a packet towards a “C or D” address coming in via interface B gets a routing mark BD and so only the BD-marked route to D becomes eligible for it.

Now if one of the interfaces went down, the marked route for its subnet would become inaccessible, so the routing would fall back to the dynamically created route without any routing mark, so the packet would end up in a wrong destination network. To prevent this, you would have to use two /ip route rule items saying routing-mark=X action=lookup-only-in-table table=X (one with X=AC and the other one with X=BD).

So in total, it takes:

  • two mangle rules referring to two interface lists with two member interfaces each
  • four marked routes
  • two routing rules

Using the vrf configuration, you use just two configuration items to define the two virtual routing instances instead:
/ip route vrf
add interfaces=A,C routing-mark=AC
add interfaces=B,D routing-mark=BD
These guarantee that:

  • the dynamically created routes to the connected subnets A,B,C,D will be (re)created with a routing-mark AC and BD respectively instead of being created in the main routing table (so no need to manually create routes to A, B, C, D subnets and to use routing rules to prevent fallback to identical routes in main routing table in case of interface being down,
  • packets entering through any of the interfaces A, B, C, D will be assigned the corresponding routing-mark automatically, so no need to create the mangle rules.

So things are much simpler to set up and stay orderly if you take the vrf way in some scenarios. But it has some limitations - the same routing tables/routing marks are used for both the vrf and for the “usual” policy routing, so if you would want to use the “usual” policy routing together with a vrf setup, your brain may quickly reach the boiling point trying to keep track with the mutual dependencies.

Another use out of the scope of the example above is, as mentioned in previous posts, the possibility to update each vrf-created routing table independently by a dedicated instance of a dynamic routing protocol.