Traffic generated by the switch doesn't respect VRF segregation

anubisg1 · Sat May 23, 2020 10:31 am

Hello,

we purchased a few CRS305-1G-4S+IN. The main idea is that we wanted to use those low cost switches as 10Gbps "servers/endpoints" to test our vxlan fabric with high end devices (cisco/arista/juniper/accton).

we would create a spine/leaf vxlan topology, and plug the CRS305 as a host. Additionally since the switch has 4 10Gbps we could sometimes logically split it in half using VRFs so that we could simulate two hosts.

the config we use in this example is the one below (notice that we don't have any bridge, i know about HW offload and it isn't a concern at the moment and furthermore i am not interested in the CRS to be a switch, i want it to be an end client (laptop, server, router, what ever you prefer)

# may/23/2020 09:15:38 by RouterOS 6.46.4
# software id = U90R-W9QL
#
# model = CRS305-1G-4S+
# serial number = 934D09963CED
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] l2mtu=9216 mtu=9000 name=eth1
set [ find default-name=sfp-sfpplus2 ] l2mtu=9216 mac-address=B8:69:F4:99:D1:4A mtu=9000 name=eth2
set [ find default-name=sfp-sfpplus3 ] l2mtu=9216 mtu=9000 name=eth3
set [ find default-name=sfp-sfpplus4 ] l2mtu=9216 mac-address=B8:69:F4:99:D1:4C mtu=9000 name=eth4
set [ find default-name=ether1 ] name=mgmt0
/interface bonding
add min-links=1 mode=802.3ad mtu=9000 name=bond1 slaves=eth1,eth2 transmit-hash-policy=layer-3-and-4
add min-links=1 mode=802.3ad mtu=9000 name=bond2 slaves=eth3,eth4 transmit-hash-policy=layer-3-and-4
/interface vlan
add interface=bond1 name=bond1.200 vlan-id=200
add interface=bond2 name=bond2.300 vlan-id=300
/interface list
add name=WAN
add name=LAN
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/interface list member
add interface=mgmt0 list=WAN
add list=LAN
/ip address
add address=172.31.1.9/24 interface=mgmt0 network=172.31.1.0
add address=10.10.200.10/24 interface=bond1.200 network=10.10.200.0
add address=10.10.30.20/24 interface=bond2.300 network=10.10.30.0
/ip dhcp-client
add disabled=yes interface=bond1.200
add disabled=yes interface=bond2.300
/ip dns
set servers=8.8.8.8,192.168.0.127
/ip route
add check-gateway=arp distance=1 gateway=10.10.30.1 routing-mark=HOST2
add check-gateway=arp distance=1 gateway=10.10.200.1 routing-mark=HOST1
add check-gateway=arp distance=1 gateway=172.31.1.222
add check-gateway=arp distance=1 dst-address=172.31.2.0/24 gateway=172.31.1.1
add check-gateway=arp distance=1 dst-address=192.168.0.0/24 gateway=172.31.1.1
add check-gateway=arp distance=1 dst-address=192.168.1.0/24 gateway=172.31.1.1
add check-gateway=arp distance=1 dst-address=192.168.10.0/24 gateway=172.31.1.1
/ip route vrf
add interfaces=bond2.300 routing-mark=HOST2
add interfaces=bond1.200 routing-mark=HOST1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www-ssl disabled=no
set api disabled=yes
set api-ssl disabled=yes
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/system clock
set time-zone-name=Europe/Prague
/system identity
set name=MKTK-SW01
/system ntp client
set enabled=yes primary-ntp=192.168.0.129
/system routerboard settings
set boot-os=router-os
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool mac-server ping
set enabled=no
/tool sniffer
set filter-direction=rx filter-interface=bond2
/tool user-manager database
set db-path=flash/user-manager

This below is my routing table

admin@MKTK-SW01] > ip route print 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          10.10.30.1                1
 1 ADC  10.10.30.0/24      10.10.30.20     bond2.300                 0
 2 A S  0.0.0.0/0                          10.10.200.1               1
 3 ADC  10.10.200.0/24     10.10.200.10    bond1.200                 0
 4 A S  0.0.0.0/0                          172.31.1.222              1
 5 ADC  172.31.1.0/24      172.31.1.9      mgmt0                     0
 6 A S  172.31.2.0/24                      172.31.1.1                1
 7 A S  192.168.0.0/24                     172.31.1.1                1
 8 A S  192.168.1.0/24                     172.31.1.1                1
 9 A S  192.168.10.0/24                    172.31.1.1                1

Here is my problem... if i try to make a trace between 10.10.200.10 in vrf HOST-1 to 10.10.30.20 in vrf HOST-2, the traffic is NOT routed according to vrf routing table. Instead it seems to be routed/bridge/whatever internally to the box as proven by the traceroute below:

[admin@MKTK-SW01] > tool traceroute 10.10.30.20 routing-table=HOST1 src-address=10.10.200.10
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS                                                                                                                  
 1 10.10.30.20                        0%    9   0.1ms     0.1     0.1     0.3     0.1

the only way to make traffic flow as expected seems to be by forcing the outgoing physical interface

[admin@MKTK-SW01] > tool traceroute 10.10.30.20 routing-table=HOST1 src-address=10.10.200.10 interface=bond1.200 
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS                                                                                                                  
 1 10.10.200.1                        0%    5   0.2ms     0.3     0.2     0.6     0.1                                                                                                                         
 2 1.1.1.3                            0%    5   0.2ms     0.2     0.2     0.3       0                                                                                                                         
 3 10.10.30.20                        0%    5   0.1ms     0.1     0.1     0.1       0

why is this happening? is it related to the output chain? how can i ensure that traffic sourced by the switch itself is always routed according to the vrf routing table instead of being somewhat short circuited inside the switch?

thanks a lot!

sindy · Sat May 23, 2020 12:07 pm

Traffic to and from own IP addresses of the device doesn't respect VRF. The routing-mark identifying the VRF instance is assigned as the traffic gets in via a VRF interface, but there is no in-interface for locally originated traffic. For incoming traffic to own addresses, the matching of the destination address to local addresses, and sending the matching traffic down the input chain, takes place before routing is done, and routing-mark is only taken into account by the routing process.

So you might use an action=mark-routing rule in chain=output of /ip firewall mangle to make the locally originated pakets use the desired routing table used by one of the VRFs, but the question is what other criteria would you use - for locally originated packets, the source address is normally chosen according to the route, not vice versa. Plus it is even more complex - a locally originated packet is first routed using routing table main; if a route is found, the packet passes through chain output of mangle, and if it eventually gets a routing-mark there, it gets routed again.

anubisg1 · Sat May 23, 2020 12:22 pm

I thought that the problem would be related to this thanks for confirming it...

With other vendors, all traffic, even the one that is generated by the router follows the vrf routing table unless vrf leaking is used by for example, specifying the outgoing vrf table (since if generated by the router, it must be sourced by an interface belonging to the vrf itself).

Regarding the source address, typically you would pick the ip address of the selected outgoing interface, unlewthe src address is manually specified, but still, in both cases, the vrf is known.

The whole point of a VRF is to have separate routing tables, different virtual routing instances.

I am not fully into mikrotik way of thinking but this behavior sounds more like a bug to be honest...
And my understanding is that this happens since router OS doesn't really use different routing tables, but instead, everything is merged into the main one

ErikCarlseen · Wed Jul 01, 2020 8:19 pm

The whole point of a VRF is to have separate routing tables, different virtual routing instances.

I am not fully into mikrotik way of thinking but this behavior sounds more like a bug to be honest...
And my understanding is that this happens since router OS doesn't really use different routing tables, but instead, everything is merged into the main one

MikroTik RouterOS seems to use the internal functionality of the Linux kernel for interface, routing, filtering, shaping, etc., so if you're familiar with the way that works (or become familiar with it) everything makes a lot more sense. The good news is that there is a ton of documentation out there on Linux kernel network traffic management, and if you figure out the feature you want, read the Kernel documentation on it, and come back to the MikroTik side of things you will have many "ahhhh... I see now" moments.

I would strongly agree that VRF handling on RouterOS would be much better if they made it work like Cisco, Juniper, etc. by default. I'm in a similar rabbit hole myself at the moment, and it seems straightforward to emulate traditional router VRF handling with firewall and route policy rules. It would also be very nice if we could specify which interface(s) various RouterOS services are bound to, which should allow the functionality I think we're both looking for.

mutluit · Wed Jul 01, 2020 9:12 pm

The whole point of a VRF is to have separate routing tables, different virtual routing instances.

I am not fully into mikrotik way of thinking but this behavior sounds more like a bug to be honest...
And my understanding is that this happens since router OS doesn't really use different routing tables, but instead, everything is merged into the main one
MikroTik RouterOS seems to use the internal functionality of the Linux kernel for interface, routing, filtering, shaping, etc., so if you're familiar with the way that works (or become familiar with it) everything makes a lot more sense.

In Linux you can define multiple routing tables and assign them to the interfaces as described here: http://www.linuxhorizon.ro/iproute2.html

Traffic generated by the switch doesn't respect VRF segregation

Traffic generated by the switch doesn't respect VRF segregation

Re: Traffic generated by the switch doesn't respect VRF segregation

Re: Traffic generated by the switch doesn't respect VRF segregation

Re: Traffic generated by the switch doesn't respect VRF segregation

Re: Traffic generated by the switch doesn't respect VRF segregation

Who is online