Community discussions

MikroTik App
 
anubisg1
just joined
Topic Author
Posts: 10
Joined: Sat Feb 23, 2019 1:29 pm
Contact:

Traffic generated by the switch doesn't respect VRF segregation

Sat May 23, 2020 10:31 am

Hello,

we purchased a few CRS305-1G-4S+IN. The main idea is that we wanted to use those low cost switches as 10Gbps "servers/endpoints" to test our vxlan fabric with high end devices (cisco/arista/juniper/accton).

we would create a spine/leaf vxlan topology, and plug the CRS305 as a host. Additionally since the switch has 4 10Gbps we could sometimes logically split it in half using VRFs so that we could simulate two hosts.

the config we use in this example is the one below (notice that we don't have any bridge, i know about HW offload and it isn't a concern at the moment and furthermore i am not interested in the CRS to be a switch, i want it to be an end client (laptop, server, router, what ever you prefer)
# may/23/2020 09:15:38 by RouterOS 6.46.4
# software id = U90R-W9QL
#
# model = CRS305-1G-4S+
# serial number = 934D09963CED
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] l2mtu=9216 mtu=9000 name=eth1
set [ find default-name=sfp-sfpplus2 ] l2mtu=9216 mac-address=B8:69:F4:99:D1:4A mtu=9000 name=eth2
set [ find default-name=sfp-sfpplus3 ] l2mtu=9216 mtu=9000 name=eth3
set [ find default-name=sfp-sfpplus4 ] l2mtu=9216 mac-address=B8:69:F4:99:D1:4C mtu=9000 name=eth4
set [ find default-name=ether1 ] name=mgmt0
/interface bonding
add min-links=1 mode=802.3ad mtu=9000 name=bond1 slaves=eth1,eth2 transmit-hash-policy=layer-3-and-4
add min-links=1 mode=802.3ad mtu=9000 name=bond2 slaves=eth3,eth4 transmit-hash-policy=layer-3-and-4
/interface vlan
add interface=bond1 name=bond1.200 vlan-id=200
add interface=bond2 name=bond2.300 vlan-id=300
/interface list
add name=WAN
add name=LAN
/tool user-manager customer
set admin access=own-routers,own-users,own-profiles,own-limits,config-payment-gw
/interface list member
add interface=mgmt0 list=WAN
add list=LAN
/ip address
add address=172.31.1.9/24 interface=mgmt0 network=172.31.1.0
add address=10.10.200.10/24 interface=bond1.200 network=10.10.200.0
add address=10.10.30.20/24 interface=bond2.300 network=10.10.30.0
/ip dhcp-client
add disabled=yes interface=bond1.200
add disabled=yes interface=bond2.300
/ip dns
set servers=8.8.8.8,192.168.0.127
/ip route
add check-gateway=arp distance=1 gateway=10.10.30.1 routing-mark=HOST2
add check-gateway=arp distance=1 gateway=10.10.200.1 routing-mark=HOST1
add check-gateway=arp distance=1 gateway=172.31.1.222
add check-gateway=arp distance=1 dst-address=172.31.2.0/24 gateway=172.31.1.1
add check-gateway=arp distance=1 dst-address=192.168.0.0/24 gateway=172.31.1.1
add check-gateway=arp distance=1 dst-address=192.168.1.0/24 gateway=172.31.1.1
add check-gateway=arp distance=1 dst-address=192.168.10.0/24 gateway=172.31.1.1
/ip route vrf
add interfaces=bond2.300 routing-mark=HOST2
add interfaces=bond1.200 routing-mark=HOST1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set www-ssl disabled=no
set api disabled=yes
set api-ssl disabled=yes
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/system clock
set time-zone-name=Europe/Prague
/system identity
set name=MKTK-SW01
/system ntp client
set enabled=yes primary-ntp=192.168.0.129
/system routerboard settings
set boot-os=router-os
/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN
/tool mac-server ping
set enabled=no
/tool sniffer
set filter-direction=rx filter-interface=bond2
/tool user-manager database
set db-path=flash/user-manager
This below is my routing table
admin@MKTK-SW01] > ip route print 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          10.10.30.1                1
 1 ADC  10.10.30.0/24      10.10.30.20     bond2.300                 0
 2 A S  0.0.0.0/0                          10.10.200.1               1
 3 ADC  10.10.200.0/24     10.10.200.10    bond1.200                 0
 4 A S  0.0.0.0/0                          172.31.1.222              1
 5 ADC  172.31.1.0/24      172.31.1.9      mgmt0                     0
 6 A S  172.31.2.0/24                      172.31.1.1                1
 7 A S  192.168.0.0/24                     172.31.1.1                1
 8 A S  192.168.1.0/24                     172.31.1.1                1
 9 A S  192.168.10.0/24                    172.31.1.1                1
Here is my problem... if i try to make a trace between 10.10.200.10 in vrf HOST-1 to 10.10.30.20 in vrf HOST-2, the traffic is NOT routed according to vrf routing table. Instead it seems to be routed/bridge/whatever internally to the box as proven by the traceroute below:
[admin@MKTK-SW01] > tool traceroute 10.10.30.20 routing-table=HOST1 src-address=10.10.200.10
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS                                                                                                                  
 1 10.10.30.20                        0%    9   0.1ms     0.1     0.1     0.3     0.1                                                                                                                         
the only way to make traffic flow as expected seems to be by forcing the outgoing physical interface
[admin@MKTK-SW01] > tool traceroute 10.10.30.20 routing-table=HOST1 src-address=10.10.200.10 interface=bond1.200 
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS                                                                                                                  
 1 10.10.200.1                        0%    5   0.2ms     0.3     0.2     0.6     0.1                                                                                                                         
 2 1.1.1.3                            0%    5   0.2ms     0.2     0.2     0.3       0                                                                                                                         
 3 10.10.30.20                        0%    5   0.1ms     0.1     0.1     0.1       0     
why is this happening? is it related to the output chain? how can i ensure that traffic sourced by the switch itself is always routed according to the vrf routing table instead of being somewhat short circuited inside the switch?

thanks a lot!
------------------------------------------
Andrea Florio - Senior Network Architect
CCIE R&S #46985
------------------------------------------
 
sindy
Forum Guru
Forum Guru
Posts: 5099
Joined: Mon Dec 04, 2017 9:19 pm

Re: Traffic generated by the switch doesn't respect VRF segregation

Sat May 23, 2020 12:07 pm

Traffic to and from own IP addresses of the device doesn't respect VRF. The routing-mark identifying the VRF instance is assigned as the traffic gets in via a VRF interface, but there is no in-interface for locally originated traffic. For incoming traffic to own addresses, the matching of the destination address to local addresses, and sending the matching traffic down the input chain, takes place before routing is done, and routing-mark is only taken into account by the routing process.

So you might use an action=mark-routing rule in chain=output of /ip firewall mangle to make the locally originated pakets use the desired routing table used by one of the VRFs, but the question is what other criteria would you use - for locally originated packets, the source address is normally chosen according to the route, not vice versa. Plus it is even more complex - a locally originated packet is first routed using routing table main; if a route is found, the packet passes through chain output of mangle, and if it eventually gets a routing-mark there, it gets routed again.
Instead of writing novels, post /export hide-sensitive. Use find&replace in your favourite text editor to systematically replace all occurrences of each public IP address potentially identifying you by a distinctive pattern such as my.public.ip.1.
 
anubisg1
just joined
Topic Author
Posts: 10
Joined: Sat Feb 23, 2019 1:29 pm
Contact:

Re: Traffic generated by the switch doesn't respect VRF segregation

Sat May 23, 2020 12:22 pm

I thought that the problem would be related to this thanks for confirming it...

With other vendors, all traffic, even the one that is generated by the router follows the vrf routing table unless vrf leaking is used by for example, specifying the outgoing vrf table (since if generated by the router, it must be sourced by an interface belonging to the vrf itself).

Regarding the source address, typically you would pick the ip address of the selected outgoing interface, unlewthe src address is manually specified, but still, in both cases, the vrf is known.

The whole point of a VRF is to have separate routing tables, different virtual routing instances.

I am not fully into mikrotik way of thinking but this behavior sounds more like a bug to be honest...
And my understanding is that this happens since router OS doesn't really use different routing tables, but instead, everything is merged into the main one
------------------------------------------
Andrea Florio - Senior Network Architect
CCIE R&S #46985
------------------------------------------

Who is online

Users browsing this forum: amt, Google [Bot], Shalom and 167 guests