Community discussions

MUM Europe 2020
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

CRS VLAN Routing

Sun Dec 01, 2019 4:16 pm

In my setup i have a WAN router which provides my VLAN to my network, however this is only a 1gbit link therefore i have purchased a CRS317-1G-16S+ to have between my ESXi hosts providing
10gbit vlan routing. Im new to Mikrotik but do i understand this correctly.

rt-core01 (non mikrotik 1gbit) vlan 10 192.168.10.1/24
sw-core-01 (non mikrotik 10gbit) trunk
rt-core02 (CRS 10gbit) vlan 10 192.168.10.2/24
esxi01 (trunk) vlan 10,20,30 etc..
esxi02 (trunk) vlan 10,20,30 etc..
esxi03 (trunk) vlan 10,20,30 etc..

Should i create a VLAN interface on the CRS or just a VLAN on the bridge? My assumption is that if i only create a bridge then the routing will fall back on rt-core01 not on the new CRS? Is this correct or am i totally wrong here?
 
kamillo
Member Candidate
Member Candidate
Posts: 158
Joined: Tue Jul 15, 2014 5:44 pm

Re: CRS VLAN Routing

Mon Dec 02, 2019 11:51 am

 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Mon Dec 02, 2019 3:22 pm

And ... CRS devices are switches with L3 functionality. They are capable of routing (e.g. between VLANs), but nowhere wirespeed. Official test results, together with my own interpretation[*] indicate that your unit of choice can route with speeds around 500Mbps.

[*] My experience goes that the number most relevant to real-life cases in that result table is under "Routing, 25 ip filter rules, [packet size] 512 byte". If you do your best (no firewall filter rules), you might get near 1Gbps.
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Mon Dec 02, 2019 4:41 pm

Many thanks, i never saw those speed tests results!

According to those results, this switch never operates (switches) at the full 10bge speed? (9536mbps) ?

If i wanted to use the switch with vlan routing, then i still need RouterOS correct? or can i just use the SwOS?
 
mbovenka
just joined
Posts: 19
Joined: Mon Oct 14, 2019 10:14 am

Re: CRS VLAN Routing

Mon Dec 02, 2019 5:02 pm

According to those results, this switch never operates (switches) at the full 10bge speed? (9536mbps) ?

If i wanted to use the switch with vlan routing, then i still need RouterOS correct? or can i just use the SwOS?

It can switch at wire speed or close to it (see the 'Switching' results). It just can't route that fast. And yes, if you want it to route, you need RouterOS; SwOS will not let you do that.
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Fri Dec 06, 2019 6:56 pm

I have come a bit further with this.
# jan/02/1970 13:17:13 by RouterOS 6.45.7
# software id = SP97-FB0N
# model = CRS317-1G-16S+

/interface bridge
add admin-mac=6C:3B:6B:EB:75:43 auto-mac=no name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] comment=esxi03
/interface vlan
add interface=bridge name=VLAN-001-DEFAULT vlan-id=1
add interface=bridge name=VLAN-010-MGMT vlan-id=10
add interface=bridge name=VLAN-2-WIFI vlan-id=2
add interface=bridge name=VLAN-100-SERVER vlan-id=100
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1 pvid=100
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge comment=defconf interface=sfp-sfpplus9
add bridge=bridge comment=defconf interface=sfp-sfpplus10
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp-sfpplus13
add bridge=bridge comment=defconf interface=sfp-sfpplus14
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
/interface bridge vlan
add bridge=bridge tagged=ether1,bridge vlan-ids=10
add bridge=bridge tagged=bridge,ether1 vlan-ids=1
add bridge=bridge tagged=ether1,bridge vlan-ids=100
add bridge=bridge tagged=bridge,ether1 vlan-ids=2
/ip address
add address=192.168.10.5/24 interface=VLAN-010-MGMT network=192.168.10.0
add address=192.168.1.5/24 interface=VLAN-001-DEFAULT network=192.168.1.0
add address=10.10.10.5/24 interface=VLAN-100-SERVER network=10.10.10.0
add address=192.168.2.5/24 interface=VLAN-2-WIFI network=192.168.2.0
/ip route
add distance=1 gateway=192.168.10.1
add disabled=yes distance=1 gateway=10.10.10.1
add disabled=yes distance=1 gateway=192.168.1.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Copenhagen
/system routerboard settings
set boot-os=router-os boot-protocol=dhcp
/system swos
set address-acquisition-mode=static allow-from-vlan=1 static-ip-address=192.168.10.5
However with this setup, when i ping two hosts which are on the CRS switch on different vlans, the CRS doesnt route them it falls back to rt-core-01.. i can see ICMP packets on my core 1 router, and my intention was that the core 2 (CRS) would route them, thus providing a 10gbit routing link

so sfp-sfpplus1 pvid=100 with ip 10.10.10.106 tries to ping sfp-sfpplus2 pvid=1 192.168.1.228 and those boths both have the gatway 10.10.10.5 and 192.168.1.5 respectively, yet the ICMP packets travel outside of this router.

any ideas_
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Fri Dec 06, 2019 10:21 pm

Two things ...

First one: bridge interface has implicit default setting pvid=1 ... so in your case where you use VLAN 1 tagged on bridge you either have to set bridge pvid to some unused value or set frame-types=admit-only-vlan-tagged on /interface bridge

Second: CRS is a switch with some low-capacity L3 (IP) capabilities ... already mentioned above. So inter-VLAN routing will peak at half a Gbps (combined over all interfaces).
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 11:52 am

many thanks for your reply.

I added
add admin-mac=6C:3B:6B:EB:75:43 auto-mac=no frame-types=admit-only-vlan-tagged name=bridge pvid=999 vlan-filtering=yes
to /interface bridge

however i still get ICMP traffic past the CRS, this means the Mikrotik is not routing vlan traffic as it should for some reason. What could i be doing wrong here?

It seems i wont be able to get what i need out of this switch in terms on speed, so as long as the traffic is routed properly on this device and not sent back to the rt-core-01 that would suffice until i can find a device that can do what i need.
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 1:24 pm

What does traceroute between those test hosts show?

What does /ip route print show?

Just noticed: there's missing config detail in /interface bridge vlan: sfp-sfpplus1 should be declared as untagged member of VLAN 100, setting interface with pvid is not sufficient
set [ find vlan-ids=100 ] untagged=sfp-sfpplus1
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 3:38 pm

What does traceroute between those test hosts show?

What does /ip route print show?

Just noticed: there's missing config detail in /interface bridge vlan: sfp-sfpplus1 should be declared as untagged member of VLAN 100, setting interface with pvid is not sufficient
set [ find vlan-ids=100 ] untagged=sfp-sfpplus1

ok now its gone really weird..
 dec/07/2019 14:34:52 by RouterOS 6.45.7
# software id = SP97-FB0N
#
# model = CRS317-1G-16S+
# serial number = 7A1406FA5C7F
/interface bridge
add admin-mac=6C:3B:6B:EB:75:43 auto-mac=no frame-types=admit-only-vlan-tagged name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=sfp-sfpplus1 ] name=sfp-sfpplus1-esxi01
set [ find default-name=sfp-sfpplus2 ] name=sfp-sfpplus2-esxi02
set [ find default-name=sfp-sfpplus3 ] name=sfp-sfpplus3-esxi03
set [ find default-name=sfp-sfpplus4 ] name=sfp-sfpplus4-nas01
/interface vlan
add interface=bridge name=VLAN-001-DEFAULT vlan-id=1
add interface=bridge name=VLAN-010-MGMT vlan-id=10
add interface=bridge name=VLAN-2-WIFI vlan-id=2
add interface=bridge name=VLAN-100-SERVER vlan-id=100
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1-esxi01
add bridge=bridge comment=defconf interface=sfp-sfpplus2-esxi02
add bridge=bridge comment=defconf interface=sfp-sfpplus3-esxi03
add bridge=bridge comment=defconf interface=sfp-sfpplus4-nas01
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge comment=defconf interface=sfp-sfpplus9
add bridge=bridge comment=defconf interface=sfp-sfpplus10
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp-sfpplus13
add bridge=bridge comment=defconf interface=sfp-sfpplus14
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
/interface bridge vlan
add bridge=bridge tagged=ether1,bridge,sfp-sfpplus1-esxi01,sfp-sfpplus2-esxi02,sfp-sfpplus3-esxi03 vlan-ids=10
add bridge=bridge tagged=sfp-sfpplus1-esxi01,bridge,ether1,sfp-sfpplus2-esxi02,sfp-sfpplus3-esxi03 vlan-ids=100
add bridge=bridge tagged=bridge,ether1,sfp-sfpplus1-esxi01,sfp-sfpplus2-esxi02,sfp-sfpplus3-esxi03 vlan-ids=2
add bridge=bridge tagged=bridge,ether1 vlan-ids=1
/ip address
add address=192.168.10.5/24 interface=VLAN-010-MGMT network=192.168.10.0
add address=192.168.1.5/24 interface=VLAN-001-DEFAULT network=192.168.1.0
add address=192.168.2.5/24 interface=VLAN-2-WIFI network=192.168.2.0
add address=10.10.10.5/24 interface=VLAN-100-SERVER network=10.10.10.0
/ip dns
set servers=10.10.10.10
/ip route
add distance=1 gateway=10.10.10.1
add distance=1 gateway=192.168.1.1
add distance=1 gateway=192.168.10.1
add distance=1 gateway=192.168.10.1
add distance=1 gateway=192.168.2.1
/ip service
set telnet disabled=yes
set ftp disabled=yes
set api disabled=yes
set winbox disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Europe/Copenhagen
/system routerboard settings
set boot-os=router-os boot-protocol=dhcp
/system swos
set address-acquisition-mode=static allow-from-vlan=1 static-ip-address=192.168.1.218

With this configuration, (i re-ordered the ports so they make more sense) not only do my vlans route to the wrong router, but when pinging within the same vlan, in this case vlan100 from port spf3 to spf1 it goes through ethernet1, meaning this link is 1gbit max. not the 10gbit it should be

:shock:
[admin@MikroTik] /interface bridge vlan> /ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          10.10.10.1                1
 1   S  0.0.0.0/0                          192.168.1.1               1
 2   S  0.0.0.0/0                          192.168.10.1              1
 3   S  0.0.0.0/0                          192.168.10.1              1
 4   S  0.0.0.0/0                          192.168.2.1               1
 5 ADC  10.10.10.0/24      10.10.10.5      VLAN-100-SERVER           0
 6 ADC  192.168.1.0/24     192.168.1.5     VLAN-001-DEFAULT          0
 7 ADC  192.168.2.0/24     192.168.2.5     VLAN-2-WIFI               0
 8 ADC  192.168.10.0/24    192.168.10.5    VLAN-010-MGMT             0
I think im a bit confused right now as to how these vlans are routed.
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 4:44 pm

I'd first remove all those default routes under /ip route ... for purpose of testing inter-vlan routing you don't need any of them. You'll need one to provide internet access (that's one for all VLANs) but your main router will have to be aware of it ...

If it doesn't start to behave, I'd save exported config (to management PC) and perform full reset (possibly doing netinstall), after that transfer saved config back to the unit. Could be that there's some lost config lurking behind the scenes and it's hard to get rid of it in any other way.
BR,
Metod
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 4:50 pm

... but when pinging within the same vlan, in this case vlan100 from port spf3 to spf1 it goes through ethernet1,

How do you verify it performs this way?
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 6:07 pm

... but when pinging within the same vlan, in this case vlan100 from port spf3 to spf1 it goes through ethernet1,

How do you verify it performs this way?

This was an iperf3 test between two hosts on the same vlan, the speed came to 1gbit and i could see the ether1 port was maxed out.
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 7:11 pm

In this setup, would ethernet1 need to be apart of the bridge? i will eventually replace ethernet1 to a sfp+ port to uplink to the switch.

Here is a diagram to better try and explain what i am doing.

Image

As you can see, when linux01 and linux02 try to use iperf3, it uses all the bandwidth on ether1, which is not what i want. The idea is that all traffic on these hosts gets routed or switched by sw-core-02
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Sat Dec 07, 2019 10:27 pm

If it is a routing issue, then a plain "traceroute" should show it. But limitation of throughput to 1Gbps can be due to many things...
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Mon Dec 09, 2019 7:44 am

If it is a routing issue, then a plain "traceroute" should show it. But limitation of throughput to 1Gbps can be due to many things...
traceroute to 192.168.10.182 (192.168.10.182), 30 hops max, 60 byte packets
 1  10.10.10.5 (10.10.10.5)  0.515 ms  0.374 ms  0.224 ms
 2  192.168.1.1 (192.168.1.1)  0.274 ms  0.237 ms  0.210 ms
 3  192.168.10.182 (192.168.10.182)  0.429 ms  0.467 ms  0.495 ms
Here you can see linux01 trying to reach nas01 (currently set to 192.168.10.182) both have their default gw 10.10.10.5, but then the traffic goes back to 192.168.1.1 .
[admin@sw-core-02] > /ip route print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          192.168.1.1               1
 1 ADC  10.10.10.0/24      10.10.10.5      VLAN-100-SERVER           0
 2 ADC  192.168.1.0/24     192.168.1.214   bridge                    0
 3 ADC  196.168.10.0/24    196.168.10.5    VLAN-10-MGMT              0
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Mon Dec 09, 2019 4:12 pm

When run from CRS (and NAS is running), does this command succeed?
/ping address=192.168.10.182 interface=VLAN-10-MGMT

Nothing in config doesn't strike me as odd ...
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Mon Dec 09, 2019 5:54 pm

When run from CRS (and NAS is running), does this command succeed?
/ping address=192.168.10.182 interface=VLAN-10-MGMT

Nothing in config doesn't strike me as odd ...

Ah there it is, its a timeout when doing that..

[admin@sw-core-02] /interface bridge> /ping address=192.168.10.182 interface=VLAN-10-MGMT
  SEQ HOST                                     SIZE TTL TIME  STATUS
    0 192.168.10.182                                          timeout
    1 192.168.10.182                                          timeout
    2 192.168.10.182                                          timeout
    sent=3 received=0 packet-loss=100%
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Mon Dec 09, 2019 6:27 pm

Does it work if you run ping command but without setting interface?

It still seems to me that config somehow got borken and that the only way out is full configuration reset (you can try with factory reset first).
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Mon Dec 09, 2019 6:36 pm

Does it work if you run ping command but without setting interface?

It still seems to me that config somehow got borken and that the only way out is full configuration reset (you can try with factory reset first).
/ping address=192.168.10.182 works as expected.

I have done a full configuration reset a few time already, and have fallen to the same issue. I have however just done an upgrade to the latest firmware, so i guess i can try that.

I mean if you cannot see anything wrong with my configuration (the latest is attached below) then could it be a defective device?
# dec/09/2019 17:35:37 by RouterOS 6.45.7
# software id = SP97-FB0N
#
# model = CRS317-1G-16S+
# serial number = 7A1406FA5C7F
/interface bridge
add admin-mac=6C:3B:6B:EB:75:43 auto-mac=no name=bridge vlan-filtering=yes
/interface ethernet
set [ find default-name=ether1 ] comment=uplink l2mtu=9000
set [ find default-name=sfp-sfpplus1 ] comment=esxi01 l2mtu=9000
set [ find default-name=sfp-sfpplus2 ] comment=esxi02 l2mtu=9000
set [ find default-name=sfp-sfpplus3 ] comment=esxi03 l2mtu=9000
set [ find default-name=sfp-sfpplus4 ] comment=nas-01 l2mtu=9000
/interface vlan
add interface=bridge name=VLAN-10-MGMT vlan-id=10
add interface=bridge name=VLAN-100-SERVER vlan-id=100
/interface bridge port
add bridge=bridge comment=defconf interface=ether1
add bridge=bridge comment=defconf interface=sfp-sfpplus1
add bridge=bridge comment=defconf interface=sfp-sfpplus2
add bridge=bridge comment=defconf interface=sfp-sfpplus3
add bridge=bridge comment=defconf interface=sfp-sfpplus4 pvid=10
add bridge=bridge comment=defconf interface=sfp-sfpplus5
add bridge=bridge comment=defconf interface=sfp-sfpplus6
add bridge=bridge comment=defconf interface=sfp-sfpplus7
add bridge=bridge comment=defconf interface=sfp-sfpplus8
add bridge=bridge comment=defconf interface=sfp-sfpplus9
add bridge=bridge comment=defconf interface=sfp-sfpplus10
add bridge=bridge comment=defconf interface=sfp-sfpplus11
add bridge=bridge comment=defconf interface=sfp-sfpplus12
add bridge=bridge comment=defconf interface=sfp-sfpplus13
add bridge=bridge comment=defconf interface=sfp-sfpplus14
add bridge=bridge comment=defconf interface=sfp-sfpplus15
add bridge=bridge comment=defconf interface=sfp-sfpplus16
/interface bridge vlan
add bridge=bridge tagged=bridge,ether1,sfp-sfpplus1,sfp-sfpplus3 untagged=sfp-sfpplus4 vlan-ids=10
add bridge=bridge tagged=bridge,ether1,sfp-sfpplus1,sfp-sfpplus3 vlan-ids=100
/ip address
add address=196.168.10.5/24 interface=VLAN-10-MGMT network=196.168.10.0
add address=10.10.10.5/24 interface=VLAN-100-SERVER network=10.10.10.0
/ip dhcp-client
add dhcp-options=hostname,clientid disabled=no interface=bridge
/system clock
set time-zone-name=Europe/Copenhagen
/system identity
set name=sw-core-02
/system routerboard settings
set boot-os=router-os boot-protocol=dhcp
/system swos
set address-acquisition-mode=static allow-from-vlan=1 static-ip-address=192.168.1.218
 
mkx
Forum Guru
Forum Guru
Posts: 3351
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS VLAN Routing

Mon Dec 09, 2019 6:50 pm

I mean if you cannot see anything wrong with my configuration (the latest is attached below) then could it be a defective device?

I highly doubt that device is defective. There have been cases with unexplicable behaviour and netinstall fixed the problem.
BR,
Metod
 
rygo
just joined
Topic Author
Posts: 11
Joined: Sun Dec 01, 2019 4:09 pm

Re: CRS VLAN Routing

Mon Dec 09, 2019 6:57 pm

Ok i am not aware of netinstall.. I will have a read and report back

Who is online

Users browsing this forum: Baidu [Spider] and 23 guests