I’m trying to understand the setup described https://help.mikrotik.com/docs/display/ROS/CRS3xx%2C+CRS5xx%2C+CCR2116%2C+CCR2216+VLANs+with+Bonds as I want to achieve similar with my gear: CRS328-24P-4S+RM, CRS317-1G-16S+RM and RB5009UG+S+IN. I’m running 2.5Gbe Internet fibre connection using the SFP+ port of the RB5009. As per the example, I have set a bond between CRS328 and CRS317 on 2 ports, and that works just fine.
A few things that I want to achieve:
Most of my 10G gear will be connected to CRS317
Leverage CRS317 for Inter-VLAN access/routing
Use the RB5009 only for the NAT with Internet
There are a few things that I don’t understand.
What is going on with ether1? CRS317 also has an ether1 port which isn’t mentioned in the configuration. Why is it set to “Admit all”? I would have expected to get only untagged traffic and use the ether1 port a way to get on the management network.
What is the best practise here? Is it to restrict access to VLAN 99 from a few hosts only or to have a dedicated port to access management network?
For the VLANs on the bridge, only VLAN 99 also has “bridge” on the tagged list. The others don’t (VLAN 10, 20 and 30). Why is that? Is that preventing VLANs to “talk” to each other by default?
If I want to use the CRS317 as a core switch to do the inter-VLAN routing, should I consider it as the "Router’ in the example and playing the L3 hardware features?
Default setting is frame-types=admit-all … so if it’s not changed explicitly according to needs, it’ll remain that way.
Do as you see fit. IMO access to management VLAN should be as restricted as possible but also depends on particular use case.
Bridge is (also) interface which allows ROS to communicate with nerworks passing bridge the switch-like entity. To enable it, bridge port has to be made member of appropriate VLAN(s). See this explanation of different bridge personalities.
If you want to use CRS317 as router, then you definitely want to go L3HW way. Without L3HW this device has shitty routing performance.
For now, I can ping 192.168.99.2 and 192.168.99.3 (management interface of both switches) when my PC is connected in the LAN VLAN (30) on CRS328 (for instance port ether17)
For some reasons though, I cannot access 192.168.10.210 which is connected on sfpplus1 (CRS317). I have 192.168.30.2 as the gateway on my client (PC).
Here are the full configurations (looks pretty much like the example I initially linked):
I disagree with MKX, cannot recommend, in general, a switch for routing.
The CRS317, can route up to about 400Mbps but thats it, so depending upon your internet connectivity???
The RB50009 is designed to router and would be my first choice and then feed one of the switches with the SFP+ port.
You’re talking software routing. @mkx is talking L3HW routing, where everything gets offloaded to the switch chip. The CRS317 is one of the handful devices in MT’s lineup that can do this well, but even then, it isn’t capable of much of what you can do in software routing.
If you need any of that, then yes, the device lands hard on its poorly-padded processing posterior.
That was my understanding of this kind of product, it can do near wire-speed routing with some limitations. I wanted to see if my small setup could fit those requirements or if I was in need of something bigger (like a proper CCR unit).
Does any of you would know why inter-VLAN routing does not work from scratch (not talking about L3HW here)? All my VLANs are onto the same bridge, ports are tagged as they should. Still, I can reach both server and PC from the CRS317 but I cannot ping server to PC through CRS317 and vice-versa.
I checked with my RB5009 currently being used for my Internet access, and I don’t see any particular differences. I remember very well that by default, all VLANs were talking to each other the moment they were added to the bridge.
As for your setups. If 317 is a router, where are the router settings?
Make up your mind!
If the RB5009 is just for internet then you need to have double nat and make up all the subnets on the switch.
The switch will get a private IP as its WAN ( private IP from 5009 default subnet I guess )
If the plan is to have the 5009 create all the vlans then the only vlan you need to identify ( not create subnet ) is the management vlans of which the two switches will get their own IP address from.
In that case I can have a look… Assuming a standard switch setup then,
Get rid of frame types on bridge itself, use frame types on /interface bridge ports
Use off bridge port to do all configuration, makes life easy when bridge burps.
Add ingress filtering to /bridge ports
Assuming the BOND, 15,16 are ties to the CRS354 but where is the port from 5009 ??? sigh…
will use etheport 14 for that… will use 10.100.1000 gig port for off bridge access.
Only tag bridge for management vlan at /interface bridge vlans.
AFAIK, the bridge on CRS317 has an IP in all VLANs which should be enough to enabled inter-VLAN routing (I’m not talking about performance here, just to trying to make it work first).
I have no idea what you are trying to do… but the setup I gave you should enable HW offloading on the bridge, its done. Intervlan routing is something that is foreign to me. Either the unit is a switch or a router in terms of setup… it can or cannot have hw offloading.
Do not confuse yourself, there is no need to identify any vlans other than management vlan on the switch and that is because the switch needs its IP address from that subnet.
The only difference in the setup I would provide for the 354 is that you have an incoming BOND (taking up two SPF+ ports ) as trunk and maybe no other trunk ports.
Plus the IP address will likely by 192.168.99.3/24… also should have an off bridge access for this switch as well.
Frame types on the bridge is to forbid the default VLAN ID =1 traffic if I understood everything right. It is part of the MikroTik VLAN tutorial and I have been using this on my current RB5009 setup.
I learnt this one the hard way when I was setting up the RB5009 initially (I had to wipe it and reconfigure it…)
Ingress filtering is checked by default and does not appear in the configuration file
My point was to setup inter-VLAN routing first before adding the NAT and the rest. I found what my issue was on Windows side and now, inter-VLAN routing works fine (all VLANs can access all VLANs, I will lock that up later on). I changed all VLANs and IPs to make sure there are no overlap with any of my currently used setup. I will change configuration before putting everything live.
I have untagged ports on CRS317 as well (future 10G devices), I need to tag those ports and have some IP addresses set if I want to be able to route between VLANs.
To give a bit more context and background about my setup: I’m running a little homelab at home, mainly built around NUCs right now but I will retire those to get devices that can support 10G NIC. My current Ceph cluster is on 2.5Gbe and consumer drives, it works but it is far from being optimal.
“An image is worth a thousand words”
Here is what I’m trying to achieve right now:
I managed to get the L3HW offloading working on both switches. Just need to remember to disable/reenable L3HW to get modifications taken in (I think the rules are “copied” over to the switch chip when you do that).
I’m now connected to the Internet via the RB5009 (sfp-sfpplus13 on CRS317). I have set the MTUs and I will now try to test with iperf3 to see how much I get performance wise.
Your point is something that I did not realise: double nat. Do you have any ideas how I could prevent that? I could remove the NAT from RB5009 and treat it as firewall/bridge, no?
You have bad info and are not completing the config as provided on the second bridge device.
neighbours discovery, bridge itself still has frame types, IP DNS server wrong etc etc.
The first one acting as a router is a complete abomination mixing non CRS3 and CRS3 switch setup types and MKX politely, via a question, pointed out how your router setup was woefully inadequate.
You are barking up a wrong tree here.
Use the 5009 for routing and DHCP and use the switches for switching is my advice.
I am moving on.
I came here to seek help and see if/how people are using those products. I don’t know much about MikroTik nor network configurations, homelab is a way for me to learn. And to be honest, I don’t feel that I’m one barking right now…