I am very new to Mikrotik and RouterOS. I have purchased on switch CRS326-24S+2Q+RM. When I go to the block diagram of this switch at https://cdn.mikrotik.com/web-assets/product_files/CRS326-24S2Q_230231.png it shows it has got a single switch chip 98DX8332. But when I login to mu switch and switch section it shows me two switch chips.
Switch1: Marvell 98DX8332
Switch2 Atheros 8227
As I understand from the Mikrotik support pages that Marvell 98DX8332 does not support Inter VLAN filtering at the switch level and hence a very week switch chip. I want to use this switch to it’s fullest potential. I have the following doubts if any of you can clear them.
Why are there two switch chips and what is the diagram for the port connectivity.
Should I use the single Switch chip or both (if possible).
If it is better to use a single switch chip which one should I use?
If it is better to use both how i can unleash the full potential of this switc?
Why is there such a week switch chip in this comparable high end switch?
As you can see on the blockdiagram (go to mikrotik.com, click on hardware then switches then locate your model and finally select Downloads & Documentation and you will see the link to the blockdiagram etc) https://cdn.mikrotik.com/web-assets/product_files/CRS326-24S2Q_230231.png the two “switchchips” is the actual switchchip (Marvell 98DX8332) and the other is the mgmt-cpu (Qualcomm QCA9531L) and its network adapter (ether1 in the config, MGMT/BOOT on the device itself) named Atheros 8227.
For proper L2 (and L3) offloading you should only have one bridge configured to which you add all interfaces except ether1 (that MGMT/BOOT interface who is a Atheros 8227 connected to the mgmt-cpu qualcomm QCA9531L).
So to wrap it up and answer your questions:
See above.
See above.
See above.
See above.
The Marvell 98DX8332 is anything but weak. Its good enough to push all interfaces at once at wirespeed (24x10G + 2x40G). The mgmt interface doesnt really need anything faster than 100M which it currently is. Sure it can be handy with 1Gbps if you want to push alot of logs to your management-network but you rarely do that from a switch/rotuer anyway (the CRS520 have 2x10G as mgmt-interfaces).
What Mikrotik have specialized in is to have good enough hardware at a very competitive price. Also you dont need to pay any annual (yearly) fee for support and updates as you need with other vendors.
Comparing Mikrotik to lets say Arista then yes the Arista EOS is much more mature NOS compared to RouterOS (or SWOS) which Mikrotik use. But for Arista to get the same features as the CRS326 you will have to pay approx more than 10x when you buy the device and then add a annual (yearly) fee for support and updates which you dont have to with Mikrotik.
This page does not listed the features of Marvell 98DX8332 switch chip found in CRS326. I count it in the others column category, which says Inter VLAN routing is not supported in this switch chip (VLAN Table). As compared with other Mikrotik switches whose switch chips are listed here, say Atheros 8227, it supports VLAN table. So, I want to know, if this switch chip supports VLAN table which I understand as Inter VLAN routing?
Regarding Mikrotik’s price vs specifications as compared to other well known brands, I know Mikrotik is much cost effective. That’s why I have purchased the Mikrotik switch and I am here in this forum asking questions to better understand how I can configure this switch. Otherwise, I would have been in say Arista’s forum.
Regarding showing two switch chips in CRS326, is still the doubt that remains. But first my main concern is does the CRS326 Marvell Chipset support Inter VLAN Routing with 10G and 40G ports that it has?
Also note that except for the limits of when you cant enable L3-offloading (VRF, MLAG etc) once you do the options are limited when it comes to number of routing entries, ACL’s and whatelse.
So currently I see the CRS series as a L2+ option. Mikrotik never comes anywhere close to lets say Arista in quality and performance (if we talk about the CRS series) but again the price is alot lower.
So if you just need a good enough L2 switch then I would most likely go with Mikrotik rather than Arista.
But if you need wirespeed L3 without limits and have all features enabled at once (and your wallet allows for it) I would go for Arista or similar.
For full performance, configure a single Bridge and put all ports in it, with Bridge VLAN filtering enable. Right?
I just have some VLANs with Inter VLAN Routing. No other requirement.
Correct, single bridge with VLANs … and appropriate routing rules so that this part can be L3HW offloaded.
And, as already mentioned, don’t even think of pushing MGMT ether port into same bridge, use it for OOB management access (or cover it with a piece of silver tape so it won’t bother anybody).
Inter-VLAN routing is just a (not so special) case of routing.
L3HW offload: https://help.mikrotik.com/docs/display/ROS/L3+Hardware+Offloading (I already linked this document). It’s not entirely different than “normal” routing, it’s just a few things have to be set up in a particular way while for “normal routing” it’s possible to do it in a few slightly different ways.
Example of untagged interface (where the untagged vlan is 111 (which towards the bridge becomes tagged internally)):
/interface bridge port add bridge=bridge1 frame-types=admit-only-untagged-and-priority-tagged interface=sfp-sfpplus1 pvid=111
Example of tagged interface (where vlan 333 will be the tagged vlan, pvid 222 (untagged) is only defined here otherwise it will by default be vlan 1 - untagged is used for LACP, LLDP etc):
/interface bridge port add bridge=bridge1 frame-types=admit-only-vlan-tagged interface=sfp-sfpplus2 pvid=222
Example of hybrid interface (accepting both tagged and untagged frames, 444 untagged and 555 tagged):
/interface bridge port add bridge=bridge1 frame-types=admit-all interface=sfp-sfpplus3 pvid=444
And then to add tagged vlans (note that pvid from above will automatically be added as untagged so you dont have to specify them and the frame-types will be if untagged will be let through or not):
Note that we didnt define vlan 222 above since that will be a dummy (blackhole) in our case aka not used.
The problem by exposing the Linux DSA the way Mikrotik does is that the above would more or less on other NOS be written as (and easier to configure and understand/troubleshoot):
int ge 1/0/1
switchport mode access
switchport allowed vlan 111
switchport pvid vlan 111
!
int ge 1/0/2
switchport mode trunk
switchport allowed vlan 333
switchport pvid vlan 222
!
int ge 1/0/3
switchport mode hybrid
switchport hybrid vlan 444 untagged
switchport hybrid vlan 555 tagged
switchport pvid vlan 444
!
Yes, as soon as you have configured an IP-address for the vlan (lets say vlan 111 and vlan 333 from above example) and you didnt disable /ip routing then the Mikrotik will route between the VLANs (aka inter-vlan routing). Make sure to properly enabled L3HW offloading (unless you use MLAG or VRF or such) to get the full performance (otherwise the packets will be pushed through the mgmt-cpu which will drop the total performance down to about 450Mbps):
So it seems that the CRS326-24S-2Q+RM will struggle if I have L3 Inter VLAN Routing for a large network.
Will it be advisable then to use my CRS326 purely as a Layer2 Switch (may be SwOS) with VLANs and push the L3 features to some other Router/Firewall, say OPNsense?
Anyways, the CRS326 will be doing only Inter VLAN routing among VLANs created. All my other L3 functions like NATing, Routing, Firewall, etc is being done on OPNsense server.
My question is how do I make such a setup? Whats will be the better way of Integrating Mikrotik L2 Switch with OPNsense firewall, there by shifting L3 functions to very powerfull OPNsense server.
Will it be better performance wise for a large user network.
Your CRS can do wirespeed routing between different VLANs … the only limitation (when you’re talking about large network) is number of connected hosts (total in all VLANs) … if networks are IPv4 only, then limit is at 16.000 hosts. If networks are IPv6 only, then limit is at 8.000 hosts. If networks are dual-stack (IPv4 and IPv6), then limit is at around 5300 hosts.
All of that assuming simple routing topology (i.e. the only next hop is your Opensense firewall/router).
I don’t think that your Opensense router xan toute at 40Gbps (limit of CRS’ QSFP ports).
The key to above is that CRS is configured properly … i.e. all ports are members of same bridge, VLANs are done properly on the bridge, addresses and routes are properly set up. The document about L3HW has all the needed information.
Ah, just to avoid any confusion: devices in VLANs have to use CRS as their gateway explicitly. CRS won’t “hijack” packets, sent to Opensense in order to be delivered to another VLAN, such packets will be simply switched over to correct port but will still be sent towards Opensense. So essentially CRS becomes core router and Opensense is only a firewall between CRS and intetnet.
If you need firewall between your VLANs, then CRS can do it (to certain extent still wirespeed), but the practical limits are pretty low … so it wouldn’t handle “large network”, it probably could handle a mid-sized network, a couple of hundreds hosts.
Thanks for the valuable information this forum is providing to me, I should have joined this forum before purchasing a CRS326-24S+2Q+RM. Maybe would have purchased a higher model.
I think, I should provide some more information about my use case so that exact and proper help can be sought from all those who can through some light on the setup.
I am a Professor in an Indian University and I am trying to upgrade my existing network setup. Currently there are very old 2 Nos. of Core Switches (HP Chassis Switches) working as the core switches. But the Switches are just 1Gbps Copper and Fiber optic (24 Ports each)
There are 30 odd VLANs in the network for the Departments, Cameras, WLAN, and in future VOIP phones. This will be extended to may be 40 VLANs as the departments and building grow.
There are around 2500 users connected simultaneously through Desktops, Mobiles and Laptops at any given time.
The Internet connectivity is of 1 Gbps which is, very shortly, going to be increased to 10 Gbps.
Currently the 2 Nos. of HP Layer 3 Chassis Switches serve as the backbone and does Inter VLAN routing only. No DHCP, No Firewalling/Routing, ACLs etc.
There is one VLAN (25) which is housing the OPNSense Firewall and connected to the Internet with Public IP addresses. The users on the different VLANs connect to Internet through this Firewall/Router through “Squid Proxy” running on the OPNSense firewall. This Firewall Server is a powerful server with 10Gbps fiber optic ports. Will modify it to support (40 Gbps) as pointed out above. However, as HP Core Switches are only 1Gbps, that is the limitation, I am facing for now.
The traffic between VLANs is not much except for all the users using VLAN 25 for Internet and one VLAN for the WiFi connectivity. The traffic withing departments mostly remains within.
Besides the Internet Service, we host our own Website, Moodle LMS, Email Server, DNS, etc.
We have all IPv4 setup for now. No IPv6. Although, we plan to move to it as and when we feel comfortable.
Now with this background, I want to achieve the following:
Replace the L3 HP Core Switches to something else so to remove the bottleneck of 1Gbps between these core switches and other Distributions switches. Upgrading these core switches to 10 Gbps is currently not possible.
Maybe CRS326 is not suitable for this, so will purchase higher-end switch, than CRS326 for the job, and will use CRS326 purely as the L2 Switch if required. But at present want to learn RouterOS configurations on this CRS Switch so that other higher-end switch configuration becomes easier.
Will put the CRS326 in such a position that it is attached to the OPNSense at one end and all the other other departments switches and Core Switch-II (HP at present) on the other end. In the next step will replace the other Core 2 Switch with a 10G one. The step after that will replace the other departments switches with 10G ones.
I want to keep the Firewall Setup to OPNSense and does not want any L3 features (except Inter VLAN routing) to be done by the core switches even if I purchase the higher-end switch (at present CRS326). This is because OPNsense has a better Firewall features than RouterOS (I may be wrong).
Will CRS326 server the purpose at least for now?
Which higher-end Mikrotik switch will server my process, if nor CRS326?
What will be the proper way to L3 offload my Inter-VLAN routing ?
As, can be seen in my setup, I want to do Inter-VLAN routing without the Router on a Stick setup. As all the tutorials talk about Router-On-a-Stick configuration, I want a document which can help me in doing Inter-VLAN routing in the CRS326 with proper L3 Hardware offloading.
Lastly, I don’t have the command-line familiarity with the RouterOS. If possible, I want to do this initially with the Winbox GUI, and then when I get more comfortable with the command-line, will do the finer details with the command line. Is it possible?
Just to confirm I understand this point correctly, do you have in mind that the existing HP switches cannot be upgraded to 10 Gbps?
All the models of Mikrotik switches that support L3 routing in hardware behave the same in terms that the decisive functions (as in firewall filtering), i.e. those that require matching of L3 and L4 fields against some criteria, are executed in software, and the routing itself (that only takes the destination IP address into account) is then offloaded to hardware if possible. So no “higher model” within the Mikrotik product line will give you a qualitative leap, as in “more L3 and above functionality in hardware compared to other L3HW models”; the CRS models with L3HW “only” differ from each other in the bandwidth of the interfaces and of the fabric and also in the capacity of the L3 routing tables.
Unless you want to policy the routing between VLANs using some kind of access lists (who is allowed to talk to whom), only the limits as @mkx has summarized play some role. Without the external router, you configure the CRS326 like any other Mikrotik router, which at the very basic level just means that you set up the single bridge with vlan filtering, create internal access interfaces to the individual VLANs, and attach IP addresses in the corresponding subnets to those interfaces. “The bear will take care about the rest” if you know the dark joke - by adding an address in a subnet to a local interface, a route to that subnet is automatically added to the routing table, and L3 HW offload is allowed on it, like in the following example:
So the reason why there is no specific document for local routing among VLANs is probably that the configuration is this simple. Of course, the routes on the hosts in those VLANs/subnets must have the own addresses of the CRS set as their gateway addresses in order that the CRS would act as a router for them.
Whatever you can do using the command line interface can be achieved also using Winbox and vice versa, except for very fresh functionalities that may be, like with most other software products, only available via command line for a few releases. The visualisation of the structure of the configuration database differs in a few places but it’s nothing really important. The advantage of the command line is mostly speed (unless you are really fast with the mouse) and higher information density in bits per unit of screen area, better possibilities of filtering including in logs… but again, for this simple configuration, you are not supposed to get into a situation where this would be of any relevance.
On a philosophical note, I actually think that the Winbox GUI is in a way a missed opportunity because although it uses a graphical mode interface, it does not visualize the various topologies any better than the command line one, it’s still a bunch of tables, just nicer than the text ones.
These HP switches are Chassis switches with 1Gbps cards for Fiber optic and 1 Gbps cards for copper connectivity. These were sufficient quite sometime back. Although the Supervisory Engine supports 10G, but I am not able to get 10G SFP+ cards for these switches here in India. Maybe the model is old enough.
I am very new to the Mikrotik world with RouterOS. Windox is preferred over CLI only for now, as I fail to understand the command syntax, as I have not gone through it.
One more thing, If someone can clear my doubts. If I take, few server grade machines, add some 10G and 40G cards, Install Debian/VyOS/RouterOS onto it and make my own Core Switch. Will this perform well, if I choose the right hardware. This way I can create as many core switches and scale as an when needed. Just an idea that crept my mind. Please throw some light on this.
All measurements are based upon TCP traffic unless stated otherwise. Total Firewall Throughput is calculated based on maximum PPS and standard 1500 byte packets. Maximum PPS is measured using 500 byte packets. IPS performance is measured using ET Open and standard 1500 byte package size (using Suricata version 6.0.12 / Netmap version 14). SSL VPN is measured using AES256GCM16+SHA512. Concurrent sessions are based upon memory available, where one state consumes 1KB of memory and 1GB of memory is reserved for system tasks. Latency is measured as an average over 60 seconds.
Also that particular model uses an older embedded EPYC.
If you want a monster grab a AMD EPYC Genoa F-series cpu with 12 memory channels populated along with Mellanox 400G NICs
This is a good read on 100G and beyond networking on x86 (and ARM) platforms:
Regarding winbox there is also the webbased edition named webfig which I prefer over installing a 3rd party software on the mgmt-computer.
This way you only need a webbrowser to connect to your Mikrotik and once you get the grip of it start using SSH (and consolebased access) aswell. When you login through webfig there is a terminal available too but using SSH is a smoother experience (specially if you try to copy/paste stuff).
That is also how I got used to and learned the Mikrotik syntax. Started with webfig to get a headstart and then went over to SSH/console to find out the actual commands and now only use webfig if I want to have a quick look of statistics (like when you go to interfaces that view is updated every other second to display current pps and bps etc) while all configuration I do through SSH/console.
To me the CRS series are nice L2+ switches that is you can offload L3 but there are limitations to that. You can use CRR if you need fullblown routing (which basically compared to the CRS series have more powerful mgmt-cpu to also be able to push packets through software when/if L3 offloading isnt available).
There are of course other “cheap” competitors to Mikrotik but they will also have other cons. Such as getting gear from FS.com (made in China) who now work together with Pica8 and use PicOS as NOS (meaning additional licensing fees even if this is cheaper than Cisco/HPE/Aruba/Juniper/Arista).
Buying new equipment today (of course depending on wallet size) would most likely in my case end up with at least SFP28 (25G) as backend which is almost the same price as SFP+ (10G) these days. And then as access depending on usecase use either RJ45 (1Gbps) or SFP/SFP+ to get 1 or 10G. IMHO avoid the 2.5G and 5G through NBASE-T who seems to have all kind of issues (where one end autoneg to one speed and the other to another speed resulting in no traffic can pass at all).
