Community discussions

MikroTik App
 
User avatar
IPANetEngineer
Trainer
Trainer
Topic Author
Posts: 1316
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

CRS 3xx - L3 ASIC performance testing

Mon Oct 12, 2020 8:40 pm

Did some work on testing the L3 performance last week in 7.1beta2 and published it today.

https://stubarea51.net/2020/10/12/mikro ... e-testing/

Image
Global - MikroTik Support & Consulting - English | Español | Serbian | Danish +1 855-645-7684
https://iparchitechs.com/ecosystem/mikr ... consulting mikrotiksupport@iparchitechs.com
 
Dude2048
Member Candidate
Member Candidate
Posts: 129
Joined: Thu Sep 01, 2016 4:04 pm

Re: CRS 3xx - L3 ASIC performance testing

Mon Oct 12, 2020 8:49 pm

Thanx, higher than I expected. Nicely done.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Tue Oct 13, 2020 10:11 am

Regarding small MTU tests (tests #1 and #2), I suppose that the bottleneck is on the packet generator or receiver side, not the CRS317. As you see, PPS (packets per second) value is almost the same in all three cases, and the transfer speed depends purely on packet size. That is a typical case for CPU, where each packet causes an interrupt, which, in turn, adds performance overhead. ASIC doesn't care much about the packet count.
 
User avatar
IPANetEngineer
Trainer
Trainer
Topic Author
Posts: 1316
Joined: Fri Aug 10, 2012 6:46 am
Location: Jackson, MS, USA
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Tue Oct 13, 2020 3:18 pm

Thanks for the feedback...i'll check the hypervisor and see if it's creating a bottleneck somewhere.
Global - MikroTik Support & Consulting - English | Español | Serbian | Danish +1 855-645-7684
https://iparchitechs.com/ecosystem/mikr ... consulting mikrotiksupport@iparchitechs.com
 
MrYan
Member Candidate
Member Candidate
Posts: 115
Joined: Sat Feb 27, 2010 6:13 pm

Re: CRS 3xx - L3 ASIC performance testing

Tue Oct 13, 2020 11:14 pm

That is a typical case for CPU, where each packet causes an interrupt, which, in turn, adds performance overhead. ASIC doesn't care much about the packet count.
This. In fact you did well to get 1 Mpps from a Linux box (Proxmox/KVM) without any tuning. CloudFlare had to put a lot of effort into tuning to get that number - https://blog.cloudflare.com/how-to-rece ... n-packets/
 
morf
Member Candidate
Member Candidate
Posts: 181
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CRS 3xx - L3 ASIC performance testing

Sun Oct 18, 2020 12:05 am

Regarding small MTU tests (tests #1 and #2), I suppose that the bottleneck is on the packet generator or receiver side, not the CRS317. As you see, PPS (packets per second) value is almost the same in all three cases, and the transfer speed depends purely on packet size. That is a typical case for CPU, where each packet causes an interrupt, which, in turn, adds performance overhead. ASIC doesn't care much about the packet count.
+++
 
User avatar
Maggiore81
Member
Member
Posts: 367
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Sun Nov 08, 2020 5:18 pm

Hello to all.
May the CRS with routeros be used as BGP router to forward almost wirespeed packets?
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
morf
Member Candidate
Member Candidate
Posts: 181
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: CRS 3xx - L3 ASIC performance testing

Sun Nov 08, 2020 5:49 pm

I think it's possible. Only if there aren't enough routes to fit in memory.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Mon Nov 09, 2020 8:41 am

CRS with RouterOS can be used as a BGP router unless the number of routes exceeds the hardware memory capabilities.

Refer to "List of supported devices and their limits" table on the link below:
https://wiki.mikrotik.com/wiki/Manual:C ... Offloading
 
User avatar
Maggiore81
Member
Member
Posts: 367
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Mon Nov 09, 2020 10:26 am

On the link it says:
Depending on the complexity of routes in routing table, max HW accelerated route count could change (see table below for min-max supported route count for each hardware). Whole-byte IP prefixes (/8, /16, /24, etc.) occupy less HW space than others (e.g., /22).
If HW route limit is reached new routes will fall back to CPU, except cases when newly added route overlaps with already existing routes processed by hardware. In this case destinations that were processed in hardware will continue to be processed in hardware. The user should choose the device with HW capability large-enough to store all the routes


Yes. I have seen that doc.
Is there a way to raise that limit?
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
mozerd
Member
Member
Posts: 481
Joined: Thu Oct 05, 2017 3:39 pm
Location: Canada
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Mon Nov 09, 2020 3:46 pm

@IPANetEngineer
Very Nice L3 Forwarding test ... hopefully in 2021 the production stable version of ROS7 will be completed.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Mon Nov 09, 2020 4:36 pm

Unfortunately, it is the hardware limitation. There is not enough internal memory in the switch chip to offload the full BGP table. However, if possible, there is an option to limit the incoming BGP route prefixes via
/routing/filter/
Also, we are working on an option to filter out the prefixes for offloading, i.e., to offload routes with potentially the most traffic while the rest gets processed by the CPU.

If the router needs to handle the full BGP table, I suggest looking forward to CCR devices rather than CRS.
 
User avatar
Maggiore81
Member
Member
Posts: 367
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Wed Nov 11, 2020 3:28 pm

Very very interesting.
Using RouterOS we could use BGP to have some internal routes (less than 1000).
we could route them L3 in hardware...
Is something related to fastpath here? Or can we use some firewall filters?
we wont need conntrack or something similar.
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
Maggiore81
Member
Member
Posts: 367
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Wed Nov 11, 2020 3:29 pm

Did some work on testing the L3 performance last week in 7.1beta2 and published it today.

https://stubarea51.net/2020/10/12/mikro ... e-testing/

Image
Hello.
In your article are missing the notes,
I mean in the table of the max number of connections, are notes, but are not in the page.
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Wed Nov 11, 2020 3:39 pm

Very very interesting.
Using RouterOS we could use BGP to have some internal routes (less than 1000).
we could route them L3 in hardware...
Is something related to fastpath here? Or can we use some firewall filters?
we wont need conntrack or something similar.

There are two distinct L3HW modes in RouterOS v7.1beta2:
  • l3hw=yes (a.k.a. full routing or l3-switching) - the entire routing table gets offloaded to the hardware; traffic gets routed entirely by HW; nothing goes though CPU, and therefore, ROS stateful firewall does not work.
  • l3hw=fw - Firewall-compatible routing. Initially, packets go through CPU/Firewall, then Fasttrack connections get offloaded to the hardware. Consider this as a hardware-accelerated L4 stateful firewall. Unfortunately, the number of hardware connections is strictly limited by the capacity of the internal hardware memory.

Please note that we are talking about a stateful firewall here. Stateless firewall still can be set in l3hw=yes mode via switch ACL rules:
/interface/ethernet/switch/rule/
 
User avatar
Maggiore81
Member
Member
Posts: 367
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Wed Nov 11, 2020 3:50 pm

Hello
Perfect.
But the question is:
a) l3hw=yes (a.k.a. full routing or l3-switching) - the entire routing table gets offloaded to the hardware; traffic gets routed entirely by HW; nothing goes though CPU, and therefore, ROS stateful firewall does not work.
In routerOS will be enabled fastpath then?
If we set some rules on the INPUT chain just to protect the router, we lose the hardware feature?

b) l3hw=fw - Firewall-compatible routing. Initially, packets go through CPU/Firewall, then Fasttrack connections get offloaded to the hardware. Consider this as a hardware-accelerated L4 stateful firewall. Unfortunately, the number of hardware connections is strictly limited by the capacity of the internal hardware memory.

Is there a table? I have seen in the link at the first post, but it is not clear what the number means... 3750 connections, really? it is very low...

thank you
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Wed Nov 11, 2020 4:38 pm

In routerOS will be enabled fastpath then?
No, ROS firewall (/ip/firewall) does not work simply because packets never enter CPU.

If we set some rules on the INPUT chain just to protect the router, we lose the hardware feature?
The traffic to the router itself (packet destination IP = router IP; INPUT chain) is unaffected by the l3hw. The firewall stays fully functional here. The same applies to outgoing traffic (OUTPUT chain).
Regarding routed traffic (FORWARD chain, or PRE/POSTROUTING chains for forwarded packets), in the case of l3hw=yes, setting those rules does nothing because the firewall (/ip/firewall) does not get triggered. You need to set l3hw=no or l3hw=fw to make the stateful firewall to work. However, a stateless firewall still is an option via switch ACL rules. For example, you can allow/block specific IP addresses/prefixes or TCP/UDP ports. More info here: https://wiki.mikrotik.com/wiki/Manual:C ... _.28ACL.29

Is there a table? I have seen in the link at the first post, but it is not clear what the number means... 3750 connections, really? it is very low...
Yes, unfortunately, the number of hardware connections is limited. Actually, it is 4500 if used without MPLS. Mikrotik smart offloading algorithm picks the heaviest (traffic-wise) connections for offloading at any given time. Other (slower) connections get processed by the CPU. So the number of connections can be much greater. For instance, we tested CRS317 with 10k connections, and it worked fine.

Please take into account that CRS (Cloud Router Switch) series are more "switch" than a "router". Consider the ability to run an L4 hardware-accelerated firewall more like a bonus feature rather than a common use-case. For heavy routing, please look into the CCR series.

Currently, Mikrotik engineers are working on a "hybrid l3hw mode" which allows running both l3hw=yes + l3hw=fw on the same device. For example, it will allow hardware inter-VLAN routing (with an unlimited number of connections) while running Firewall/NAT on the upstream port(-s).
Last edited by raimondsp on Thu Nov 12, 2020 9:28 am, edited 1 time in total.
Reason: Fixed description of inbound/outbound traffic
 
User avatar
Maggiore81
Member
Member
Posts: 367
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Wed Nov 11, 2020 5:58 pm

Thank you for you explanations.
The idea was to use a CRS to route l3 between interfaces at FAAAAST speed via BGP.
The issue is how can I protect the router itself then ?
Never tried the switch rules...
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
mkx
Forum Guru
Forum Guru
Posts: 5271
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS 3xx - L3 ASIC performance testing

Wed Nov 11, 2020 7:04 pm

@raimondsp: can you kindly compare different modes of operation of l3hw to HW-offloaded L2? I can imagine many parallelisms, but as I don't have any experience with CRS3xx L3 offloading I can't say if those parallelisms are real or imaginary.
BR,
Metod
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Thu Nov 12, 2020 9:39 am

Thank you for you explanations.
The idea was to use a CRS to route l3 between interfaces at FAAAAST speed via BGP.
The issue is how can I protect the router itself then ?
Never tried the switch rules...

I'm so sorry for misleading. INPUT/OUTPUT chains are unaffected by l3hw because the hardware redirects those packets to/from the CPU. The firewall (that is running on the CPU) stays fully functional in such cases. Hence, enabling l3hw does not affect your abilities to protect the router itself.

What I really meant (but originally failed to explain) is that, in the case of l3hw=yes, you cannot enable the firewall on forwarded traffic. For example, to protect a server behind a router.

I edited my original post to avoid confusion.
 
User avatar
Maggiore81
Member
Member
Posts: 367
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: CRS 3xx - L3 ASIC performance testing

Thu Nov 12, 2020 9:41 am

Hello and good morning.
I was not sure about your first claim about the input/output. Thank you very much for claryfing it.
So the CRS can be a full functional BGP router, with hw forwarding, I dont see the traffic passing by, it is not an issue, but I can protect the router itself.
At the moment I use a CCR1036+10G switch with fasttrack, It could be easily replaced with a CRS317 that has all the 10G ports on it.
Dott. Elia Spadoni
---
Network Administrator
MTCNA, MTCRE, MTCTCE, MTCINE, MTCWE, MTCSE
Spadhausen Internet Provider
Ravenna, ITALY
http://www.spadhausen.com
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Thu Nov 12, 2020 10:31 am

@raimondsp: can you kindly compare different modes of operation of l3hw to HW-offloaded L2? I can imagine many parallelisms, but as I don't have any experience with CRS3xx L3 offloading, I can't say if those parallelisms are real or imaginary.

Basically:
  • L2 HW offloading = bridging on the hardware level.
  • L3 HW offloading = routing on the hardware level.


L3HW does not affect L2HW since the L2-forwarded (bridged) traffic is not subject to routing.


Let's look at a basic example: two VLANs configured on a CRS317 bridge:
  • VLAN100 on interfaces sfp-sfpplus1 - sfp-sfpplu4
  • VLAN200 on sfp-sfpplu5 - sfp-sfppl8.
For simplicity, let's name the interfaces SFP1-SFP8.

The device connected to SFP1-SFP4 can communicate with each other on the L2 level because those belong to the same LAN. The same applies to SFP5-SFP8. All CRS3xx devices provide L2 hardware offloading; therefore, expect near to wire-speed performance.

Now imagine that there is a server connected to SFP1 that needs to be accessed by all devices, including VLAN200*. Hosts that belong to different VLAN cannot communicate on the L2 level; therefore, packet forwarding must be escalated to L3, i.e., routing. If the switch does not support L3HW (in case of CRS317: v6, or before v7.1beta1, or l3hw=no), the routing is performed by the CPU, which dramatically decreases the network speed. While SFP2-SFP4 still benefit from wire-speed communication to SFP1 due to L2HW, SFP5-SFP8 are bottlenecked at CPU performance. Enabling L3HW allows inter-VLAN routing to be almost as fast as hardware bridging. Now all SFP1-SFP8 enjoy wire-speed communication.

* Actually, in such a basic example, you can avoid routing by making SFP1 a member of both VLAN100 and VLAN200, but that is not always a solution.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Thu Nov 12, 2020 10:35 am

Hello and good morning.
I was not sure about your first claim about the input/output. Thank you very much for claryfing it.
So the CRS can be a full functional BGP router, with hw forwarding, I dont see the traffic passing by, it is not an issue, but I can protect the router itself.
At the moment I use a CCR1036+10G switch with fasttrack, It could be easily replaced with a CRS317 that has all the 10G ports on it.

Hi there,

I think CRS317 is a perfect solution for your case.
 
mkx
Forum Guru
Forum Guru
Posts: 5271
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS 3xx - L3 ASIC performance testing

Thu Nov 12, 2020 7:21 pm

@raimondsp: can you kindly compare different modes of operation of l3hw to HW-offloaded L2? I can imagine many parallelisms, but as I don't have any experience with CRS3xx L3 offloading, I can't say if those parallelisms are real or imaginary.

Basically:

I was thinking more in direction: if bridge is not offloaded at all, then device can enforce both firewall (if use-ip-firewall=yes) and bridge filters. If bridge is offloaded to HW, then neither can be used and only ACLs (if supported by switch chip) can affect traffic between two interfaces. So I imagine that the first case (no offload) can be compared to l3hw=no and second case (full HW offload) can be compared to l3hw=yes. I guess there is no bridge mode of operation that could be compared to l3hw=fw because bridge can either be fully offloaded or not at all. Or am I mistaken?
BR,
Metod
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Tue Nov 17, 2020 9:48 am

Unlike l3hw option, use-ip-firewall=yes controls only the packets that enter the CPU. use-ip-firewall does not disable L2 hardware offloading. Actually, it is impossible (and does not make sense) to disable L2HW on the switch chip.
 
pubudeux
just joined
Posts: 9
Joined: Sun Oct 25, 2020 7:54 pm

Re: CRS 3xx - L3 ASIC performance testing

Sat Nov 28, 2020 3:59 pm

From the wiki, I understand that L3 hardware offloading is currently only in the CRS317.

Are there plans (and is it possible) for L3 hardware offloading to be enabled on other CRS3XX devices?

I have a CRS328, and thanks for providing these test results.
 
mbovenka
Member Candidate
Member Candidate
Posts: 220
Joined: Mon Oct 14, 2019 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Sat Nov 28, 2020 4:27 pm

From the wiki, I understand that L3 hardware offloading is currently only in the CRS317.

No, in the newer betas more devices are supported:

https://wiki.mikrotik.com/wiki/Manual:C ... Offloading, look for 'List of supported devices and their limits'

The CRS328 isn't among them (yet), though.
 
pubudeux
just joined
Posts: 9
Joined: Sun Oct 25, 2020 7:54 pm

Re: CRS 3xx - L3 ASIC performance testing

Sat Nov 28, 2020 7:28 pm

My question really is - should I expect to eventually get support for L3 hardware switching with my CRS328, or should I plan to find a different solution.

I have time as a homelab user, so if the only thing standing in between that is a software release I can wait, but I know 328 has a different chip than the one on all the currently supported switches for the beta.
 
mkx
Forum Guru
Forum Guru
Posts: 5271
Joined: Thu Mar 03, 2016 10:23 pm

Re: CRS 3xx - L3 ASIC performance testing

Sat Nov 28, 2020 7:36 pm

Recently one of MT support guys wrote that currently they're running feasibility study for supporting L3 switching on CRS328. He explicitly said that nothing is determined yet ... so it may end up with no L3 switching on this device ... and even if it does happen, it may take a while before it gets implemented.
BR,
Metod
 
pubudeux
just joined
Posts: 9
Joined: Sun Oct 25, 2020 7:54 pm

Re: CRS 3xx - L3 ASIC performance testing

Mon Nov 30, 2020 5:42 am

Thanks for that. I've tried searching all over but have not seen anyone from MT mention the CRS328 yet re: hardware L3. In the meantime I am exploring a software solution, maybe VyOS to run on a VM.
 
User avatar
raimondsp
MikroTik Support
MikroTik Support
Posts: 39
Joined: Mon Apr 27, 2020 10:14 am

Re: CRS 3xx - L3 ASIC performance testing

Mon Nov 30, 2020 8:54 am

Recently one of MT support guys wrote that currently they're running feasibility study for supporting L3 switching on CRS328. He explicitly said that nothing is determined yet ... so it may end up with no L3 switching on this device ... and even if it does happen, it may take a while before it gets implemented.
^ this

Who is online

Users browsing this forum: tpb and 9 guests