Community discussions

MikroTik App
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

New High Performance Routers ! ?

Sun Oct 06, 2019 10:17 pm

Even CCR 1072 is not enough for us. Many low performance cores are not the way to go and they are easily saturated.
Are there any plans for new high performance routers from mikrotik ?
[r23@noc] /system resource cpu> pr 
 # CPU                                                                                                                   LOAD         IRQ        DISK
 0 cpu0                                                                                                                   28%         28%          0%
 1 cpu1                                                                                                                   44%         43%          0%
 2 cpu2                                                                                                                   22%         22%          0%
 3 cpu3                                                                                                                   27%         27%          0%
 4 cpu4                                                                                                                   24%         24%          0%
 5 cpu5                                                                                                                   61%         59%          0%
 6 cpu6                                                                                                                   59%         59%          0%
 7 cpu7                                                                                                                   34%         34%          0%
 8 cpu8                                                                                                                   29%         29%          0%
 9 cpu9                                                                                                                   42%         42%          0%
10 cpu10                                                                                                                  50%         50%          0%
11 cpu11                                                                                                                   8%          8%          0%
12 cpu12                                                                                                                  44%         44%          0%
13 cpu13                                                                                                                  46%         46%          0%
14 cpu14                                                                                                                  59%         59%          0%
15 cpu15                                                                                                                  43%         43%          0%
16 cpu16                                                                                                                  40%         40%          0%
17 cpu17                                                                                                                  54%         54%          0%
18 cpu18                                                                                                                  50%         50%          0%
19 cpu19                                                                                                                  18%         15%          0%
20 cpu20                                                                                                                  68%         67%          0%
21 cpu21                                                                                                                  38%         38%          0%
22 cpu22                                                                                                                  62%         62%          0%
23 cpu23                                                                                                                  42%         42%          0%
24 cpu24                                                                                                                  35%         35%          0%
25 cpu25                                                                                                                  40%         40%          0%
26 cpu26                                                                                                                  51%         51%          0%
27 cpu27                                                                                                                  36%         36%          0%
28 cpu28                                                                                                                  69%         69%          0%
29 cpu29                                                                                                                  61%         61%          0%
30 cpu30                                                                                                                  62%         62%          0%
31 cpu31                                                                                                                  22%         22%          0%
32 cpu32                                                                                                                   8%          8%          0%
33 cpu33                                                                                                                  11%         11%          0%
34 cpu34                                                                                                                  44%         44%          0%
35 cpu35                                                                                                                  20%         20%          0%
36 cpu36                                                                                                                  14%         14%          0%
37 cpu37                                                                                                                  68%         68%          0%
38 cpu38                                                                                                                  43%         43%          0%
39 cpu39                                                                                                                  44%         44%          0%
40 cpu40                                                                                                                  61%         61%          0%
41 cpu41                                                                                                                  32%         13%          0%
42 cpu42                                                                                                                  36%         36%          0%
43 cpu43                                                                                                                  39%         39%          0%
 
User avatar
xvo
Forum Guru
Forum Guru
Posts: 1237
Joined: Sat Mar 03, 2018 1:12 am
Location: Moscow, Russia

Re: New High Performance Routers ! ?

Sun Oct 06, 2019 11:24 pm

There were some gentle hints some time ago that something is on the way, but no specifics, no timeline.
 
User avatar
krafg
Forum Guru
Forum Guru
Posts: 1020
Joined: Sun Jun 28, 2015 7:36 pm

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 4:43 am

You say that is not enough for you, but you are not using 100% of the CPU. How much traffic are you passing in total?
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 9:32 am

We are passing 9Gbit/s on the uplink. So it makes 18Gbit/s RX/TX traffic as total inside the router all interfaces combined.
Router only does NAT and nothing else. We had to use the "raw" filter to decrease the CPU utilisation. Without "raw" some cores were stuck at %100.
We are not sure what is the next step after ccr1072.
CHR is not stable at all for such traffic. Even it is not stable enough to run only "the dude" on it. Keeps locking up and needs reboot every week.

There is definitely a need for a high performance platform from mikrotik.
 
User avatar
krafg
Forum Guru
Forum Guru
Posts: 1020
Joined: Sun Jun 28, 2015 7:36 pm

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 1:32 pm

18 Gbit/s, when in the web page says that it can pass until 80 Gbit/s. There is something weird. Can you post your export with hide-sensitive option?

Regards.
 
savage
Forum Guru
Forum Guru
Posts: 1262
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 1:41 pm

18 Gbit/s, when in the web page says that it can pass until 80 Gbit/s. There is something weird. Can you post your export with hide-sensitive option?

Regards.
MT's estimates, are extremely optimistic. The tests are done virtually with a blank router, doing absolutely nothing at all - it doesn't represent real world scenarios at all. Feel free to read up a bit on how MT actually conducts the tests, and what specifically, are tested.

9Gbps on a CCR, is quite impressive actually. I've also maxed out these devices in the 7 to 9 Gbps region.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 2:18 pm

Those tests are "standardized tests", like the gas mileage tests on cars.
A test standard is required to be able to compare between different routers or cars, but the results do not represent at all what your actual daily performance will be.

Indeed, that is unfortunate. But it would probably be difficult to define more realistic tests and still do it in an implementation-agnostic way.
You would not want a test definition that by nature works much better on manufacturer X's implementation than on your own, while in actual usage there would be no such difference.

I also sometimes think "gas mileage for cars test should be defined as the amount of gas required to drive on normal roads between two defined points divided by the distance" but in reality that would likely also cause some issues. Same with router performance tests.
 
User avatar
normis
MikroTik Support
MikroTik Support
Posts: 26293
Joined: Fri May 28, 2004 11:04 am
Location: Riga, Latvia

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 2:24 pm

Those tests are described here below the produt, in the Test tab:
https://mikrotik.com/product/CCR1072-1G ... estresults
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11383
Joined: Thu Mar 03, 2016 10:23 pm

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 3:50 pm

Those tests are described here below the produt, in the Test tab:
https://mikrotik.com/product/CCR1072-1G ... estresults

I think that the point of the whole thread is not about the tests, it's about potential new high-end routers. And I think we'd all love to get some clarifications from Mikrotik about that.
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Mon Oct 07, 2019 11:46 pm

@Normis
As I have stated before, we and I am sure most other ISPs as well are maxing out the current CCR products.
Are there any plans for new and more powerful architectures/products ?
 
Michaelcrapse
newbie
Posts: 26
Joined: Wed Oct 25, 2017 9:57 pm

Re: New High Performance Routers ! ?

Tue Oct 08, 2019 10:17 pm

CHR is indeed the only way forward for the forseeable future.
However, you should be able to get 20(40) or so gbps out of your CCR1072, Number 1: how many nat connections do you have. Number 2: don't use masquerade anywhere. You're using fast track, correct?
There are many small things that you can change that will drastically improve the performance of your router. That being said, this seems to be an edge router, are you overloading your router with tasks? Is it also acting as a core router, etc?
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Thu Oct 10, 2019 1:13 pm

We are already getting 18gbit with RAW table populated for our public IPs not participating in conn.track table. Also with fast-track rules.
There is not much to do in terms of configuration.
CCR1072 CPU consumption is %50 with 18gbit/s total throuput + firewall + NAT plus some cores hitting %80.

CHR is super unstable product (at least on vmware). We use it only for "the dude" monitoring and 3 ppptp servers for managment. Total throuput is not even 10mbit/s
And Even in that case, it locks up. Needs reboot every week. It looses its winbox port and ssh accesibility. We reboot it from the vm console to make it work.
we cant put CHR in production at all.

What is left ? Should we go for other brands or mikrotik has a timeframe or any plan for the successor of the CCR products ?
 
User avatar
krafg
Forum Guru
Forum Guru
Posts: 1020
Joined: Sun Jun 28, 2015 7:36 pm

Re: New High Performance Routers ! ?

Thu Oct 10, 2019 1:18 pm

What ROS is running your CCR?

Regards.
 
mikruser
Long time Member
Long time Member
Posts: 578
Joined: Wed Jan 16, 2013 6:28 pm

Re: New High Performance Routers ! ?

Sat Oct 12, 2019 8:58 pm

I am very surprised that Mikrotik does not use hardware NAT'ing.
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: New High Performance Routers ! ?

Mon Oct 14, 2019 2:00 pm

We use it only for "the dude" monitoring and 3 ppptp servers for managment.
i know it will not improve the performance, but you should really move away from pptp, esp if it is used for sensitive stuff, like management.
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: New High Performance Routers ! ?

Mon Oct 14, 2019 4:19 pm

I am very surprised that Mikrotik does not use hardware NAT'ing.
A10, one of the real high performance CGN manufacturers doesn't necessarily use hw NAT either. we have some boxes from them, they handle several 10s of gbps with sw only nat on dual xeon CPUs (their hw appliance). believe or not, NAT - unlike encryption - is not really a CPU intensive process - i mean not more intensive than regular packet forwarding.
in general, the session setup will be always going along the slow path, and once the session is set up, data can flow either through a fastpath-like accelerated sw path (like DDTB/DDNB) or using a dedicated CPU instruction set (hw accelerated nat) which then consumes less resources compared to kernel mode sw path. in exchange you'll limited by the hw-bound nat table size, thus the number of concurrent sessions - but those limits aren't necessarily reached in many cases.

in either way, you should consider that hw-accelerated-nat - if it is available as CPU instruction, like with MT7621 (yes, mmips as in hexr3) - only shows its power when dealing with longer flows with lots of data. with short flows you will not see the big a wow effect. with hw accelerated nat you'll have packet header rewrites done by a single (or few) instruction, sometimes even in cut-through-mode, where the rewrite happens as soon the packet header is received (and the rest of the packet is still in transit). to my best knowledge, RouterOS is using store-and-forward logic only.
on the contrary you'll lose certain high level functions and most likely "nat helpers" - depending on how the code is actually written for specific types of streams (when to switch over to accelerated path).
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: New High Performance Routers ! ?

Mon Oct 14, 2019 7:47 pm


CHR is super unstable product (at least on vmware). We use it only for "the dude" monitoring and 3 ppptp servers for managment. Total throuput is not even 10mbit/s
And Even in that case, it locks up. Needs reboot every week. It looses its winbox port and ssh accesibility. We reboot it from the vm console to make it work.
we cant put CHR in production at all.

I have clients with very stable CHR builds that are doing more than 10 Gbps in production ISP and Data Center networks. If you can only get 10 Mbps, something is wrong in your build.

Even in the testing I did for CHR MUM presentations where ESXi wasn't the best, it still did 5 Gigs of traffic.

https://mum.mikrotik.com/presentations/ ... 562405.pdf
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Tue Oct 15, 2019 12:03 am

We dont use CHR for real world customer traffic. we use it only for our managment traffic which is 10mbits. it freezes and lockes up. maybe dude is making it locking up. I have no idea but it is not stable.

Nevertheless, I really hoped for a CCR successor. I see that it wont be happening at all in the near future.
 
joegoldman
Forum Veteran
Forum Veteran
Posts: 766
Joined: Mon May 27, 2013 2:05 am

Re: New High Performance Routers ! ?

Tue Oct 15, 2019 12:37 am

I see that it wont be happening at all in the near future.
I'd hazard a guess at something by mid-2020, we have QSFP switches now I'm expecting a companion router - and not tile, less cores, more power per core, based on Router OS 7.0 (perhaps even a high powered ARM device given the 7.0beta has been exclusively released on ARM for now)

It wont be too far away.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Tue Oct 15, 2019 8:32 am

There is a X86 Package. So you could go with plain PC hardware. For sure a core i7 will outperform CCR.
 
User avatar
mkx
Forum Guru
Forum Guru
Posts: 11383
Joined: Thu Mar 03, 2016 10:23 pm

Re: New High Performance Routers ! ?

Tue Oct 15, 2019 8:53 am

... based on Router OS 7.0 (perhaps even a high powered ARM device given the 7.0beta has been exclusively released on ARM for now)

There is a X86 Package. So you could go with plain PC hardware. For sure a core i7 will outperform CCR.

Right now ROS V7 beta is available for ARM architecture and as CHR. I wouldn't hold my breathe waiting for native x86 (or x64) package, I can understand MT devs that supporting all the different NIC hardware directly (and storage technologies as well) is huge PITA and leaving that to some VM layer is a very productive approach for them (yeah, sure, it comes with performance degradation but what's worse, 5% lower performance or non-working setup?) ...
 
savage
Forum Guru
Forum Guru
Posts: 1262
Joined: Mon Oct 18, 2004 12:07 am
Location: Cape Town, South Africa
Contact:

Re: New High Performance Routers ! ?

Tue Oct 15, 2019 9:39 am

Can also confirm, I've done some serious traffic under serious loads on a CHR (ESX).

The problem is however, it's not always feasible to put down a x86 host for virtualization. When it's possible though, it's a no-brainer.
 
mikruser
Long time Member
Long time Member
Posts: 578
Joined: Wed Jan 16, 2013 6:28 pm

Re: New High Performance Routers ! ?

Tue Oct 15, 2019 1:18 pm

doneware
NAT - is not really a CPU intensive process

but in real life author writes something else:

doush
Router only does NAT and nothing else.
CCR1072 CPU consumption is %50 with 18gbit/s total throuput + firewall + NAT plus some cores hitting %80.


doneware
using a dedicated CPU instruction set (hw accelerated nat)
It's not hardware nat.
Hardware NAT implemented completely in hardware, work on wire speed, and do not use CPU.
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 12:07 pm

It's not hardware nat.
Hardware NAT implemented completely in hardware, work on wire speed, and do not use CPU.
following your logic, CCRs don't have encryption in HW either, right?
btw, can you please point out any _non_cpu_hardware_element_ in the CCR series routers?

network operations "done in hw" are in many cases just header rewrites using some pre-programmed cache. i.e. no traditional lookup is done using generic CPU operations and header element rewrites using again generic CPU operations, but based on cached entries the ASIC logic or dedicated NP/CPU SIMD instruction copies elements from a simplified lookup, like: src/dst MAC, outgoing interface descriptor and in case of NAT src/dst ip and ports. this is still far more efficient than issuing separate generic CPU instructions (like loading an entry from/to stack, copying from/to memory address, comparison, etc) and essentially the actual instruction logic is stored in the CPU/NP microcode.

a potential realisation of this can be done in modern switching chips (in a processor complex/SoC) - if their microcode supports it - which essentially do the same with mac addresses. still the FIB needs to be programmed in a way that the 'hw accelerated' components expect it.
you can keep referring to things as 'done in hardware' if it is done by the microcode of the CPU and as 'done in software' when the CPU is running kernel or user land code to achieve a certain functionality.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 12:31 pm

It's not hardware nat.
Hardware NAT implemented completely in hardware, work on wire speed, and do not use CPU.
following your logic, CCRs don't have encryption in HW either, right?
btw, can you please point out any _non_cpu_hardware_element_ in the CCR series routers?

network operations "done in hw" are in many cases just header rewrites using some pre-programmed cache. i.e. no traditional lookup is done using generic CPU operations and header element rewrites using again generic CPU operations, but based on cached entries the ASIC logic or dedicated NP/CPU SIMD instruction copies elements from a simplified lookup, like: src/dst MAC, outgoing interface descriptor and in case of NAT src/dst ip and ports. this is still far more efficient than issuing separate generic CPU instructions (like loading an entry from/to stack, copying from/to memory address, comparison, etc) and essentially the actual instruction logic is stored in the CPU/NP microcode.

a potential realisation of this can be done in modern switching chips (in a processor complex/SoC) - if their microcode supports it - which essentially do the same with mac addresses. still the FIB needs to be programmed in a way that the 'hw accelerated' components expect it.
you can keep referring to things as 'done in hardware' if it is done by the microcode of the CPU and as 'done in software' when the CPU is running kernel or user land code to achieve a certain functionality.
+1

CPU ist HW, too :-) If there are enough CPUs which are fast enough there is no need for specific ASICs.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 12:36 pm

The attractive property of MikroTik routers (and switches), its low price, is made possible by the use of standard SoC chips which have only a basic processor and some optimization of encryption.
Everything runs in software, made possible by the Linux system which has a very advanced networking stack, and some MikroTik proprietary patches to enable some things that standard Linux can not do.

Other manufacturers make special ASICs with associative memory and special microcode to do things faster. This is more expensive, both because of chip design/manufacture in lower quantities but also because of more expensive microcode development.
Of course the result is not comparable. Commonly available "L3 switches" can do wirespeed IP routing and make the CRS products look pale.
Routers can do multi-gigabit routing and filtering and still aren't overloaded.

But to know why MikroTik does not do it that way, you only have to look at the price. When you want and can affort such a router, go buy it from Cisco, Juniper etc.
They serve a different market segment.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 12:52 pm

The attractive property of MikroTik routers (and switches), its low price, is made possible by the use of standard SoC chips which have only a basic processor and some optimization of encryption.
Everything runs in software, made possible by the Linux system which has a very advanced networking stack, and some MikroTik proprietary patches to enable some things that standard Linux can not do.

Other manufacturers make special ASICs with associative memory and special microcode to do things faster. This is more expensive, both because of chip design/manufacture in lower quantities but also because of more expensive microcode development.
Of course the result is not comparable. Commonly available "L3 switches" can do wirespeed IP routing and make the CRS products look pale.
Routers can do multi-gigabit routing and filtering and still aren't overloaded.

But to know why MikroTik does not do it that way, you only have to look at the price. When you want and can affort such a router, go buy it from Cisco, Juniper etc.
They serve a different market segment.
"standard SoC chips" is not what I think with Tilera based routers. This is specialized routing HW. So this are not top notch routers but perform at a very high level. You have to buy quite big cisco boxes to do the same. Of course this is no switch. So combine a CCR with a CRS and you get a very high level of performance. We combine CRS317 with CCR1036-8G-2S+. Switching is done by the switch and routing with the router.

With CRS326-24S+2Q+RM there is a Switch with 40G Port. I guess next RoutingHW will have 40G too. So there will be new faster combinations.
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 1:24 pm

With CRS326-24S+2Q+RM there is a Switch with 40G Port. I guess next RoutingHW will have 40G too. So there will be new faster combinations.
i on the other hand would cherish solutions with NBase-T (like the sole crs with nXG ports) so it supports 2.5GbE and 5GbE on a single port rather than using LAG.
and if it is for the big guns, i hope 25/50GE gear will come out soon. or god forgive QSFP28+ based 100GE devices - with breakout to 4x25G :-) port density is a bitch and you might not want to run you ports anyway at top speed but just to overcome the previous step w/o LAG, like 25G for 10-12G traffic. the price difference is not as frightening and until it's in DC scale, the only difference is really the transceiver and the port.

40GE is cool, but it is considered by some as legacy and not because of the less throughput, but more about not flowing with the 25/50/100G thing. for me, seeing 4x10G and 40G on mikrotik gear is amazing as it really comes at an affordable price tag. i've never thought we get here this quick.

still a long way until the above stuff becomes truly mainstream...
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 1:52 pm

"standard SoC chips" is not what I think with Tilera based routers. This is specialized routing HW. So this are not top notch routers but perform at a very high level. You have to buy quite big cisco boxes to do the same. Of course this is no switch. So combine a CCR with a CRS and you get a very high level of performance. We combine CRS317 with CCR1036-8G-2S+. Switching is done by the switch and routing with the router.
Well, Tilera Tile is just a multicore CPU that could be used in applications like massively parallel systems for scientific calculations.
And in the MikroTIk way of doing things, it is useful as a CPU in a software-based router.

But when you buy L3 switches from the known manufacturers in the field (Cisco, HP/Procurve/Aruba etc) you get CRS-like functions at much higher performance because the routing is done by the switch ASIC that operates at wirespeed, not by the CPU (which is only used for management).
However, those L3 switches have less overall functionality than a CRS. E.g. no tunneling, IPsec, etc.
The more advanced models DO have things like RIP/OSPF/BGP routing but of course performed by the management CPU so not for internet BGP.
And they have "enterprise" switching functionality for many years now, which MikroTik is very slowly adding feature by feature. (DHCP/ARP tracking, 802.1x, MAC authentication, LLDP, etc).

Again, they usually are more expensive. Sometimes a little more, sometimes a lot more. But for each task you need to select the matching device. MikroTik routers are fine for many purposes, but when it appears you are pushing the limits it may be time to look at others.
 
User avatar
Cha0s
Forum Guru
Forum Guru
Posts: 1135
Joined: Tue Oct 11, 2005 4:53 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 2:51 pm

MikroTik routers are fine for many purposes, but when it appears you are pushing the limits it may be time to look at others.
Sadly, this is very true.

I would love to see higher end models using ASICs (with the appropriate price tag) to be able to use MikroTik hardware for enterprise solutions.
CCRs are nice, but are nowhere near enterprise level.
 
mbovenka
Member
Member
Posts: 337
Joined: Mon Oct 14, 2019 10:14 am

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 4:05 pm

40GE is cool, but it is considered by some as legacy and not because of the less throughput, but more about not flowing with the 25/50/100G thing. for me, seeing 4x10G and 40G on mikrotik gear is amazing as it really comes at an affordable price tag. i've never thought we get here this quick.
This. A switch with 24x10G & 2x40G ports for $499,=? I see it and *still* don't believe it.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 4:22 pm

It is nice as long as you do not need it to route (or route at most management network or other aux network), and as long as you do not need features like Private VLAN or VXLAN or other similar techniques that you often need where you would want to use such a switch for a little more than plain VLAN switching...
 
mbovenka
Member
Member
Posts: 337
Joined: Mon Oct 14, 2019 10:14 am

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 5:09 pm

It is nice as long as you do not need it to route (or route at most management network or other aux network), and as long as you do not need features like Private VLAN or VXLAN or other similar techniques that you often need where you would want to use such a switch for a little more than plain VLAN switching...

They do private VLAN of a sort. VXLAN? Well, then you can go and buy, say, a Cisco 3850-24XS (same 24xSFP+ & 2xQSFP) which...oops...costs more per port than the entire CRS326-24S+2Q+ does. Does it do a lot more? Sure. But you pay for it too.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 5:34 pm

That is what I mean. People say "switching has never been so cheap" but they forgot to mention they compare apples with oranges.
Sure it is cheap when you need only plain VLAN switching (I would not even know how to to Private VLAN on them) but where in your infrastructure do you require a multi-10gigabit switch that only needs to do plain switching? This of course depends on your network, but I think in many cases you would want such a switch e.g. in a datacenter with many VM hosts, and then such a switch is not very useful.
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 7:12 pm

Sure it is cheap when you need only plain VLAN switching.
modern switch chips used by Mikrotik, like the Marvell Prestera series can do much more in hw assisted manner:
https://www.marvell.com/switching/asset ... 336_pb.pdf

just take 802.1BR port extender: pair this with a router and you have a ton of ports. or VRF support in 'hw', or ipv4/ipv6 forwarding in 'hw'. they even support nvGRE tunnelling.
this is all there, but must be supported by the sw ecosystem. this part is extremely hard. routeros is just like Cisco IOS was back in the early 2000s: absolutely versatile. you can to anything on top of the other. just like it was with the 7200 series routers.

once you go down the 'hw' assisted road, it is full of pitfalls. it can be blazing fast, but with a crapload of limitations, that are very specific to the hw. usually people think that cisco/juniper has no problems, but that is far from true. they lost tremendous amount of flexibility during this 10-15 years. in many cases there's just _one_ way to do a thing. this is scary to me. or some L3 functionality - normally totally unrelated to link layer - is available on ethernet, on dot1q VLAN, but not on QinQ or .1ad. or you can't have labels over GRE tunnels. or some random stuff you can easily do with routeros - because that part is done in sw.

modern network operating systems just program the hw components and try to keep some sort of balance between limitations and the hw capabilities. sometimes they manage to do something new using a neat trick, misusing some present functionality - but that is very very rare.

we sacrificed flexibility on the altars of speed :-)
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 8:35 pm

modern switch chips used by Mikrotik, like the Marvell Prestera series can do much more in hw assisted manner
But that does not bring you much when it is not supported by the software, as you already wrote.
Maybe it can do VLAN filtering on a bridge without falling back to software, but will that still work when you use horizon etc?
Even when you want to run a simple ISP, you will need lots of those functions unless you use PPPoE all the way to a central router.
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Fri Oct 18, 2019 11:01 pm

I think if there is one thing to be offloaded to specific hw, it has to be Connection Tracking. I dont know if it is possible but it is the most resource consuming thing in RouterOS.
Apart from that, I think Mikrotik should ditch this Tilera stuff and come up with new hardware based on x86-64 or ARM (Maybe Neoverse ?).
We need higher clocked CPUs. And a lot of it.
 
r00t
Long time Member
Long time Member
Posts: 672
Joined: Tue Nov 28, 2017 2:14 am

Re: New High Performance Routers ! ?

Sat Oct 19, 2019 3:11 am

Releasing fully featured x64 ROS would be a good start. Not CHR, but native OS, with all recent drivers linux kernel supports. Who cares if it might be big (1GB?), this version would not be for some embedded boards with limited flash or ram. Just use full potential of current hardware, because right now you can either run CHR with overhead of hypervisor or to try to find rare hardware that will work with x86 ROS.
Hoping now that ROS 7 will be available sooner than later, there will be a x64 build that supports all recent hardware. Compared to custom embedded boards, putting together x86 based high performance router is far easier and would be a good next step for Mikrotik.
 
glueck05
newbie
Posts: 37
Joined: Fri Jan 26, 2018 12:49 pm

Re: New High Performance Routers ! ?

Mon Oct 21, 2019 11:02 am

I fully agree with the previous speaker.
We use these Systems:

1) X86_64: http://www.lannerinc.com/products/netwo ... s/nca-5510
2) 8 Port-Copper Port: http://www.lannerinc.com/products/x86-n ... cm-igm801a
3) 4 Port SFP+: https://www.landitec.com/products/x86-n ... 05a-detail

regards,
glueck
 
User avatar
doneware
Trainer
Trainer
Posts: 647
Joined: Mon Oct 08, 2012 8:39 pm
Location: Hungary

Re: New High Performance Routers ! ?

Mon Oct 21, 2019 4:53 pm

Releasing fully featured x64 ROS would be a good start.
there's some x64 support in the current x86 version.
packages.png
resource.png
i use this to run various freebsd/linux VMs on the KVM part of routeros. it's a quite old box:
[bat@cloudtik] > /sys resource pci print 
 # DEVICE   VENDOR                                   NAME                                          IRQ
 0 ff:06.3  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 1 ff:06.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 2 ff:06.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 3 ff:06.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 4 ff:05.3  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 5 ff:05.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 6 ff:05.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 7 ff:05.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 8 ff:04.3  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
 9 ff:04.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
10 ff:04.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
11 ff:04.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
12 ff:03.4  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
13 ff:03.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
14 ff:03.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
15 ff:03.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
16 ff:02.5  Intel Corporation                        Xeon 5500/Core i7 QPI Physical 1 (r...          0
17 ff:02.4  Intel Corporation                        Xeon 5500/Core i7 QPI Link 1 (rev: 5)           0
18 ff:02.1  Intel Corporation                        Xeon 5500/Core i7 QPI Physical 0 (r...          0
19 ff:02.0  Intel Corporation                        Xeon 5500/Core i7 QPI Link 0 (rev: 5)           0
20 ff:00.1  Intel Corporation                        Xeon 5500/Core i7 QuickPath Archite...          0
21 ff:00.0  Intel Corporation                        Xeon 5500/Core i7 QuickPath Archite...          0
22 fe:06.3  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
23 fe:06.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
24 fe:06.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
25 fe:06.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
26 fe:05.3  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
27 fe:05.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
28 fe:05.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
29 fe:05.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
30 fe:04.3  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
31 fe:04.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
32 fe:04.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
33 fe:04.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
34 fe:03.4  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
35 fe:03.2  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
36 fe:03.1  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
37 fe:03.0  Intel Corporation                        Xeon 5500/Core i7 Integrated Memory...          0
38 fe:02.5  Intel Corporation                        Xeon 5500/Core i7 QPI Physical 1 (r...          0
39 fe:02.4  Intel Corporation                        Xeon 5500/Core i7 QPI Link 1 (rev: 5)           0
40 fe:02.1  Intel Corporation                        Xeon 5500/Core i7 QPI Physical 0 (r...          0
41 fe:02.0  Intel Corporation                        Xeon 5500/Core i7 QPI Link 0 (rev: 5)           0
42 fe:00.1  Intel Corporation                        Xeon 5500/Core i7 QuickPath Archite...          0
43 fe:00.0  Intel Corporation                        Xeon 5500/Core i7 QuickPath Archite...          0
44 06:00.0  Matrox Graphics, Inc.                    MGA G200e [Pilot] ServerEngines (SE...         11
45 01:00.0  Intel Corporation                        82574L Gigabit Network Connection (...         11
46 00:1f.3  Intel Corporation                        82801JI (ICH10 Family) SMBus Contro...          9
47 00:1f.2  Intel Corporation                        82801JI (ICH10 Family) SATA AHCI Co...          9
48 00:1f.0  Intel Corporation                        82801JIR (ICH10R) LPC Interface Con...          0
49 00:1e.0  Intel Corporation                        82801 PCI Bridge (rev: 144)                     0
50 00:1d.7  Intel Corporation                        82801JI (ICH10 Family) USB2 EHCI Co...         11
51 00:1d.2  Intel Corporation                        82801JI (ICH10 Family) USB UHCI Con...         11
52 00:1d.1  Intel Corporation                        82801JI (ICH10 Family) USB UHCI Con...         11
53 00:1d.0  Intel Corporation                        82801JI (ICH10 Family) USB UHCI Con...         11
54 00:1c.4  Intel Corporation                        82801JI (ICH10 Family) PCI Express ...         11
55 00:1c.0  Intel Corporation                        82801JI (ICH10 Family) PCI Express ...         11
56 00:1a.7  Intel Corporation                        82801JI (ICH10 Family) USB2 EHCI Co...          5
57 00:1a.2  Intel Corporation                        82801JI (ICH10 Family) USB UHCI Con...          5
58 00:1a.1  Intel Corporation                        82801JI (ICH10 Family) USB UHCI Con...          5
59 00:1a.0  Intel Corporation                        82801JI (ICH10 Family) USB UHCI Con...          5
60 00:19.0  Intel Corporation                        82567LM-2 Gigabit Network Connectio...         11
61 00:16.7  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...          5
62 00:16.6  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...          9
63 00:16.5  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...         10
64 00:16.4  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...         11
65 00:16.3  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...          5
66 00:16.2  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...          9
67 00:16.1  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...         10
68 00:16.0  Intel Corporation                        5520/5500/X58 Chipset QuickData Tec...         11
69 00:15.0  Intel Corporation                        5520/5500/X58 Trusted Execution Tec...          0
70 00:14.3  Intel Corporation                        5520/5500/X58 I/O Hub Throttle Regi...          0
71 00:14.2  Intel Corporation                        5520/5500/X58 I/O Hub Control Statu...          0
72 00:14.1  Intel Corporation                        5520/5500/X58 I/O Hub GPIO and Scra...          0
73 00:14.0  Intel Corporation                        5520/5500/X58 I/O Hub System Manage...          0
74 00:13.0  Intel Corporation                        5520/5500/X58 I/O Hub I/OxAPIC Inte...          0
75 00:11.1  Intel Corporation                        5520/5500 Routing & Protocol Layer ...          0
76 00:11.0  Intel Corporation                        5520/5500 Physical and Link Layer R...          0
77 00:10.1  Intel Corporation                        5520/5500/X58 Routing and Protocol ...          0
78 00:10.0  Intel Corporation                        5520/5500/X58 Physical and Link Lay...          0
79 00:09.0  Intel Corporation                        5520/5500/X58 I/O Hub PCI Express R...         11
80 00:07.0  Intel Corporation                        5520/5500/X58 I/O Hub PCI Express R...         11
81 00:03.0  Intel Corporation                        5520/5500/X58 I/O Hub PCI Express R...         11
82 00:01.0  Intel Corporation                        5520/5500/X58 I/O Hub PCI Express R...         11
83 00:00.0  Intel Corporation                        5500 I/O Hub to ESI Port (rev: 19)             11

[bat@cloudtik] > /sys resource irq print 
Flags: ro - read-only 
 #    IRQ USERS                                                                   CPU ACTIVE-CPU         COUNT
 0 ro   4 serial                                                                 auto                        7
 1      9 acpi                                                                   auto          4             1
 2     16 usb8                                                                   auto          2             0
          usb7                                                            
          usb6                                                            
          usb2                                                            
 3     19 usb5                                                                   auto          3        49 924
          usb4                                                            
          usb3                                                            
          usb1                                                            
 4     66 ahci                                                                   auto          7       159 435
 5     67 ether1                                                                 auto          1       414 288
 6     68 eth1-rx-0                                                              auto          5     7 739 762
 7     69 eth1-tx-0                                                              auto          6     4 255 655
 8     70 ether2                                                                 auto          0             5
You do not have the required permissions to view the files attached to this post.
 
r00t
Long time Member
Long time Member
Posts: 672
Joined: Tue Nov 28, 2017 2:14 am

Re: New High Performance Routers ! ?

Mon Oct 21, 2019 10:24 pm

Main issue is that there is not even official list of hardware supported by x86/x64 ROS. If you have a lot of time and components laying around, you can probably find a combination that will work for you. But it's extremely inconvenient if you are planning to buy a new hardware and you need to know beforehand if it will work or not, before you buy it. Intel devices are more likely to work work then others, but it's all just try and error. And that's just not acceptable in business environment, where you have to specify what new hardware to buy and get it approved etc. For some issues there are workarounds, like booting ROS from USB flash, no need for any SAS drivers. But if you want to use any modern hardware (think QSPF card), you are out of luck, I don't know any such card that works. It just all feels like ROS drivers/modules weren't updated for years... even if the OS itself have seen some updates, supported hardware is still stuck in the past.
Well, will see how ROS 7 plays out...
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Mon Oct 28, 2019 1:18 pm

ROS7 needs decades to be stabilized.
 
Sanity
Member Candidate
Member Candidate
Posts: 198
Joined: Sun Mar 06, 2011 8:51 am

Re: New High Performance Routers ! ?

Wed Oct 30, 2019 10:48 am

> Main issue is that there is not even official list of hardware supported by x86/x64 ROS.

Hyper-V. As logn as you get that running, you can run CHR. That is what I do - and the trests from Mikrotik showing Hyper-V having the most superior network scalabiity compared to VM Ware and KVM.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Wed Oct 30, 2019 11:06 am

Hyper-V introduces yet another software license to worry about and pay for... and an OS that has to be rebooted at least once a month.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Wed Oct 30, 2019 11:36 am

Hyper-V introduces yet another software license to worry about and pay for... and an OS that has to be rebooted at least once a month.
+1

VMWare Hypervisor is free and stable. Running on good HW (We use DELL Rack-Servers) gives troublefree service. There are scripts to backup Files to a NAS so we have complete restartable image backups. A second spare Server to get services online ASAP on failure.

Dont wont to bother with another MS Software.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Wed Oct 30, 2019 1:01 pm

The point is/was that in direct comparisons of RouterOS CHR performance under different hypervisors, Hyper-V comes out the best.
I have no personal experience with that, I don't operate routers where such high performance is required. But like you I regularly use VMware free as well (not with CHR atm, I use CCR1009).
Also no idea if this is due to a difference between Hyper-V and VMware or if there is something in CHR that could be optimized for VMware as it has been for Hyper-V.
(like a driver for a paravirtualized network adapter)
 
guipoletto
Member Candidate
Member Candidate
Posts: 195
Joined: Mon Sep 19, 2011 5:31 am

Re: New High Performance Routers ! ?

Wed Oct 30, 2019 4:57 pm

As far as i remember, mikrotik supports VmXnet3, so it's down to vmware's driver support for the specific NIC people are playing with.
 
karwos
Frequent Visitor
Frequent Visitor
Posts: 96
Joined: Thu Apr 02, 2015 7:28 pm
Location: Poland

Re: New High Performance Routers ! ?

Thu Oct 31, 2019 3:13 am

Dont even think of chr.

viewtopic.php?t=148310

"Hello,

TSO & LRO are disabled on virtual Ethernet drivers since 6.41rc14. Our devs are looking possibility to fix TCP connection offloading issue."

It simply breaks end2end connectivity by reassembling tcp conns.
Lots of retransmission and real world tcp performance is lower than expected.
Im tired more and more of Mikrotik bugs.
This HW and SWs ones.
They dont care about support at all.

And I will tell you why they will NOT fix it. Because VM CPU will be overkilled. They reassemble like 10-15 packets in row and save 10-15x CPU performance.
 
mveselic
just joined
Posts: 8
Joined: Sun Jan 29, 2017 9:14 am

Re: New High Performance Routers ! ?

Thu Oct 31, 2019 4:44 pm

Hyper-V introduces yet another software license to worry about and pay for... and an OS that has to be rebooted at least once a month.
Hyper-V Server is free as well. You pay only if you use Hyper-V role in Windows Server.
https://www.nakivo.com/blog/microsoft-h ... er-v-role/
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Fri Dec 11, 2020 3:55 pm

Even CCR 1072 is not enough for us. Many low performance cores are not the way to go and they are easily saturated.
Are there any plans for new high performance routers from mikrotik ?

Can you please share your configuration and describe in detail wich issues are you encountering?
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: New High Performance Routers ! ?

Fri Dec 11, 2020 8:50 pm

RouterOS is essentially a software based router. Thats a fact.

I also think a lot of Mikrotik users are trying do to much in the same box. Pushing a lot of PPPoE, Queues, tunneling, NAT and connection-tracking is very very heavy and can only be done i software basically. Not sure that a Cisco or Juniper had done a better job there.
 
mducharme
Trainer
Trainer
Posts: 1777
Joined: Tue Jul 19, 2016 6:45 pm
Location: Vancouver, BC, Canada

Re: New High Performance Routers ! ?

Fri Dec 11, 2020 9:06 pm

The CCR2016 should be coming eventually - presumably that is a 16-core ARM64 device. I would hope that we see that in the next year but no official word yet.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Fri Dec 11, 2020 10:23 pm

The problem of course is that it requires a very specific workload to keep 72 cores busy and it does not help that there are tasks that are not multithreaded.
Those people that run BGP on internet (not me, fortunately) have performance issues that can only be solved with faster cores, not with more cores.
(especially on RouterOS v6 where all BGP processing is performed by a single core. but unless you have many more peers than someone using a MikroTik-class router would normally have, even RouterOS v7 is not going to solve that, because it still looks like the load will be distributed over at most npeers+1 cores, right?)

But yes, it could be better to have a couple of boxes each with some dedicated task (maybe even duplicate them) than to spend all your money on a single CCR1072 and make it do everything. Could also be easier when configuring it.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Fri Dec 11, 2020 10:51 pm

The problem of course is that it requires a very specific workload to keep 72 cores busy and it does not help that there are tasks that are not multithreaded.
Those people that run BGP on internet (not me, fortunately) have performance issues that can only be solved with faster cores, not with more cores.
(especially on RouterOS v6 where all BGP processing is performed by a single core. but unless you have many more peers than someone using a MikroTik-class router would normally have, even RouterOS v7 is not going to solve that, because it still looks like the load will be distributed over at most npeers+1 cores, right?)

But yes, it could be better to have a couple of boxes each with some dedicated task (maybe even duplicate them) than to spend all your money on a single CCR1072 and make it do everything. Could also be easier when configuring it.
Yes. We need much higher single core performance. BGP needs 15 minutes now on a 1036. A actual smartphone processor would do the job much faster.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 9:22 am

Well.
I have replaced a single 1036 for core, into two routers:
1x CCR2004 in fastpath to take care of the BGP only.
the 1036 do to all the firewall and eventual NAT needed.
Now it all runs flawlessy
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 9:25 am

Well.
I have replaced a single 1036 for core, into two routers:
1x CCR2004 in fastpath to take care of the BGP only.
the 1036 do to all the firewall and eventual NAT needed.
Now it all runs flawlessy
Full feed? Time for learning all routes?
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 10:48 am

Large ISP often has several platforms for different workloads. They never do everything in one single platform.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 11:16 am

Well.
I have replaced a single 1036 for core, into two routers:
1x CCR2004 in fastpath to take care of the BGP only.
the 1036 do to all the firewall and eventual NAT needed.
Now it all runs flawlessy
Full feed? Time for learning all routes?
Since we use MT, we no longer have full feed but in this particular router we have a total of 1.3M routes from 3 IXP. For everything we dont'have, we receive a default router.
We have a geo-backup with another router. each one with their IXP and their single upstreamer. I wait for the future the CRS317 with layer3 forwarding and bgp.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 12:16 pm

Well.
I have replaced a single 1036 for core, into two routers:
1x CCR2004 in fastpath to take care of the BGP only.
the 1036 do to all the firewall and eventual NAT needed.
Now it all runs flawlessy
Yes, when you are doing BGP with multiple peers on internet and you want to do NAT and stateful firewalling it really is best to use 2 separate routers: one doing only the BGP and with no connection tracking, and then the other which sees a fixed routing to outside (i.e. a default route to a single device) and does the tracking and NAT.
Otherwise you have issues at each route flap.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 12:43 pm

Well.
I have replaced a single 1036 for core, into two routers:
1x CCR2004 in fastpath to take care of the BGP only.
the 1036 do to all the firewall and eventual NAT needed.
Now it all runs flawlessy
Yes, when you are doing BGP with multiple peers on internet and you want to do NAT and stateful firewalling it really is best to use 2 separate routers: one doing only the BGP and with no connection tracking, and then the other which sees a fixed routing to outside (i.e. a default route to a single device) and does the tracking and NAT.
Otherwise you have issues at each route flap.
We are considering replacing our 1036 with a CHR running on a capable machine.

Something like this: https://www.storagereview.com/review/in ... sxi-server with an additional 10G Intel Card.
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 1:00 pm

NUCs are really good for home use and this one seems to have a lot of power but may not be the most suitable choice for a "carrier grade" installation. Pick something with at least double PSU and a sufficient cooling system. And needless to say always have a backup unit at hand.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 1:38 pm

NUCs are really good for home use and this one seems to have a lot of power but may not be the most suitable choice for a "carrier grade" installation. Pick something with at least double PSU and sufficient cooling system. And needless to say always have a backup unit at hand.
This NUC is a more PRO-One (XEON) than the 1036 (old with single PSU) running now. 45W TDP makes it a quite energy efficient system. Housing is a bit bigger for thermal reasons. There are some interesting systems from Supermicro but not available.
We have a second BGP-Uplink at a second site ...
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 1:58 pm

Well I agree that in such a situation it is probably better to go for a "professional rack server" system from companies like HPE, Dell, IBM etc.
When you have no money it could be an idea to buy from a refurbishing company.
 
glueck05
newbie
Posts: 37
Joined: Fri Jan 26, 2018 12:49 pm

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 2:18 pm

We use 5x X86 as BRAS System, each System does 3k PPPoE Users (CPU is at 20%).

We use this Systems with XEON (XEON E5-1650v4):

https://www.landitec.com/products/x86-n ... 10-detail/

And 10G Cards:
https://www.landitec.com/products/x86-n ... 5a-detail/

On Monday i am starting to test (Intel Xeon Silver 4215R):
https://www.landitec.com/products/x86-n ... 20-detail/

10G Cards:
Revsion-A has 10G onboard

40G Cards (i am not sure if Intel XL 710 is supported by Ros 6.x):
https://www.lannerinc.com/products/netw ... cs2-iqm201

100G Card (Will be delivered in 12 weeks):
https://www.lannerinc.com/products/netw ... 2s-mhm202a

regards,
glueck
 
User avatar
Larsa
Forum Guru
Forum Guru
Posts: 1025
Joined: Sat Aug 29, 2015 7:40 pm
Location: The North Pole, Santa's Workshop

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 2:31 pm

This NUC is a more PRO-One (XEON) than the 1036 (old with single PSU) running now. 45W TDP makes it a quite energy efficient system. Housing is a bit bigger for thermal reasons. There are some interesting systems from Supermicro but not available...

It's not just about the xeon procs but rather the entire hardware echo system that needs to be constructed for use at 24/7 operations with critical components like electrolytes, fans, psu, remote kvm (ipmi), etc.

I'd follow pe1chl's advice and buy a refurbished rack server from a well know brand. We've done that several times and you often get plenty of hw for the bucks. Ebay and sometimes Amazon are a pretty good places if you have time to do you own research otherwise there are a lot of hw brokers or refurbishing companies that usually might help out.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Sat Dec 12, 2020 5:52 pm

This NUC is a more PRO-One (XEON) than the 1036 (old with single PSU) running now. 45W TDP makes it a quite energy efficient system. Housing is a bit bigger for thermal reasons. There are some interesting systems from Supermicro but not available...

It's not just about the xeon procs but rather the entire hardware echo system that needs to be constructed for use at 24/7 operations with critical components like electrolytes, fans, psu, remote kvm (ipmi), etc.

I'd follow pe1chl's advice and buy a refurbished rack server from a well know brand. We've done that several times and you often get plenty of hw for the bucks. Ebay and sometimes Amazon are a pretty good places if you have time to do you own research otherwise there are a lot of hw brokers or refurbishing companies that usually might help out.
So then. Why considering CHR for BGP anyway. Going your route I should use a refurbished juniper/cisco for bgp.
I have dell servers. They run just fine. The 320 in the same rack now has 1/5 the performance of this nuc but eats 5 times the energy.
Very many rotating stuff to move air through these narrow housings ...
These nuc pros are no cheap desktop pcs. They are very good designed workhorses. We have the normal nucs as desktops with 0 failures in years.
From Brochure: "Tested and qualified by Intel for 24x7 sustained operation, and backed by a 3-year warranty "
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Mon Dec 14, 2020 5:18 pm

I did not mention using a refurbished juniper/cisco for bgp, but it could be a better idea yes.
What I meant was using a refurbished rackserver instead of a workstation.
Of course a more or less recent server with decent CPU, NOT a Dell R320 or other minimal server.
And it is only a general advice. As always, your mileage may vary. Your A-Brand server with lots of redundancy may go defective where your NUC continues to work.
 
User avatar
StubArea51
Trainer
Trainer
Posts: 1739
Joined: Fri Aug 10, 2012 6:46 am
Location: stubarea51.net
Contact:

Re: New High Performance Routers ! ?

Mon Dec 14, 2020 5:23 pm

Large ISP often has several platforms for different workloads. They never do everything in one single platform.

Even in smaller ISPs, this is how I design. Trying to fit every service into a couple of routers almost always ends up being more complex than splitting workloads out into Internet Edge, Core, Aggregation/BNG, etc.

More devices with simpler configs makes for a more scalable and resilient network design.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Tue Dec 15, 2020 1:44 pm

Well.
I have replaced a single 1036 for core, into two routers:
1x CCR2004 in fastpath to take care of the BGP only.
the 1036 do to all the firewall and eventual NAT needed.
Now it all runs flawlessy
Yes, when you are doing BGP with multiple peers on internet and you want to do NAT and stateful firewalling it really is best to use 2 separate routers: one doing only the BGP and with no connection tracking, and then the other which sees a fixed routing to outside (i.e. a default route to a single device) and does the tracking and NAT.
Otherwise you have issues at each route flap.


It works. It is quite fast in doing BGP convergence.
the "issue" is that the CCR runs worse than planned.
At 3.0 Gb fastpath v4 only traffic, it is at about 50% CPU...

No pratical issues.
I wait the new CCR2016 when it will come.
I have to check for packet loss issue of 2004
Last edited by Maggiore81 on Sun Dec 20, 2020 3:48 pm, edited 1 time in total.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Sun Dec 20, 2020 12:35 am

[/quote]

Full feed? Time for learning all routes?
[/quote]

The issue for us is not the time to load all the routes (quite fast), but the convergence time in case of flap... horribile.
 
morf
Member Candidate
Member Candidate
Posts: 182
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: New High Performance Routers ! ?

Sun Dec 20, 2020 1:32 pm

Full feed? Time for learning all routes?
[/quote]

The issue for us is not the time to load all the routes (quite fast), but the convergence time in case of flap... horribile.
[/quote]

Beta 7 is much better in this regard. I tested.
Have you tried enabling FastTrack Connection and disable Connection Tracking in firewall?
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Sun Dec 20, 2020 2:25 pm

Full feed? Time for learning all routes?
The issue for us is not the time to load all the routes (quite fast), but the convergence time in case of flap... horribile.
[/quote]

Beta 7 is much better in this regard. I tested.
Have you tried enabling FastTrack Connection and disable Connection Tracking in firewall?
[/quote]

For sure 7 Beta is better. But BGP routers are no place for beta versions. Only long-term and then only after a while and weeks between updating BGP Routers. One reboot of a BGP router causes hours of "angry phone call time".
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Sun Dec 20, 2020 3:18 pm

I was in fastpath.
 
morf
Member Candidate
Member Candidate
Posts: 182
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: New High Performance Routers ! ?

Sun Dec 20, 2020 3:44 pm

I was in fastpath.
Please, show a screenshot of IP ->Settings.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Sun Dec 20, 2020 3:47 pm

At the moment i am jot at the pc but i was in fadtpath with the 2004. I made a special configuration to have bgp router only doing bgp. I have now replaced a 2004 with a 1036 to avoid issues of packet loss.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Mon Dec 21, 2020 11:38 am

@morf
It is difficult to believe when I write that I am in fastpath with the bgp router?
You do not have the required permissions to view the files attached to this post.
 
morf
Member Candidate
Member Candidate
Posts: 182
Joined: Tue Jun 21, 2011 5:31 pm
Location: Saint-Petersburg

Re: New High Performance Routers ! ?

Mon Dec 21, 2020 12:06 pm

@morf
It is difficult to believe when I write that I am in fastpath with the bgp router?
i'm sorry, i didn't ask you to take a screenshot to check you out :) Do you have "connection tracking" enabled or disabled in your firewall settings ?
2020-12-21_13-05-27.jpg
You do not have the required permissions to view the files attached to this post.
 
pe1chl
Forum Guru
Forum Guru
Posts: 10186
Joined: Mon Jun 08, 2015 12:09 pm

Re: New High Performance Routers ! ?

Mon Dec 21, 2020 12:08 pm

It should not make that much of a difference when you have a dedicated border router that is not doing anything else than BGP to some peers and forwarding the traffic, and it does not have all kinds of extra functions like a complicated forward firewall.

Remember such a router has two different tasks that can cause overload:
- the BGP protocol itself
- the forwarding of packets according to the route table established by BGP

These two things are completely separate. BGP itself is not at all affected by forwarding or fast path, it is a CPU-bound task which unfortunately can use only a single core in v6 so the performance is limited by the single-core performance.
Forwarding can be handled by multiple cores. The processing overhead of forwarding each packet can be limited by "fast path" but it is not some miracle.
And it will not help anything at all when your problem is BGP performance e.g. in case of route flap.
 
User avatar
Maggiore81
Trainer
Trainer
Posts: 559
Joined: Sun Apr 15, 2012 12:10 pm
Location: Italy
Contact:

Re: New High Performance Routers ! ?

Mon Dec 21, 2020 12:12 pm

Hello @morf :)
the conntrack is to AUTO, and no connections appears on the conntrack.
The fastpath is enabled and I have carefully disabled all the functions that are not needed according to the fastpath wiki page.
however the 2004 made a lot of packet loss and I replaced with a 1036 with no issues.
tomorrow I will put the 1072 in the same conf (fastpath, no firewall and obvious bgp limits) since I need 4 10g ports.
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Mon Dec 21, 2020 1:05 pm

Mikrotik product portfolio somehow got weird.
Mikrotik have switches now which can do HW NAT
Mikrotik have switches that can do HW Routing.
Mikrotik have Routers that CAN NOT do HW NAT or HW Routing but does everything on CPU :)

I think the new CCR 200x series has to have a capable SoC somehow.
CHR is also not an alternative to high performance Routing/NAT.
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: New High Performance Routers ! ?

Mon Dec 21, 2020 1:54 pm

Mikrotik product portfolio somehow got weird.
Mikrotik have switches now which can do HW NAT
Mikrotik have switches that can do HW Routing.
Mikrotik have Routers that CAN NOT do HW NAT or HW Routing but does everything on CPU :)

I think the new CCR 200x series has to have a capable SoC somehow.
CHR is also not an alternative to high performance Routing/NAT.
MT use what's there and dont design own chipsets. Chipsets have different capabilities...
With v7 (once stable) CCR should work ok with bgp. Doing some more stuff in parallel will help a lot.
 
mhugo
Member Candidate
Member Candidate
Posts: 179
Joined: Mon Sep 19, 2005 11:48 am

Re: New High Performance Routers ! ?

Tue Dec 22, 2020 5:05 pm

What we miss are more than 2 highspeed interfaces on the CCR2004s and upcoming 2016.

Two interfaces makes it hard to mesh a core network as only rings can be made.

/Mikael
 
guipoletto
Member Candidate
Member Candidate
Posts: 195
Joined: Mon Sep 19, 2011 5:31 am

Re: New High Performance Routers ! ?

Tue Dec 22, 2020 5:49 pm

From my perspective:

1036 has enough CPU power to deal with the 10GB loads, but lacks a third(or fourth) 10GB port, and has ballancing/scaling problems for some workloads, due to 36 somewhat low performance cores.

2004 has a good IO distribution, an abundance of 10GB ports, very nice! but is a bit underpowered CPU-wise, having only 4 (albeit high/er performance) cores. Good for BGP until you have to do any kind of CPU-intensive malabarisms (read firewall).

For now, we're using them for the roles formerly occupied by the CCR1016-12S-1S+, even though the total forwarding capacity (with firewall,et al) is a bit lower than the 1016. in real scenarios it is probably equivalent, due to easier ballancing between the cores.

we really need something to put in place of our aging 1036's, with enoug CPU power to allow for some flexibility in firewall implementations.
 
PortalNET
Member Candidate
Member Candidate
Posts: 126
Joined: Sun Apr 02, 2017 7:24 pm

Re: New High Performance Routers ! ?

Thu Aug 04, 2022 3:29 pm

We use 5x X86 as BRAS System, each System does 3k PPPoE Users (CPU is at 20%).

We use this Systems with XEON (XEON E5-1650v4):

https://www.landitec.com/products/x86-n ... 10-detail/

And 10G Cards:
https://www.landitec.com/products/x86-n ... 5a-detail/

On Monday i am starting to test (Intel Xeon Silver 4215R):
https://www.landitec.com/products/x86-n ... 20-detail/

10G Cards:
Revsion-A has 10G onboard

40G Cards (i am not sure if Intel XL 710 is supported by Ros 6.x):
https://www.lannerinc.com/products/netw ... cs2-iqm201

100G Card (Will be delivered in 12 weeks):
https://www.lannerinc.com/products/netw ... 2s-mhm202a

regards,
glueck

Hi, i know this is an old thread, but i would love to know what sort of pppoe client speeds you are offering on that specific hardware.. as we are now on the nex level GPON, XG-PON customers with plan speeds higher then 2.5gbps... how are they handling ?
Is there a newer hardware server type running RouterOS.. for 10k pppoe subscribers? as the average fiber speeds now offering in the region by the ISPs and Telecom providers average 300 to 500mbps but some already started xg-pon 10g offerings.
 
mada3k
Long time Member
Long time Member
Posts: 682
Joined: Mon Jul 13, 2015 10:53 am
Location: Sweden

Re: New High Performance Routers ! ?

Sat Aug 06, 2022 12:25 pm

I would recommend ditching PPPoE at such high speeds. It's just unnecessary overhead.
 
doush
Long time Member
Long time Member
Topic Author
Posts: 665
Joined: Thu Jun 04, 2009 3:11 pm

Re: New High Performance Routers ! ?

Sat Aug 06, 2022 12:28 pm

I would recommend ditching PPPoE at such high speeds. It's just unnecessary overhead.
What is the alternative for PPPoE ?
Is DHCP + RADIUS mature enough in Mikrotik to implement ?

Who is online

Users browsing this forum: aoravent and 15 guests