What do you think is faster at BGP, an RB4011 or a CCR1009? By how much?


https://youtu.be/yxLWbUoFE-M

Thank you for the benchmark. The outcome of this must literally feel like a punch in the face for those business customers who bought the fairly expensive CCR1036 or CCR1072 models in the past. The new RB4011 seems to outperform the CCR-series in BGP convergence time - and that for just 200 bucks!

@MT: Multi-threaded BGP must be implemented more urgently than ever!!! Please make this your top priority for the next ROS releases. Why someone would need a 9/16/36/72-core Edge-Router when the BGP implementation can't even utilize the hardware's full potential?

Thank you for testing.

A15:

https://www.arm.com/files/pdf/AT-Explor ... ex-A15.pdf

There's some basic TILE arch info here:

https://en.wikipedia.org/wiki/TILE64

A TILE core seems to be very simple. A15 looks very complex.

The outcome of this must literally feel like a punch in the face for those business customers who bought the fairly expensive CCR1036 or CCR1072 models in the past.

Not at all. Things improve over time. The fact that it was released, the pricing level, this is a complement to using the MikroTik platform.

Things improve over time.

Improvement needs time, I agree, but many of the issues related to the routing-engine are known since at least 2012, the year in which the CCR-series and 6.x were released. Since then we've been told multiple times to just be patient and wait for ROSv7. Now it's 2018 and still no significant improvements to the routing-engine have been made, no multi-threading thus poor convergence time. Buying a CCR feels like buying a Porsche while living inside Vatican City: not worth it because you can't even drive faster than 20mph. Instead of throwing new hardware onto the market every few months, improving the existing software to better utilize the current hardware models would be the way to go.

Newer hardware is still needed because Mikrotik does not fit well all basic needs.

Newer hardware is still needed because Mikrotik does not fit well all basic needs.

Of course it is, but throwing new hardware on the market still won't fix any of the existing software-related issues and limitations.

Thank you for the benchmark. The outcome of this must literally feel like a punch in the face for those business customers who bought the fairly expensive CCR1036 or CCR1072 models in the past. The new RB4011 seems to outperform the CCR-series in BGP convergence time - and that for just 200 bucks!

@MT: Multi-threaded BGP must be implemented more urgently than ever!!! Please make this your top priority for the next ROS releases. Why someone would need a 9/16/36/72-core Edge-Router when the BGP implementation can't even utilize the hardware's full potential?

Stop with the multithreaded BGP.

https://m.facebook.com/story.php?story_ ... 4744771319

Stop with the multithreaded BGP.
https://m.facebook.com/story.php?story_ ... 4744771319

Ok then what about a new CCR series with beefy x86 quad/hexa/octa/deca-cores

Thanks this was great guys.Now if they could add some RAM , redundant PSU's and some extra SFP+ slots I can finally junk these CCR's

Stop with the multithreaded BGP.

https://m.facebook.com/story.php?story_ ... 4744771319

Not going to stop at all. BGP processing and FIB updates definitely need to become more multithreaded. I have only 1 million routes in one of my CCR1036's, thats only a single full table and some IX routes. Even at that level it takes 3-5 minutes to make it into the RIB and another 3-5 minutes before its added to the FIB. This is whether I'm adding a static route or waiting for a BGP peer to converge. Even small BGP peers i.e. customer ones with 1-5 routes still take 5-10 minutes for BGP convergence to happen. This is vastly different to single threaded BGP on Cisco, Juniper etc because they are more highly optimised and likely faster cores (and less of them) the TILE architecture has a quantity over quality approach (36 cores, 72 cores etc) so really needs to play to its strengths and be able to use the many cores to do these updates.

If the workload could be cut out to 72 threads we could be looking at <1 minute converge / update time.

Not going to stop at all. BGP processing and FIB updates definitely need to become more multithreaded. I have only 1 million routes in one of my CCR1036's, thats only a single full table and some IX routes. Even at that level it takes 3-5 minutes to make it into the RIB and another 3-5 minutes before its added to the FIB. This is whether I'm adding a static route or waiting for a BGP peer to converge. Even small BGP peers i.e. customer ones with 1-5 routes still take 5-10 minutes for BGP convergence to happen. This is vastly different to single threaded BGP on Cisco, Juniper etc because they are more highly optimised and likely faster cores (and less of them) the TILE architecture has a quantity over quality approach (36 cores, 72 cores etc) so really needs to play to its strengths and be able to use the many cores to do these updates.

If the workload could be cut out to 72 threads we could be looking at <1 minute converge / update time.

Didn't take the time to read the link, I take it?

Is it even possible to do table updates in paralel mode without causing some other issues? Why would EVERY manufacturer end up with single-threaded BGP?
I saw couple of research papers with experimental implementations of multithreaded BGP, but I am unaware of real implementation by any manufacturer.

Despite the fact I do not deal with BGP in my routers, I support your pursue of speed. However, if the request goes against nature of task, it simply can't be fulfilled.

Not going to stop at all. BGP processing and FIB updates definitely need to become more multithreaded. I have only 1 million routes in one of my CCR1036's, thats only a single full table and some IX routes. Even at that level it takes 3-5 minutes to make it into the RIB and another 3-5 minutes before its added to the FIB. This is whether I'm adding a static route or waiting for a BGP peer to converge. Even small BGP peers i.e. customer ones with 1-5 routes still take 5-10 minutes for BGP convergence to happen. This is vastly different to single threaded BGP on Cisco, Juniper etc because they are more highly optimised and likely faster cores (and less of them) the TILE architecture has a quantity over quality approach (36 cores, 72 cores etc) so really needs to play to its strengths and be able to use the many cores to do these updates.

If the workload could be cut out to 72 threads we could be looking at <1 minute converge / update time.

Didn't take the time to read the link, I take it?

Some people don't click on facebook.

Those without facebook. Actually Andrew has written it correctly. Here is what he says there:

My new response to people saying Mikrotik BGP being single threaded is causing them performance issues:
A few points:
1) Control Plane ≠ Forwarding Plane
While the processing of routing updates happens in a single thread, packet forwarding (the actual movement of your packets) is already multi-core capable.

2) BGP is single threaded on Cisco IOS, IOS-XE and IOS-XR as well as on Juniper JunOS and Nokia SRos
The processing of routing updates, and in particular the application of route-filters needs to happen in a &quot;run to completion&quot; matter. That means that running it across multiple threads will result in unpredictable results. This is why you cant just &quot;make BGP multi-threaded&quot;. What can be done is to split each protocol to a separate process with a central &quot;conductor&quot; process, this is what Quagga/FRR do with the zebrad process, and what JunOS does with RPD.

Moral of the story...

We hope that Mikrotik is going to update the CCR line to support ARM processors. Tilera has had its day. At least we will get an immediate benfit in raw CPU power for the likes of BGP, firewall, routing etc..
I can only then presume that v7 or whatever you want to call it will be easier to code and implement as an industry standard CPU architecture is being used rather than a obscure Tilera architecture.

There was a thread of at least 6 months ago or more of what ports we would like to see on new "100Gbps" routers. Like I have being saying this is all conjecture without a roadmap from Mikrotik.

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Awesome! Please consider a new CCR with ARM, 12G-4S+ and redudant PSUs.
Would be ideal for smaller environments where you have fiber uplinks and access with copper.

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Good Looking forward to them.

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

That is extremely good news.

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Please do some CCR with ARM or x86

Tilera has had its day.

It is now Mellanox BlueField based on ARM.

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Awesome! Please consider a new CCR with ARM, 12G-4S+ and redudant PSUs.
Would be ideal for smaller environments where you have fiber uplinks and access with copper.

I would also love this --^ ^-- 10-12G eth ports and 4sfp+'s would be AMAZING! With a quad core Arm or equivalent that can handle core functions like BGP properly!

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Awesome! Please consider a new CCR with ARM, 12G-4S+ and redudant PSUs.
Would be ideal for smaller environments where you have fiber uplinks and access with copper.

I would also love this --^ ^-- 10-12G eth ports and 4sfp+'s would be AMAZING! With a quad core Arm or equivalent that can handle core functions like BGP properly!

BlueField goes up to 16 A72 cores. It has PCIe 4.0 and 100GbE/IB.

http://www.mellanox.com/related-docs/np ... ld_SoC.pdf

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Awesome! Please consider a new CCR with ARM, 12G-4S+ and redudant PSUs.
Would be ideal for smaller environments where you have fiber uplinks and access with copper.

I would also love this --^ ^-- 10-12G eth ports and 4sfp+'s would be AMAZING! With a quad core Arm or equivalent that can handle core functions like BGP properly!

BlueField goes up to 16 A72 cores. It has PCIe 4.0 and 100GbE/IB.

http://www.mellanox.com/related-docs/np ... ld_SoC.pdf

That's cute. A fit replacement for the aging Tilera CPUs. We can only hope.

BlueField goes up to 16 A72 cores. It has PCIe 4.0 and 100GbE/IB.

http://www.mellanox.com/related-docs/np ... ld_SoC.pdf

That SoC indeed looks really nice.

It seems more appropriate for storage systems.

It seems more appropriate for storage systems.

Mellanox lists this new product line in the same category as Tile.

Thanks for doing the testing MIke! I'm looking forward to putting a 4011 in our lab and benchmarking it against a hardware router.

I'm excited about where MikroTik is headed with more ARM based routers

It seems more appropriate for storage systems.

Mellanox lists this new product line in the same category as Tile.

Are you sure?

It seems more appropriate for storage systems.

Mellanox lists this new product line in the same category as Tile.

Are you sure?

Mellanox CPU differences.PNG

They constitute the multicore CPU category in the navigation.

It does not mean there will not be other SoC's with more network interfaces, although I would not be surprised if Mellanox just expected OEMs to hang switch chips.

I hope it's a case of "once burned twice shy" with the choice of chipset for the CCR replacement. Tilera while technically very interesting turned out to be a bit of a lemon.

I want to see Mikrotik use the Cavium Octeon TX processors in their CCR replacement.

Cavium and Marvell are big supporters of OpenFastPath and OpenDataPlane making it easy to develop software that can make use of hardware offloads/TCAM's on platforms that support it, and kernel path on platforms that don't.
Marvell/Cavium have a full portfolio of SoC's to suit low end to high end routers. On top of this, they are ARM64 processors and have various accelerators specifically for crypto and packet forwarding.

Have been saying for a long time there is room for a CCR with a quad core and the 4011 is close to making it (that rack mount though).

Almost like the CCR line went AMD mentality (more cores are better) than the Intel way of faster better cores.

Excited about this new generation of CCR

Mellanox basically finished the EZchip design integrating their stuff, it is not obvious where they will take it going forward.

Tile-Mx was going to have A53. It seems 16 A72 would be equivalent to 40 A53.

Cavium was acquired by Marvell.

NXP also has a 16x A72 processor with PCIe 4.0 and 100GbE:

https://www.nxp.com/products/processors ... or:LX2160A

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

Awesome! Please consider a new CCR with ARM, 12G-4S+ and redudant PSUs.
Would be ideal for smaller environments where you have fiber uplinks and access with copper.

I would also love this --^ ^-- 10-12G eth ports and 4sfp+'s would be AMAZING! With a quad core Arm or equivalent that can handle core functions like BGP properly!

BlueField goes up to 16 A72 cores. It has PCIe 4.0 and 100GbE/IB.

http://www.mellanox.com/related-docs/np ... ld_SoC.pdf

surely 16cores A72, A73, A75 at 2.8ghz like snapdragon 845 will beat tilera 72 core at 1000mhz, and the benefits of better single thread performance will be huge

Mellanox basically finished the EZchip design integrating their stuff, it is not obvious where they will take it going forward.

Tile-Mx was going to have A53. It seems 16 A72 would be equivalent to 40 A53.

Cavium was acquired by Marvell.

NXP also has a 16x A72 processor with PCIe 4.0 and 100GbE:

https://www.nxp.com/products/processors ... or:LX2160A

better to go with A72 73 75 out of order cores (like A15 we see on rb4011/1100ahx4) to improve single threaded performance, at least at 2.8ghz

the experience is learned, it is easy to reach a point where it is useless to have a large number of light cores, it is better to have fewer cores but to be heavier

some body have tried overclocking the rb1100ahx4 or rb4011?

Mellanox basically finished the EZchip design integrating their stuff, it is not obvious where they will take it going forward.

Tile-Mx was going to have A53. It seems 16 A72 would be equivalent to 40 A53.

Cavium was acquired by Marvell.

NXP also has a 16x A72 processor with PCIe 4.0 and 100GbE:

https://www.nxp.com/products/processors ... or:LX2160A

better to go with A72 73 75 out of order cores (like A15 we see on rb4011/1100ahx4) to improve single threaded performance, at least at 2.8ghz

the experience is learned, it is easy to reach a point where it is useless to have a large number of light cores, it is better to have fewer cores but to be heavier

Note that Tile-Mx was supposed to start from the top with 100x A53 cores, which would be more powerful than 16x A72. Mellanox kept the 28nm to avoid a more major redesign.

A 1100AHx4 with more memory and sfp+ interfaces would be great.

Mellanox basically finished the EZchip design integrating their stuff, it is not obvious where they will take it going forward.

Tile-Mx was going to have A53. It seems 16 A72 would be equivalent to 40 A53.

Cavium was acquired by Marvell.

NXP also has a 16x A72 processor with PCIe 4.0 and 100GbE:

https://www.nxp.com/products/processors ... or:LX2160A

better to go with A72 73 75 out of order cores (like A15 we see on rb4011/1100ahx4) to improve single threaded performance, at least at 2.8ghz

the experience is learned, it is easy to reach a point where it is useless to have a large number of light cores, it is better to have fewer cores but to be heavier

Note that Tile-Mx was supposed to start from the top with 100x A53 cores, which would be more powerful than 16x A72. Mellanox kept the 28nm to avoid a more major redesign.

surely in some cases many ligth cores like A53 can be helpfull specially if runing at 1.8ghz vs tilegx 1.0 and 1.2 ghz, in any case the major advantage comes from clock

but is very frequent specially in ccr1072 (72 cores) getting limited by single core performance with a bunch of cores working while 30 or 40 cores are idling

because that is better to seek heavy cores

this topic examples that, rb4011 whit half the cores than the ccr1009 gives him a beating in single core performance, and i am confident in some other real world scenarios rb4011 can beat ccr1009

but why??

because rb4011 have A15 "heavy" cores, what do I mean with "heavy"? i mean an out-of-order superscalar pipeline core

ligth cores like tilera gx and A53 are superscalar, in-order execution pipeline cores

whats the difference?? in some cases 2.x the performance at the same clock

because that we need some A72,73,75 cores runing at 2.8 ghz to get almost 6.0x the tilera gx 1.0ghz single core performance or 3.0x the A53 at 1.8ghz single core performance

is not a trivial thing, we are talking between 3.0x to 6.0 x advantages

that explain the success of rb4011, a success that is just beginning

A 1100AHx4 with more memory and sfp+ interfaces would be great.

yes that's the only way to make it a valid choice, paying 100us only for dual power-supply losing sfp+ and 512mb of nand does not make sense

rb1100ahx4 with 2gb of ram and spf+ and the same 512mb nand of 4011 at 300us will be nice to see and a valid proposal

like a 4011 but with a more serious envelope to support heavier implementations

Mellanox basically finished the EZchip design integrating their stuff, it is not obvious where they will take it going forward.

Tile-Mx was going to have A53. It seems 16 A72 would be equivalent to 40 A53.

Cavium was acquired by Marvell.

NXP also has a 16x A72 processor with PCIe 4.0 and 100GbE:

https://www.nxp.com/products/processors ... or:LX2160A

This is an interesting SOC, but they have limited software support (like Tilera) and also do not have as wide a portfolio of SOC's as Marvell.

With Marvell you could use the Armada 7040 with a few Mochi ethernet SB's and create an even better RB4011 type product (no switch chip), the 8040 for a faster RB1100AHx4 type product, or you could use the Cavium Octeon TX's for a CCR type product. All with the same code base...

Now it seems there are two companies interested in buying Mellanox.

That would be the third increasinggly bigger gulp of TILE technology.

Yes, we are aware of this peculiarity and we are working also on new routers that have higher power per core, not just many cores.

The Tile GX series consists of several models with different amount of CPU cores. It would be great if the new router series would start with 80 Gigabit/s routing performance with the lowest model and going up to several 100 Gbit/s or even 1 Terabit/s for the top router.

some body have tried overclocking the rb1100ahx4 or rb4011?

Hi, i have overclocked RB4011 to 2000Mhz.. When in properly ventilated environment (20' C) it tops about 55'C under 100% load. And goes stable. Only difference is to keep it cool enough in rack. FAN and clean headroom is a must. It is no longer passive cooled device.

some body have tried overclocking the rb1100ahx4 or rb4011?

Hi, i have overclocked RB4011 to 2000Mhz.. When in properly ventilated environment (20' C) it tops about 55'C under 100% load. And goes stable. Only difference is to keep it cool enough in rack. FAN and clean headroom is a must. It is no longer passive cooled device.

OMG that's a nice notice !!!