I'm using your products for several years. They are not bug less, but in general they are quite stable once they are configured properly. Your new CCR and CRS are very nice products as well. I would like to start to deploy them in enterprise/private cloud environments, but they are still not capable for doing that.

We are looking to connect primarily HP (optionally IBM and SuperMicro) blade enclosures with 10G uplinks. 2 SFP+ connections are simply not enough for that. They would need to act more like L3 switches than a serious router or firewall, but switching on 80Gbps+ will require high CPU performance or ASIC made for this. Unfortunately system mentioned below is not tested, so I can't state how good or bad it performs with ROS.

Here is what would be nice to have: Can you create something with much more 10G ports? Even oversubscribed system. Here is an example:
With a cheaper HP server it's possible to create a multi port router with 8SFP+ ports and 4x 1Gbps + console / remote management and optionally redundant PSU.

HP ProLiant DL380e Gen8 1 2,479.95 €

669255-B21#B19 HP ProLiant DL380e Gen8 8 LFF Configure-to-order Europe-Multilingual Localization Server [1]
726113-B21#0D1 HP 8GB SD Enterprise Mainstream Flash Media Kit [1]
663476-B21 HP 2U FIO Friction Rail Kit [1]
661404-B21#0D1 HP DL380e Gen8 CPU1 Riser Kit [1]
708497-L21 HP DL380e Gen8 Intel Xeon E5-2407v2 (2.4GHz/4-core/10MB/80W) FIO Processor Kit [1]
713977-B21#0D1 HP 4GB (1x4GB) Dual Rank x8 PC3L-12800E (DDR3-1600) Unbuffered CAS-11 Low Voltage Memory Kit [1]
614203-B21#0D1 HP NC552SFP 10Gb 2-port Ethernet Server Adapter [4]
503296-B21#0D1 HP 460W HE 12V Hot Plug AC Power Supply Kit [1]


A basic Cisco 4900M costs 3 times more. Ok it's different category and it can be expanded even more, but from other side RouterOS can do more things for less.
NetGear has 10G switches which costs about $100 per port so I would expect similar price range.

Any configuration would be welcome for example: 8x SFP+ and/or 8x 10Gbase-T, 8x 1Gbps, redundant PSU and cooling + option to stack it with another MT Switch product... Am I asking too much?

They have announced the CCR-1072-8S+

72 cores, 8 10G ports, 1 1G port, redundant PSU.

Should be nice, and close to what you are looking for.

Hopefully they will be out sometime this year.

That's it. I hope it will be under $2000, work stable and it will be released very soon

They have announced the CCR-1072-8S+

72 cores, 8 10G ports, 1 1G port, redundant PSU.

Should be nice, and close to what you are looking for.

Hopefully they will be out sometime this year.

presently each Tile core can be saturated by 1Gb, right ?
so multiply it to 10 and number of desired ports and you will get expected horsepower and price of required Tilera chip(they had Any kind of them, from 4-core to hungreds ).
MikroTik developers announced that they soon - will able to spread even fragmented TCP traffic across cores(maybe adopted Haskel or Erlang ? in future versions, so there would be very warm expectations among customers about CCR performance, achievable, thus.

presently each Tile core can be saturated by 1Gb, right ?
so multiply it to 10 and number of desired ports and you will get expected horsepower and price of required Tilera chip(they had Any kind of them, from 4-core to hungreds ).
MikroTik developers announced that they soon - will able to spread even fragmented TCP traffic across cores(maybe adopted Haskel or Erlang ? in future versions, so there would be very warm expectations among customers about CCR performance, achievable, thus.

where did you get the 1G per core limit from.

Strangely i have been running into 1Gb/s limit on TCP streams in my testing of CCR1036-8G-2S+EM

This would explain a lot ... I have logged a support request.. been silent for a while, long while..

You gotta be shitting me. I've been waiting on a reply about something similar. 1gb per core , single stream? Throw gre an ipsec in the mix and you are. Down to. 50 mbit. Rb1100 works better.

We are seeing max 1Gbps per "connection" on CCR1036-8G-2S+ as well.

We can obviously achieve much higher throughput using multiple connections, but any individual connection can only reach 1Gbps. I believe this is due to the current queuing implementation used on Tilera, and can possibly be improved.

Maybe Janis from Mikrotik can clear this up ?

For ISP aggregate traffic, which is really what MikroTik seems to be playing for, 1gbps single stream isn't that bad a limitation. We are a mid-sized wholesaler in AU and still will only do about 500-600mbit aggregate traffic.

Big single streams are more suited to switching rather than routing.

They have announced the CCR-1072-8S+

72 cores, 8 10G ports, 1 1G port, redundant PSU.

spotted some nice pix of a prototype on the web:

https://pbs.twimg.com/media/Bg_zYx7IIAEIndN.jpg:large

if you look closely on the left side there is something strange. it has a mini-pcie/m-sata
connector, as i figure the mounting screws facing towards the back of the device. and there
is something what seems to be a sim card socket. e.g. it could have integrated 3g/4g connectivity
for like OoB access.

http://lanmarket.ua/upload-files/foto/p ... 2553-b.jpg
and a connector is named as "smart card" on this image.
also note the slide-in connectors at the back of the PSUs, which - i suspect - gives us hot swappable power supplies.

+1 for hotswap PSU

+1 for PSU module (AC , DC-48V, DC+24V) or DC wide band 20-60V (with isolation +/-)

+1 for hotswap PSU

+1 for PSU module (AC , DC-48V, DC+24V) or DC wide band 20-60V (with isolation +/-)

telco dc (-48VDC) is usually 36-72V tolerant, and most of the time i measured voltages just above 50V.
i agree, a real -48vDC power supply in the ccr1072 would be a win for the telco market.
i easily added 48V->24V open frame power supplies to all mikrotik devices, because there was
plenty of free space available inside the rack mount case.

but looking at the 1072 prototypes my "retrofit" approach won't cut it anymore. either mtik delivers dc option
or we must design a compatible dc-dc power module because the stuff we normally use has a different form
factor if we go over 20w. and looking at the fans i suspect the 1072 will need around 80-100w at least.
or we need to use external 48VDC-230V inverters to get this box running in telco environments.

For ISP aggregate traffic, which is really what MikroTik seems to be playing for, 1gbps single stream isn't that bad a limitation. We are a mid-sized wholesaler in AU and still will only do about 500-600mbit aggregate traffic.

Big single streams are more suited to switching rather than routing.

Switching and routing are all the same thing these days. Can't see how you would think 1gbit throughput on a 10gb router is acceptable. Heck. A single Iscsi connection could max it.

Switching and routing are all the same thing these days. Can't see how you would think 1gbit throughput on a 10gb router is acceptable. Heck. A single Iscsi connection could max it.

In no way are they the same 'these days'. Backplanes on good switches are massive, and if you have high transfer networks, you specifically design your network so as much data as possible goes via switching rather than through a router port. Even the big boys like Cisco, there is a big difference in routing and switching still.

Granted some of the Mikrotik ranges have switch chips for good performance, the high performance routers do not, and the specifically designed switches in Mikrotik range have much lower power CPU's where you could 'route' no where near the same amount of traffic as their proper routers.

Again - large single streams are typically not 'routed', and usually more local, hence switching offers a better solution. I agree there is a performance limitation that should be dealt with, but the fact of the matter is these tools are crafted for a specific job, in my opinion...aggregate traffic, which they will do well even with this limitation. Right tool for the right job applies and if Mikrotik is not achieving what you are after, then it is not the right tool for the job.

Old switch ASICS only knew how to process VLAN tags, MAC Addresses, and maybe DSCP/TOS. Modern switch ASICS know how to process data at L4. This encompasses routing.

The $4k SWITCHES we use can ROUTE or SWITCH at line-rate across all ports simultaneously, 50 gbe ports and 2-4 10gbe ports. The core SWITCHES we use as ROUTERS can ROUTE or SWITCH 10gb x48 ports at line rate across all ports simultaneously. So, in the second case, that is 480gb full duplex = 960gb TCP, UDP, GRE, whatever, switched, routed, MPLS, running OSPF, BGP, PIM, etc, etc. The port-to-port latency is the same wether the packet was switched or routed. The handling of the packet is done in the same way wether it was switched or routed.

IE: These days, a switch and a router are the same thing. The software frontend is just that. A frontend that tells the ASICS/PGAs how to do their work, in hardware.

That differs when you are talking about a SOFT router that incorporates a switch-chip. But still, the assumption that nobody needs to ROUTE a single connection at more than 1gbit is just that, an assumption. It is a bad limitation. You don't advertise a device that can do xyz gbit fast path forwarding in bridging or routing when it can't really do that. They should have a little asterisk at the bottom of the claim *The following only applies to single/multiple stream lossy UDP/GRE/Non TCP/out of order processed/super fragmented packets. Otherwise, the single stream TCP limitation is 1gbit/second because we haven't finished making everything work right.

Mikrotiks are awesome.. Ive used them for years in the case we DONT need 960gbit of throughput with a $20k price tag.. But seriously... the snake-oil has to stop. You buy a 36 core soft-router that is advertised at being able to do ~3gb of ipsec throughput and upwards of 10gbit of slow-path forwarding and you end up with a device that can only manage a GRE or IPIP tunnel over IPSEC at 50mbit aggregate duplex using almost no CPU power, then get no response on forums or from support after multiple attempts to make contact.

Now, would a single stream limitation of 1gbit affect my desired use of these devices as a VPN concentrator? No... The max connection speed to the remote sites would be 1gbit.. BUT.. I need way more than 50mbit...

The $4k SWITCHES we use can ROUTE or SWITCH at line-rate across all ports simultaneously, 50 gbe ports and 2-4 10gbe ports. The core SWITCHES we use as ROUTERS can ROUTE or SWITCH 10gb x48 ports at line rate across all ports simultaneously. So, in the second case, that is 480gb full duplex = 960gb TCP, UDP, GRE, whatever, switched, routed, MPLS, running OSPF, BGP, PIM, etc, etc. The port-to-port latency is the same wether the packet was switched or routed. The handling of the packet is done in the same way wether it was switched or routed.

IE: These days, a switch and a router are the same thing. The software frontend is just that. A frontend that tells the ASICS/PGAs how to do their work, in hardware.

They are capable of talking routing protocols, but are not built for it. You wouldn't load a L3 switch up with multiple full routing tables plus IGP. The CPU/RAM wouldn't hold it. Same with technically you can route with the Mikrotik CRS range but you wouldn't put it at the core or edge (unless you are really small).

That differs when you are talking about a SOFT router that incorporates a switch-chip. But still, the assumption that nobody needs to ROUTE a single connection at more than 1gbit is just that, an assumption. It is a bad limitation. You don't advertise a device that can do xyz gbit fast path forwarding in bridging or routing when it can't really do that. They should have a little asterisk at the bottom of the claim *The following only applies to single/multiple stream lossy UDP/GRE/Non TCP/out of order processed/super fragmented packets. Otherwise, the single stream TCP limitation is 1gbit/second because we haven't finished making everything work right.

All their numbers and test results indicate exactly how the tests were done, including frame-size, pps and even the technology used.

Mikrotiks are awesome.. Ive used them for years in the case we DONT need 960gbit of throughput with a $20k price tag.. But seriously... the snake-oil has to stop. You buy a 36 core soft-router that is advertised at being able to do ~3gb of ipsec throughput and upwards of 10gbit of slow-path forwarding and you end up with a device that can only manage a GRE or IPIP tunnel over IPSEC at 50mbit aggregate duplex using almost no CPU power, then get no response on forums or from support after multiple attempts to make contact.

Now, would a single stream limitation of 1gbit affect my desired use of these devices as a VPN concentrator? No... The max connection speed to the remote sites would be 1gbit.. BUT.. I need way more than 50mbit...

Can't comment on the IPsec limitations I dont use the technology enough to have run into this problem, but fair enough I could see how THAT problem would be annoying.

Use l3 switches for our bgp peerings. 3 full v6 and v4 tables. All in hardware. Works perfect. College campus.

Yes. The tests show that. Which makes it even worse. They designed the tests to cover up the flaws.

Old switch ASICS only knew how to process VLAN tags, MAC Addresses, and maybe DSCP/TOS. Modern switch ASICS know how to process data at L4. This encompasses routing.

Mikrotiks are awesome.. Ive used them for years in the case we DONT need 960gbit of throughput with a $20k price tag.. But seriously... the snake-oil has to stop. You buy a 36 core soft-router that is advertised at being able to do ~3gb of ipsec throughput and upwards of 10gbit of slow-path forwarding and you end up with a device that can only manage a GRE or IPIP tunnel over IPSEC at 50mbit aggregate duplex using almost no CPU power, then get no response on forums or from support after multiple attempts to make contact.

Now, would a single stream limitation of 1gbit affect my desired use of these devices as a VPN concentrator? No... The max connection speed to the remote sites would be 1gbit.. BUT.. I need way more than 50mbit...

I have to agree, i like the brand, but when you advertise a 28Gb/s router, that is actually rate limited to max 1Gb/s tcp streams, thats more than just snake-oil.....

Use l3 switches for our bgp peerings. 3 full v6 and v4 tables. All in hardware. Works perfect. College campus.
.

Which switches do you use?

Yes, I am curious as well.
roadracer96, what switches are you using for BGP?