Community discussions

 
mauricioisp
Member Candidate
Member Candidate
Topic Author
Posts: 145
Joined: Wed Feb 06, 2013 2:48 am

What machine for 40Gbps Edge Router?

Tue Apr 10, 2018 7:06 pm

Hi guys,
We sell IP transit to some smaller ISPs and also offer High Speed to residential clients and our current Router solution is reaching the limit, I would like to deploy a new Mikrotik Router for 40Gbps of traffic. I don't consider the CCR1072 because it only has 10Gbps ports and I'm considering 40Gbps ports now. It needs to handle 4 full BGP tables and some BGP filters. What solution could you suggest me for that kind of setup?

Thanks,
 
doush
Long time Member
Long time Member
Posts: 621
Joined: Thu Jun 04, 2009 3:11 pm

Re: What machine for 40Gbps Edge Router?

Wed Apr 11, 2018 10:27 am

There are no Mikrotik Routers which has 40gbit QSFP+ ports.
You need to look at somewhere else.
 
jarda
Forum Guru
Forum Guru
Posts: 7602
Joined: Mon Oct 22, 2012 4:46 pm

Re: What machine for 40Gbps Edge Router?

Wed Apr 11, 2018 10:37 am

Or build a pc machine for that.
 
mistry7
Forum Guru
Forum Guru
Posts: 1314
Joined: Tue Oct 13, 2009 11:57 am
Location: Germany

Re: What machine for 40Gbps Edge Router?

Wed Apr 11, 2018 11:22 am

There is no peace of Hardware what can do that from Mikrotik.,
Look at needed Server and use CHR

Look for this PDF or Video from Mum Berlin:
Using MikroTik CHR as a BGP edge router by Kevin Myers (IP ArchiTechs, USA)
 
jrpaz
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jun 05, 2013 5:54 am

Re: What machine for 40Gbps Edge Router?

Wed Apr 11, 2018 2:48 pm

They don't even push close to 40 Gbps.
 
mauricioisp
Member Candidate
Member Candidate
Topic Author
Posts: 145
Joined: Wed Feb 06, 2013 2:48 am

Re: What machine for 40Gbps Edge Router?

Wed Apr 11, 2018 3:11 pm

And if I build a x86 with CHR, does RouterOS support any 40Gbps card?
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: What machine for 40Gbps Edge Router?

Wed Apr 11, 2018 3:14 pm

Only your host must support 40G interfaces. CHR uses virtual interfaces.
 
jrpaz
Frequent Visitor
Frequent Visitor
Posts: 82
Joined: Wed Jun 05, 2013 5:54 am

Re: What machine for 40Gbps Edge Router?

Thu Apr 12, 2018 2:36 am

I would love to see if anyone has it working at close to 40 Gbps.
 
mauricioisp
Member Candidate
Member Candidate
Topic Author
Posts: 145
Joined: Wed Feb 06, 2013 2:48 am

Re: What machine for 40Gbps Edge Router?

Thu Apr 12, 2018 3:01 am

Only your host must support 40G interfaces. CHR uses virtual interfaces.
What is the Virtualization platform that will offer the best performance in CHR? Xen? Hyper-V? Proxmox? ESXi?

Thanks,
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: What machine for 40Gbps Edge Router?

Thu Apr 12, 2018 12:02 pm

There was a good presentation at EU MUM about CHR performance on different hypervisors
https://youtu.be/xcgdGA1W_0o
 
User avatar
TomjNorthIdaho
Forum Veteran
Forum Veteran
Posts: 990
Joined: Mon Oct 04, 2010 11:25 pm
Location: North Idaho
Contact:

Re: What machine for 40Gbps Edge Router?

Mon Apr 16, 2018 6:19 pm

Only your host must support 40G interfaces. CHR uses virtual interfaces.
I may be wrong , however I beleive the paravirtual vmxnet3 ethernet interface is NOT limited to 10-Gig maximum throughput.
On two different virtual machines both hosted on the same physical VmWare ESXi server, I've been able to acheive about 38-Gig rates.
This was with multiple transfers at the same time going through the vmxnet3 interface.

Thus, I suspect the limiting factor is NOT the paravirtual vmxnet3 Ethernet interfaces - but rather the physical 40 gig interface talking to the real physical world.

North Idaho Tom Jones
 
PortalNET
just joined
Posts: 14
Joined: Sun Apr 02, 2017 7:24 pm

Re: What machine for 40Gbps Edge Router?

Thu Feb 14, 2019 11:56 pm

Hi guys,
We sell IP transit to some smaller ISPs and also offer High Speed to residential clients and our current Router solution is reaching the limit, I would like to deploy a new Mikrotik Router for 40Gbps of traffic. I don't consider the CCR1072 because it only has 10Gbps ports and I'm considering 40Gbps ports now. It needs to handle 4 full BGP tables and some BGP filters. What solution could you suggest me for that kind of setup?

Thanks,
Hiya

The best Mikrotik device i would recommend for your needs its called "Juniper MX104" it will suit you just fine up to 160gbps, if you require anything further then 160gbps backhaul i recomend the "Juniper MX240" and be happy.



in the past i have done some testing with with some X64 AMD octa core overclocked 4.0ghzp cpus running Mikrotik RouterOS x86/64 enabled, the max troughput i managed to run on the internal cpu core was abour 25gbps on udp and litle over 11gbps TCP..

not shure how it woudl behave on one Dell PowerEdge 6100 series with 48 cores total from all 8 cpus.. with 192GB RAM.. not shure if it would even be detected correctly as on the PC i runned i had 16GB of ram and only 6GB were detected.. i think there is some old post of mine with the pictures of the benchmark tests.

now putting that aside testing is one thing, real day to day job task , with bgp routing, nat, pppoe and so on.. it another completely differente story.
 
User avatar
PCaddict69
just joined
Posts: 17
Joined: Fri Jun 29, 2007 6:40 pm

Re: What machine for 40Gbps Edge Router?

Tue Mar 12, 2019 5:03 am

i just do some benchmarking and single thread performance make a HUGE difference . The best cpu i found so far bang for the buck performance is the 8600k i can push over 70 gbps on hyper-v
 
christopherh
newbie
Posts: 29
Joined: Sun Feb 24, 2019 7:43 am
Location: Sydney, Australia

Re: What machine for 40Gbps Edge Router?

Thu Mar 14, 2019 11:05 am

The best Mikrotik device i would recommend for your needs its called "Juniper MX104"
How is a "Juniper MX104" a Mikrotik router?
 
kevinM
just joined
Posts: 12
Joined: Mon Jul 28, 2014 8:57 pm

Re: What machine for 40Gbps Edge Router?

Thu Mar 21, 2019 7:16 pm

Just thought i'd give some input as i have been wondering about the viability of CHR and 40G interfaces. I wanted to lab up a scenario of replacing my 1072 core routers with CHR/40G.

I have a couple Dell R430 Servers with dual 8 Core Xeon servers (running at 2.1ghz) in my lab along with a Nexus 3064 Switch. I purchased a couple intel XL710 cards and setup a lab. The Dell servers each had a XL710 Card in it connecting to the 40G ports on the nexus switch. the ports on the nexus switch were simply setup as trunk ports. I installed the latest version of VMWARE 6.7 as well as 6.44 CHR using the ova off the mikrotik website. The CHR was given access to all vCPU's totaling 32 (as hyperthreading was enabled) as well as 16GB of memory and a 10G HDD. The CHR was trial licensed for Unlimted.

For my test I wanted to run a full config from my core routers to try and simulate real world performance in my specific case. This included on both routers roughly 16 input firewall rules, 11 forward rules pointing to 25 address lists. There are 32 IP addresses and 24 interfaces participating in OSPF although only one is actually neighbored to the other core router through the switch for this test. These routers were also IPV6 enabled and had roughly 9 input rules pointing to only one address list, but also had an equal amount of OSPFv3 interfaces as ospfv2. The only ipv6 address on the router is the loopback.

As I didn't have any other devices with 40G interfaces the only testing I could do at this time was BT test from one CHR through the switch to the other CHR. This was done with both TCP and UDP, from one loopback address to the other routers loopback address. Meaning it was taking the OSPF route between them through the switch as it's path. The VLAN being used for the OSPF route between the routers I adjusted to try different MTU's. I ran bi-directional BT tests, the max I could try is 8996 as VMWARE only supports up to 9000, minus the vlan equals 8996. My initital results are below:

MTU 700
UDP:3.2G/3.2G
TCP:4.4G/3.7G

MTU 1500
UDP:7G/7G
TCP:7.4G/7.1G

MTU: 8996
UDP: 36G/31G
TCP: 12G/12G

I know my use case and testing methodology might not be very exact, I just had some spare parts and was curious what would happen and thought the result might be helpful to others as I couldn't find much about CHR and 40G. If you guys have any ideas to try testing I should have the lab up for a bit longer. Although I'll be out of town next week.
 
elbob2002
newbie
Posts: 48
Joined: Tue May 15, 2018 8:15 pm
Location: Ireland

Re: What machine for 40Gbps Edge Router?

Sun Mar 24, 2019 8:33 pm

I'd be curious to see what performance you would get with hyperthreading disabled on the hosts.
 
kevinM
just joined
Posts: 12
Joined: Mon Jul 28, 2014 8:57 pm

Re: What machine for 40Gbps Edge Router?

Thu Apr 04, 2019 10:59 pm

disabled hyperthreading and adjusted CHR's to use 16 vCPU and re-ran the same tests. the results are below. In my lab testing i've found hyperthreading to be a benefit even though i've read multiple posts saying to disable it. Obviously my BT test between two routers isn't a real world working scenario which might be the difference. But i dont know how else to simulate real world traffic easily in the lab setup.

MTU 700
UDP:3.2G/2G
TCP:3.5G/3.5G

MTU 1500
UDP:5G/3.5G
TCP:6G/6G

MTU 8996
UDP:16G/16G
TCP:8G/8G
 
zyzelis
Member Candidate
Member Candidate
Posts: 212
Joined: Sun Apr 08, 2012 9:25 pm

Re: What machine for 40Gbps Edge Router?

Fri Apr 05, 2019 12:25 pm

Hi,
you should select proper kernel version for CHR on esxi - 3.x


disabled hyperthreading and adjusted CHR's to use 16 vCPU and re-ran the same tests. the results are below. In my lab testing i've found hyperthreading to be a benefit even though i've read multiple posts saying to disable it. Obviously my BT test between two routers isn't a real world working scenario which might be the difference. But i dont know how else to simulate real world traffic easily in the lab setup.

MTU 700
UDP:3.2G/2G
TCP:3.5G/3.5G

MTU 1500
UDP:5G/3.5G
TCP:6G/6G

MTU 8996
UDP:16G/16G
TCP:8G/8G
 
elbob2002
newbie
Posts: 48
Joined: Tue May 15, 2018 8:15 pm
Location: Ireland

Re: What machine for 40Gbps Edge Router?

Fri Apr 05, 2019 3:18 pm

You seem to have consistent results with and without hyperthreading (with the exception of the last MTU setting) . I'm wondering if you're not hitting a limitation elsewhere?

You could try using iperf on a couple of VMs?
 
kevinM
just joined
Posts: 12
Joined: Mon Jul 28, 2014 8:57 pm

Re: What machine for 40Gbps Edge Router?

Fri Apr 05, 2019 6:29 pm

Hi,
you should select proper kernel version for CHR on esxi - 3.x
I used the ova provided on mikrotiks website to import the VM, I assume there template has the correct kernal version. Currently it's showing as "Other 3.x Linux (64-bit)". What are you suggesting I change it to?
 
kevinM
just joined
Posts: 12
Joined: Mon Jul 28, 2014 8:57 pm

Re: What machine for 40Gbps Edge Router?

Fri Apr 05, 2019 6:40 pm

You seem to have consistent results with and without hyper-threading (with the exception of the last MTU setting) . I'm wondering if you're not hitting a limitation elsewhere?

You could try using iperf on a couple of VMs?
I did try that, i installed a centos box on both hosts and gave them a single core and minimal resources to just run iperf. The guest os's run at an mtu of 1500 by default and at that level the VM's got a little less then the bandwidth testing between the routers did, but the CPU on the guest OS was also maxing out. I could give the VM's more resources in theory but then it would take away from the chr running on that host. My guess is that the results would be similar with iperf on a vm to the router BT testing if you account for the loss of overall resources to the CHR running the VM at the same time.
 
elbob2002
newbie
Posts: 48
Joined: Tue May 15, 2018 8:15 pm
Location: Ireland

Re: What machine for 40Gbps Edge Router?

Fri Apr 05, 2019 8:39 pm

I just did a test between two CHR on one of my ESXi 6.0 servers. From a CHR with a P1 license to one with a P10 license:
                  status: done
     time-remaining: 0s
ping-min-avg-max: 57us / 106us / 264us
jitter-min-avg-max: 0s / 30us / 168us
                loss: 0% (0/200)
        tcp-download: 11.1Gbps local-cpu-load:37%
          udp-upload: 1.07Gbps local-cpu-load:17% remote-cpu-load:10%

Edit - Forgot to add that the destination is 6.44 and the one where the test was run from was 6.44.1. Both have 256MB RAM with two vCPU. VMXNET3 adapters.

Who is online

Users browsing this forum: No registered users and 3 guests