Community discussions

just joined
Topic Author
Posts: 4
Joined: Fri Mar 08, 2019 4:13 pm

Scaling Mikrotik

Fri Mar 08, 2019 4:20 pm

Good day all,

I'm new to Mikrotik but have a decent Cisco background.

I'm working on a new network setup for an ISP that intends to have 5-10k customers in 5 years.

Our backbone will be a 10gbps network with the goal of servicing 50mbps on average to our customers.

Any advice on how to build out our core and edge routing / switching? What bottlenecks may I run into?

If any additional detail is needed please ask.
just joined
Topic Author
Posts: 4
Joined: Fri Mar 08, 2019 4:13 pm

Re: Scaling Mikrotik

Sat Mar 09, 2019 4:05 am

Member Candidate
Member Candidate
Posts: 109
Joined: Sat Nov 25, 2017 10:01 pm

Re: Scaling Mikrotik

Sun Mar 10, 2019 5:44 pm

@magik20 I'm not really sure what you are expecting as replies to this. You're essentially asking "How should I build this ISP network?" but you haven't provided any real information to help give the advice or answer. What have you determined so far? What bottlenecks have you identified? What layout/topology have you been kicking around for this? What is the expected user base going to be in the first 6 months to a year? Are you doing IPv4, IPv6 or a combination of them both? What kind of address space do you have if IPv4? Are you looking at CGNAT or anything else? How is the service going to be delivered to the end user? Is this a WISP, an ISP doing copper, coax, fiber?! What is the expect growth rate and sales projections/goals per month, per year? I mean is the expectation to turn 5, 50 or 100 users per month, per quarter? Will there be varying levels of speed packages with 50Mbps being the sweet spot and the probably the most selected package?

There are a litany of questions and numerous data points that, as the Iceman would say, "Is a long and distinguished as my johnson.." that need to be looked at to provide any real and solid advice for you on this project. I can say that the biggest bottleneck you're going to run into right now with what you've provided is the bandwidth and oversubscribing users. With a 10Gbps connection and the average user have 50Mbps on their connection you can stand to have roughly 200 users max out at 50Mbps at any given time. After that you are maxed out and it will start driving the speeds down and cause issues.

Look at it this way, I currently have an 80Mbps connection right now at my house. The average usage when everyone is home usually is around 10Mbps with all the streaming and gaming happening. Now take that and say that your users will be about the same. They will have a 50Mbps package but probably use around 8Mbps as the low side average peak. If in your first year you grow to 1,500 users you are looking at roughly 1.2Gbps of consistent usage by your users. Also at this point you are looking at 1,500 users x 50Mbps which means roughly 7.5Gbps of your 10Gbps backbone is allocated to those users and you're only at 10% of your protected based.

You could launch and within your first six months end up with 3,000 users and then over the next six months only bring on 250 new ones. However at this point with 3,000+ users on the 50Mbps plan has already oversubscribed your network as that would require 15Gbps to facilitate the ability to provide the speeds you are selling to the users.
Member Candidate
Member Candidate
Posts: 212
Joined: Thu Jan 18, 2018 5:18 pm

Re: Scaling Mikrotik

Sun Mar 10, 2019 5:47 pm

You can buy more as you get bigger
Posts: 497
Joined: Mon May 27, 2013 2:05 am

Re: Scaling Mikrotik

Mon Mar 11, 2019 1:05 am

Mikrotik is horizontal scaling. Basically start with redundant pairs everywhere - once you start getting to the 50-60% resource usage, add another 1 or 2 next to it. You really dont want resources hitting up over 75% at all to be safe.

I'd personally have your core very very simple, just pure routing and forwarding with CCR1072's. Then some kind of lower-tier for PPPoE servers - if you need to do shaping, do it here, if you need to do CGNAT, do it here, maybe CCR1036's so you can use 10gbit uplink into the 1072's.

So basically you'd have 2 cores and then as many LNS's as you need. Perhaps also edge routers if you have a lot of BGP peers.

Depending on how heavy your load is you may be looking at 1 router for about 2k PPP users, but you'd just have to wait and see how much strain your users put on the routers.
just joined
Posts: 11
Joined: Fri Apr 07, 2017 5:45 pm

Re: Scaling Mikrotik

Tue Apr 23, 2019 10:38 am

My customer is using mikrotik for terminating l2tp channels. About 30 CCR.
Typical scheme is:
Aggregator CCR: CPE:
vlan ---- bridge ---- vpls---- l2tpl-----------l2tp----vpls----bridge-----ethernet

On each CCR can be from 100 to 200 clients approximately.

It has such kind of problems:
- on some of devices there are very strange situations - after rebooting there are no traffic on physical interfaces. In is possible to manage device only via console.
TechSupport said, that they were able to repeat this problem on lab, but cannot say, when this issue will be solved. Their advice is to disable/enable IP addresses on interfaces.
Our solution is - to disable-enable physical interfaces after restart this equipment.

- At this time there are some new issues:
on one of CCRs - CCR1036-8G-2S+ when it was about 550 active l2tp sessions, customers (from CPE side) complain about service degradation (lost packages, no traffic, etc.)
on CCR1036-8G-2S+ it was no problem with CPU load, no problems with bandwidth on interfaces.
when number of l2tp-sessions was more than 600, CCR1036-8G-2S+ begun to lost management traffic (ssh), the console started be delays. It was necessary to reboot it.
The solution at this time is moving part of l2tp clients from one mikrotik to other.
We send information to mikrotik techsupport, but this time it is no answer.

My opinion - it is scaling problems. But unfortunately, I was not able to find any official detail scaling information, so it is very difficult to plan network with this equipment for ISP or SP.
Other problem - a reaction of Mikrotik tech support for ISP issues. Yes I like the equipment of this vendor, but unfortunately the quality of tech support organization, scaling information is not enough for ISP.

Modest Sokolov

Who is online

Users browsing this forum: No registered users and 19 guests