A while back I took over as lead network engineer for a WISP and am still dealing with issues from a topology that was not designed to scale, the previous lead did all sorts of implementations, some of it alright but some of it very crap (EoIP over a point-to-point that was already bridged… No split-horizon, no thought STP root priorities etc). We are growing so fast that I havn’t had a chance to get around to redesigning everything, but as I predicted we are hitting some walls that are causing issues and now we really need to focus on redesign. So I thought i’d ask for a bit of help from people who have worked with moderate to large WISP’s and can provide a bit more insight into the potential drawbacks of various network designs, particularly over radio links. A lot of our customers use VoIP and their businesses rely on it, so minimizing downtime with re-convergence and providing the best QoS is desirable. At the same time we need to look at our growth and ensuring we can scale smoothly without having to put in lots of individual VLAN’s, VPN’s etc as we add new sites. And making it easy for a new tech to learn and manage our network by having it fairly uniform and consistent
Since we exclusively use MikroTik routers I feel it is appropriate to ask here, but if there is a more appropriate forum I can post there as well
Here is a sample topology that is pretty common in our network
We have main distribution routers that provide interconnects to a datacentre, and from those they go to other locations that feed multiple customers. We own the entire backbone network and most of the site and customer links are wireless
Every customer has an onsite router (approx 80% are Mikrotik), it connects to us via PPPoE (I would like to move to NAT 1:1 with DHCP Option 82 down the track, but our current RADIUS accounting software does not support it, so for now we have to stay with PPPoE)
Where the PPPoE session terminates varies in our network. Sometimes its the closest point to the customer, other times it goes through 2 or 3 routers (VLAN or VPLS)
*NOTE: Green links denote L2, Orange is L3, Green routers are also PPPoE Servers
Sample Topology
The question is how should we redesign the network to be easier to manage and more scalable
Should we move all the PPPoE sessions to be as close as possible to the customer, remove all backhaul VLAN’s and use routing between every site?
Routed
Should we use routing and MPLS, move most of the PPPoE sessions to be closest to each customer, but for some small sites (1 or 2 customers) use bridging/VLANs
MPLS
Should we terminate all PPPoE at the furthest point from the customer, and use VPLS to every site
VPLS
I can see some positives and some negatives to every topology choice
Routing:
Fastest re-convergence if a link goes down
Very easy to troubleshoot where an issue is
MPLS:
Could be a lot of radio’s and switches that need to be reconfigured or replaced due to MTU issues
Easier to setup routing between customers/sites, just add additional MPLS interconnects
Unsure of failover time compared to pure routing, especially over radio links
VPLS:
Easiest and fastest to setup new sites
Very easy to see where a customer is in our network by looking at PPPoE sessions under VPLS interfaces on just a few routers
Potentially quite slow for a customer to come back online if there is a back-haul outage, as PPPoE may need to reconnect?
Hardest to troubleshoot connectivity issues?
No idea how VPLS compares to MPLS with fail-over?
Less download traffic / dropped packets in our network as speed limiting happens at the internet edge, as opposed to going through our entire backbone to the customer before being throttled
Did you receive any feedback and or have you decided to go with a solution?
We are looking at the same senario as we are growing faster than expected and want to have our network ready before we start having major issues and outages.
Did you receive any feedback and or have you decided to go with a solution?
We are looking at the same senario as we are growing faster than expected and want to have our network ready before we start having major issues and outages.
