Recently I’ve decided to finally start moving our wireless infrastructure from good old bridged network to something more complicated but also more secure and performant.
After digging through couple of topics on forum and elsewhere my current plan looks as follow:
network nodes running MPLS/OSPF
single (?) CCR acting as concentrator: bridging “downstream” VPLS tunnels, using bridge horizon and proxy arp/L2 nat to separate customers
access points (PE routers) bridging “upstream” VPLS tunnel with wireless/cable interfaces
DHCP relay running on APs (bridge), distributing IPs based on interface number and/or MAC address
As you can see above there is no PPPoE, I assume that using bridge horizon and proxy arp provide enough to isolate customers but also avoid a bit of network overhead.
Obviously there are a couple of things to test (like DHCP relay which I didn’t tried yet on MT devices) but before that I would like to hear your opinion on few topics:
should I split network down to single devices, i.e. nowadays a lot of our nodes consist of OmniTik providing single AP and power, uplink and optionally some sector antennas. As an alternative a single device (OmniTik most likely in such configuration) would act as router, while other devices are merely “interfaces”. Are there any strong pros/cons against those solutions (routing table size/OSPF operation cost aside) ?
on concentrator side: single bridge (one big subnet) vs multiple bridges, swapping VPLS/smaller subnets when necessary - can I gain anything from such a setup (performance/stability wise), considering I’m going to use proxy-arp/L2 nat
OSPF network-type/priority - manual (which “flavor”) ? broadcast based ? maybe something hybrid, i.e. broadcast on core (connected via fiber/ethernet) while downstream uses some kind of manual configuration. In case of choosing easy way and using broadcast, should I force some of the routers to never be eligible to become DR ?
I would let the underlying topology dictate the type of OSPF interfaces you use.
If you have a switch with a bunch of routers plugged into it, then keep it simple and use broadcast.
If you have a point-to-point link, then use that type (no DR)
If you have a bunch of sites connecting to the same SSID of an AP, I would use ptmp and client isolation.
If you’re trying to limit who may join your OSPF domain, then use authentication to control that, not “unusual” interfaces.
I’ve personally done the proxy-arp thing and it is very good at causing a single supernet to appear in multiple locations as if it were a flat network, but in reality being a routed backbone. I routinely added the same “default GW” address and broadcast address to a loopback interface on all participating routers. (anycasting)
It’s interesting to be on 192.168.1.54/24 and seeing:
host ~> traceroute 192.168.1.55
1: 192.168.1.1
2: 10.12.31.97
3: 10.12.31.221
4: 10.3.15.95
5: 192.168.1.55
This seems like an interesting alternative to using VPLS - I played a bit with a few boards (wiped clean - no def config, 6.27) and managed to make it work but there are some problems with such setup, before that a quick description of my small “test lab”:
“standard” MPLS/OSPF setup with loopback interfaces, i.e. nothing special here
in addition to that I bridged each board so that I could access/debug it when playing with routing
PE running DHCP relay and arp proxy, CCR running DHCP server
CCR assigned “gateway” IP address, on PE I manually added static gateway ClientIP/32@ethX
Like mentioned before it works but.. manually adding route to each client on PE is not a very good solution, initially I thought of using DHCP on lease script but afaik there is not counterpart to remove route when not used, not to mention it’s a fragile solution.
The second problem to solve is how to separate clients, i.e. force traffic via main GW, I tried playing a bit with routing tables/marks but they are not distributed via OSPF and there doesn’t seem to be a simple solution to this.
I’m interested to see your setup, can you post some example configuration and whenever you encountered the same problems, how did you solved them? I think that at this point in time using VPLS overall is easier to implement/manage.
regarding MPLS, unless I’m missing something I think I encountered some kind of bug because I’ve issues making it work, to be exact on of the peers is not seeing the other (empty LDP neighbor table) while the other device happily list it as dynamic, operational. OSPF running the same devices runs just fine. The situation last for some time before MPLS ‘kicks in’, looks like enabling OSPF, sending a few pings helps it a bit, sniffer shows that devices are receiving related packects, weird.
BTW: is there any way to change what DHCP relay is sending back to DHCP server using option 82 ?
My primary objective with the proxy-arp solution was IP address conservation. I was working at a network services provider, and we supplied links that were already point-to-point in nature (clients were isolated at layer2 already). I was tired of assigning /30 networks for each customer.
Our routers were Cisco, which add the /32 into static route whenever a client gets DHCP. This was good for dynamic reachability - and I think I saw Mikrotik doing the same thing in one of my test beds. We just used “redistribute static” and aggregate prefix advertisements in each ASBR to keep the routing table from skyrocketing too much.
You’re right in that it’s a fragile solution if the users move frequently from one location to another, but our circuits were all hard-wired, so it didn’t matter for that.
Since Customer1 is on interface 1, and customer 2 is on interface 2, they cannot see each other’s broadcast/ARP/etc, and since Cisco answers the ARPs, it actually forces all traffic to go through the router as you requested. It also has the benefit of preventing a customer from hard-wiring any IP they want from the /24 - only their single IP would work on the link.
Again - this was first and foremost an IP address conservation technique. The IP mobility / point-to-multipoint behaviors were added benefits.
which add the /32 into static route whenever a client gets DHCP. This was good for dynamic reachability - and I think I saw Mikrotik doing the same thing in one of my test beds. We just used “redistribute static” and aggregate prefix advertisements in each ASBR to keep the routing table from skyrocketing too much.
I’m not really concerned about clients switching between APs, even now they can’t do it in our network. What I want to avoid is having some stale entries polluting the routing table, I can think of a few ways to solve it using scripting but it’s ‘itchy’.
Right now the DHCP server have the ability to add arp entries (which I planned to use as a security feature at some point), perhaps it would be a good idea to ask MT devs to add something similar which your Cisco switches have or at the very least the ‘on release run script’ feature which some people asked already.
I guess I should also try to find the presentation you mentioned.
Good luck with converting your MPLS network! It will give you many tools to deploy services that add a great deal of value to the network - for both the technical side and the business end. We build a large number of MPLS networks and wanted to share a few resources that should help you out.
Take a look at this presentation for detailed info on Carrier routing architectures using MikroTik from USA MUM 2013
