Building a 3000+ CPE network, looking for advice

RouterOS General
1

We have outgrown 802.11b on our network using Star-OS and Tranzeo with about 1500 clients. So we have decided to build a 802.11a/Nstreme network for our next 3000 clients.

I have a few challenges with the exact deployment we will use.

I want to be able to use full Class C’s (Or larger) to make sure I am efficient with my ARIN allocation.

There is no Proxy ARP bridging in the MT CPE so I have to use a different method than I currently use in my Tranzeo 2.4ghz CPEs.

Option 1
Use WDS
This will not work so great as there is no Sta isolate, I do not know if it will work with Nstreme and the overhead is not pleasant.

Option 2
NAT all the clients in their CPE
This will not be acceptable to my clients as they expect a real public ip on thier router and that is how my competitors do it.

Option 3
Route little /30 subnets to each client
This option will not fly with ARIN as it is wasting 75% of my IPs and is nightmare to route and support.

Option 4
Use PPPoE (or IPIP or L2TP) and build tunnels to each CPE and then bridge the CPE ethernet port to the PPPoE tunnel. This might work, assuming I can bridge the PPPoE to the Ethernet port?

Option 5
Any other ideas? I really want to know the manufacturers reccomended way to do this that fits my business.

2

It is difficult to propose a solution without knowing your network topology, however with that number of nodes I’d design for a strictly routed backbone if at all possible.

Can you tell us more about your network topology?


–Tom

3

MikroTik Refuses to do give us a MAC NAT option, so WDS is your only option. (dig through the forum, I’ve made a number of comments about it)

setup your AP as WDS dynamic (Bridge1 as the default), set CPE as station WDS, on the station bridge E1 and W1, on the AP, add a VLAN to your Bridge1. Enable NStream + Polling (and eagerly wait for the new NStream in v3), then have a machine setup as a PPPoE concentrator at each tower (P4, decent ram and processor) and create a PPPoE server tied to each VLAN.

things to make sure of:

  • Use a Unique VLAN for each AP on the tower, this helps maintain some client seperation (not ideal, but its decent)


  • Create Bridge rules on the AP to limit CPE to CPE communucation as much as possible given your requirements


  • use an AP with plenty of CPU (532 set to 330mhz at minimum)


  • Use Radius to define your bandwidth shaping per PPPoE customer


  • don’t overload the AP, ping times from the CPE to the AP are going to be roughly related to the number of CPE’s associated (side effect of Polling, we limit ours to roughly 25 per radio)

that should be a good starting point.

4

I thought I listed several other options?

setup your AP as WDS dynamic (Bridge1 as the default), set CPE as station WDS, on the station bridge E1 and W1, on the AP, add a VLAN to your Bridge1. Enable NStream + Polling (and eagerly wait for the new NStream in v3), then have a machine setup as a PPPoE concentrator at each tower (P4, decent ram and processor) and create a PPPoE server tied to each VLAN.

things to make sure of:

  • Use a Unique VLAN for each AP on the tower, this helps maintain some client seperation (not ideal, but its decent)

  • Create Bridge rules on the AP to limit CPE to CPE communucation as much as possible given your requirements

  • use an AP with plenty of CPU (532 set to 330mhz at minimum)

  • Use Radius to define your bandwidth shaping per PPPoE customer

  • don’t overload the AP, ping times from the CPE to the AP are going to be roughly related to the number of CPE’s associated (side effect of Polling, we limit ours to roughly 25 per radio)

This sounds like a huge step backward. We currently have 80 clients per AP each shaped to 2mbit and it works very very well. And that is on 802.11b. Why does 54mbit at 802.11a have less capacity?

Our network is 100% routed except at the CPE. I currently have each AP run DHCP with 128 IPs.

5

Option 4 is what I’ve been setting up on our WISP

Route an IP block to the MT Router (AP) big enough to give one address to each client + however many spare you think you might need soon, assign one of these to the router. Then get the CPE to PPPoE into the MT Router and hand out one of the adresses to the CPE you will then have a /32 subnet between the AP and the CPE so keeping the whole network routed.

6

I agree about the PPPoE. I have set up many networks with it. It reduces risk of network loops, more efficiently uses ip addresses, is easier to contain virus and trojan traffic, with radius it is easy to manage, offers a good user validation mechanism, you can easily static assign public ips, or dynamically assign from pools of public or private via radius, and it is easy to set up on the client side.

There are definately other ways and will be better ones some day but at this time I see it as the best way to handle such a network while not losing functionality.

Good luck.

7

So then bridging between the PPPoE tunnel and the CPE eth is supported?

I was thinking I would add a firewall rule blocking everything except unicast, ARP and DHCP. That way clients will not see each other’s traffic. (except ARP and DHCP)

8

A bridge is created from the CPE to the AP, the AP can bridge to another interface eg, an ethernet port. The CPE then creates a PPPoE tunnel over this bridge to the PPPoE Access Concentrator - which if the AP is a MikroTik can be the AP or to a more powerful machine connecte on the ethernet.

You can create whatever firewall rules you like and if you switch off “Default Forward” on the AP clients will not be able to bypass your firewall rules.

9

but due to your own requirements you listed, that is the only one that fufills all of your needs. you said you need a public IP to each customer, that rules out NAT, and basically only leaves WDS coupled with either DHCP or PPPoE, or a really complex setup where each CPE runs OSPF and assigns IPs via DHCP or Staticly to the E1 interface, because there is no way to bridge wlan1 and ether1 using mikrotik in station mode. (having a MAC NAT option would eliminate the issue of WDS, and still leave you with either DHCP or PPPoE)

it doesn’t have less capacity, and you can easily put 80 clients to a AP. the question is what kind of latency do you want to have. We have 802.11a APs running NStream+Polling, that durring peak loads push as much as 8 to 10 mbps, with average ping times of 20ms to the CPE.

NStream + Polling ensures that each and every CPE associated has a chance to talk by polling each CPE and giving it a chance to transmit, even if the CPE isn’t passing any traffic (I think there is some optimization for low traffic units, but it’s not documented that I’m aware of, this means one customer can not hog all the bandwidth on the AP, and hidden node issues also go away). One way to think of it is that your AP is always transmitting or receiving near maximum load, even if no data is being sent by anyone, they either transmit “data”, or they transmit “no data”, either way they still transmit. This causes latency on the connection to go up, roughly proportional to the number of CPE’s associated to the AP. it’s a cost that in my opnion is far outweighed by the benefits you gain.

so again, you can put as many CPEs as you want on that AP, but only do so after testing the performance impact and determine what an accecptable latency to the CPE is given your network requirements.

we route (via pppoe and wds, exactly as I described previously) public IPs and 3mbps of bandwidth to each customer, and based on our performance specifications and requirements, 25-30 CPEs to an AP is the maximum we allow (until v3 comes out with the new NStreamX, then hopefully we can nearly double that). You are free to determine your own maximum based on your situation, whatever you decide, please post on here for others to use as a basis for their future deployments.

10

And never did we see any more posts??
Its almost 4 years now since this last topic had a post, how was this network now finally set-up?
How can I learn from how it is done with the new ROS packages?

Would be nice to find out what a “best practise” set-up is for bigger networks.

11

Some Best practices I go by when looking to scale. These are very generic and some people consider basic.

1.Router at ever pop.
2.PPPoE is good. Centralized authentication is always a good thing. Easy to enable/disable people with billing issues, viruses, etc.
3.Funneling data to a central “point” is a good thing. A good billing package will help with this. If the package can poll your devices and display everything in one place you will be saving yourself a ton of time.

12

We mix Option 2 with Option 3. Most of our clients are happy with Option 2. In the simplest config client mounts
the Antenna, plug in there PC and it runs with no configuration. DHCP client is default on windows so everything
is learned from the dhpc server of the cpe. When there is some NAT settings to do we do it remote with no
additional cost. As most customers do not even know the term NAT they are happy with this.

Customers who need a known static ip or experienced customers have to pay for a ip. Then we route a /30 to
the ethernet of the cpe and disable NAT.

We have decided not to use PPPoE due to some disadvantages:

  • single point of failure
    So be sure your PPPoE Server is rock solid or you get big trouble
  • protocol overhead
    PPPoE is just enother encapsulation the CPE, PPPoE Server has to do.
    As Bandwith increases this might increase latency, limit bandwidth
  • In some situations we want to fine tune and bind cpe’s to a dedicated sector ,dont know how to do this
    with a central pppoe Server.
  • We’ve different Uplinks in different Locations. A PPPoE-Server does not fit in this scenario as Traffic has to
    run through it.
13

OK, interesting. Special due the overhead I do not particularly like PPPoe.
So you don’t use it. But… how do you manage your clients?

Your remark “In some situations we want to fine tune and bind cpe’s to a dedicated sector ,don’t know how to do this with a central PPPoe Server.” I don’t understand? Is this because your authentication server is in the tower that also has the AP? Can you explain a bit more please?

14

We use acesslists and dhcp-server on the AP.
We have a combination of omni and sector in some cases.
With acesslist I can assign cpe to sector.
We have learned that booting an AP with multiple sectors
cpe connect to wrong sector giving bad signal without
assigning cpe to sector.

15

since this thread was resurrected…

after v3 came out and was stable, we eliminated nearly all WDS links from the network, and changed all CPEs to station-psudobridge mode. This improved performance, reduced latency, and allowed more customers per AP with better performance than with WDS.

We still left the customers running PPPoE on their own devices, so we never had to manage them, it was the customers responsibility.

IPs were always assigned dynamically via the PPPoE session, with static blocks (usually /25 or /26’s) being assigned to each tower, but the actual allocation to the customer done dynamicly (or static through radius if they requested / paid for a static)

all customer accounts were setup in radius, and the radius server handed out the bandwidth package to the PPPoE server at the time of connection.

each tower was routed via OSPF

16

pppoe server is not a single point of faiuler: you need only two ppoe servers.

17

OK, interesting.

But why the WDS/station-psudobridge? Because you want to have the client deal with the PPPoE authentication themselves?
I see no advantage in that? Some clients hardly know how to switch their PC on (really! :open_mouth: ), let alone how to setup a PPPoE interface and a login. So we still end up with the need to help many of them in arranging this.
And how to manage the client’s CPE unit? A separate network structure for this? Or just a IP address in the same network as the client’s device. I run out of IP’s double as fast this way I would think..?

And what if client has viruses or trojans? By having the whole tower bridged local network storms can arise at more ease then when the client would be at least behind a nat firewall and the network operator has more control over it? Some traffic management already should take place at this client CPE unit anyway is my opinion.

And what about if the client wants more devices on-line. How to split his account up for more user devices? You hand him out a second authentication and charge him twice? Or you divide his contractual speed over two accounts?

Is it not more easy to have the CPE become the client end border router and have all the authentication and network management be handled up to into that CPE by the operator? On the client side of the router he can have any and as many devices he’d like, they just share his contractual speeds. And if he wants a phone, I can have a tunnel from my main concentrator go into this end router where we only have to setup some interface bridging or port forwarding to reach his VOIP/SKype box?

I presume it is a more personal choice but like to see what the arguments could or can be to go your chosen route in setting up the network.
If I see how clients are connected by the ´big´providers in my previous home country (Holland): they get a router pre configured, plug it to either the cable or the phone line and on the other end they have four ports available (plus wifi) with dhcp to hook up as many devices they’d like with hardly any interaction needed from the customer. Some you as a client only have to login the router once to set your account login but on some you don’t even have to do that. Probably the mac of the device is registered on the clients account file and that works only on that specific location (house address/phone line)

I am also thinking of how to setup a separate network for people that we want to offer a VOIP solution. Even if this is Skype. But Skype phones for instance have no PPoE option, only dhcp so how am I going to set something like that up?

I’ve got plenty things to think about and work to be done… :smiley:

18

Why have the clients run the PPPoE client themselves?

  1. many clients demand they have control of the IP on their own firewall (so they can setup their own firewall rules, run IPSec VPNs etc).
  2. it requires less config on your equipment, and swapping out a damaged CPE is much easier on the tech
  3. it gives greater troubleshooting capabilites to your staff (if the PPPoE session is up and stable for the last 36 days, you have already ruled out any wireless troubles, CPE problems, and issues with their router and you can immeaditly tell that the issue is between their PC and the router.
  4. it’s just the right way to do it.

each VLAN (each AP was on a seperate VLAN) was given a /24 of private IP space and had DHCP running (done at the same mikrotik running PPPoE). the CPEs pulled DHCP from the tower, upstream the core routers in the data center were configured to run webproxy on any traffic from those private IP ranges, and redirect all WWW traffic to a page with instructions on setting up your computer or router to use PPPoE, all other traffic was blocked


On the AP default forwarding was turned off, and since each AP was on a seperate VLAN, the customer was only able to see their CPE, and the PPPoE server on the network, all other traffic was isolated from them. Additionally the AP was set with a default client TX rate of 1mbps, so the CPE limited the traffic to be uploaded to 1mbps even before hitting the air, much less getting to the simple queue on the PPPoE server.

they use a router, and hook as many devices as they want up. this was our preferred way even if they only had one device.

you add complexity to your config if you were to do that, and when clients need non-natted connections (some VPN software for example) this meathod fails, or you have to make special case configs that require additional training of your staff, and create opportunities for mistakes to be made when things are upgraded.

if you follow the K.I.S.S. model, things work much better (Keep It Stupid Simple), and no one needs to remember how things are done for each customer, as they are all configured the exact same.

If you would like me to set some of your equipment up or help more with the configs and/or meathods, or even QOS for the VoIP I’d be happy to get involved.

19

Whomever posted a comment to me on my personal website back in December asking for help regarding this (I don’t know your screen name), I apoligize, the comment was flagged as spam and I just noticed it a few minutes ago. I email’d you, but wanted to post this as well to ensure you get me email.