Community discussions

 
Gombeen666
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Tue Jun 25, 2019 5:33 pm

WISP OSPF Multi Area optimum configuration

Mon Jul 01, 2019 11:49 am

As a WISP I have setup the AP's in OSPF into multi areas rather than having the complete network on Area0 but would like ask what is optimum configuration for clients who get their public ip from PPPoE!
Should the area be stub or NSSA , OSPF Instance - distribute-default setting , redistribute-connected setting ....any advice is most welcome
 
millenium7
Member Candidate
Member Candidate
Posts: 205
Joined: Wed Mar 16, 2016 6:12 am

Re: WISP OSPF Multi Area optimum configuration

Tue Jul 02, 2019 2:28 am

Need more info on the topology
For instance are PPPoE sessions terminated closest to the customer, or are they all terminated at a central PPPoE concentrator?
If the latter are you using VPLS tunnels (or something else like EoIP?)
Do you have BGP running internally in the network?

These answers change the outcome, since VPLS for instance only works with fully defined routing end-points, meaning you need a /32 route in your table. You can't use OSPF summary routes if you want to establish VPLS tunnels outside of that area
 
Gombeen666
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Tue Jun 25, 2019 5:33 pm

Re: WISP OSPF Multi Area optimum configuration

Tue Jul 02, 2019 2:44 pm

Need more info on the topology
For instance are PPPoE sessions terminated closest to the customer, or are they all terminated at a central PPPoE concentrator?
If the latter are you using VPLS tunnels (or something else like EoIP?)
Do you have BGP running internally in the network?

These answers change the outcome, since VPLS for instance only works with fully defined routing end-points, meaning you need a /32 route in your table. You can't use OSPF summary routes if you want to establish VPLS tunnels outside of that area
We have a central PPPoE concentrator for clients PPPoE sessions,
We were using VPLS but due to OSPF state interruptions ( maybe OSPF misconfigrations!) we have reverted to using L2 bridged until we sort out OSPF ,
BGP only at CCR1009,
 
millenium7
Member Candidate
Member Candidate
Posts: 205
Joined: Wed Mar 16, 2016 6:12 am

Re: WISP OSPF Multi Area optimum configuration

Wed Jul 03, 2019 7:00 am

So you're carrying the VLAN all the way through the network?
Been there, seen that, it's really really bad and not scalable.

I would make getting extended bridges out of your network a top priority, as it becomes much harder to remove the bigger you get. VLAN's shouldn't go any further than the direct legs of the distribution router in an area. A few exceptions like a customer turns in repeater to add another one, you just throw up a powerbox or a switch where its just not viable to turn it into a new distribution point running OSPF/BGP/MPLS etc. But it shouldn't extend through your backbone between routers, that should be routed only

OSPF on MikroTik is generally stable with some caveats, i've found point-to-point network type to be the most reliable so I always use that. Even when its a multipoint (use separate VLAN's for each connecting router and make them point-to-point links). Avoid BFD if you can, but use it on the primary if there's 2 radio links for faster failover
Establish a standard L2MTU value and ensure everything meets it, test every single link incase there's a dumb switch in the way or misconfigured radio dropping packets. We use 1538 (1556 if you count ethernet header + FCS), we set all radios/switches to maximum. We set MPLS MTU to 1538 on all interfaces by default. VPLS L2MTU to 1516 to allow 2 VLANS + PPPoE header + 1500 byte packet without any fragmentation. This has given us a generally quite nice fast stable network. Certainly a much better design than EoIP or VLANs

Back on the topic of OSPF area's. How big the network? It may not be worth using a multi area design if its less than 100 routers with 500 routes. I've heard of people going much larger without a problem. You do avoid some issues like VPLS not working across summarized boundaries
Another possibility is to make every router with customers attached a PPPoE server to move the PPPoE boundary as close as possible to the customer, use OSPF for your router-router links and iBGP to carry the customer /32 routes. Let their traffic be normal IP through your network instead of PPPoE, then you can get rid of VPLS entirely and have no problems with multi-area OSPF or summarization. This is where i'd like to move our network design to
 
Gombeen666
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Tue Jun 25, 2019 5:33 pm

Re: WISP OSPF Multi Area optimum configuration

Wed Jul 03, 2019 4:48 pm

@millenium7
Many thanks for your detailed reply,
...OSPF on MikroTik is generally stable with some caveats, i've found point-to-point network type to be the most reliable so I always use that. Even when its a multipoint (use separate VLAN's for each connecting router and make them point-to-point links). Avoid BFD if you can, but use it on the primary if there's 2 radio links for faster failover..
As present I don't have any PTP failover links ! So not sure if should still use BFD!
Example on segment of the network - PTP to PTP connected to 5port HEX POE - 3 AP's connected back to HEXPOE,
for PTP to PTP (OSPF point-to-point BFD /30)
( PTP to HEXPOE port1 (OSPF broadcast /30)
(on HEXPOE port 2 AP1( OSPF area=area1111 /30 )
(on HEXPOE port 3 AP2( OSPF area=area1111 /30 )
(on HEXPOE port 4 AP2( OSPF area=area1111 /30 )
not sure what the correct OSPF Instance setting on HEXPOE should be?
"Redistribute Default Route=Never" and no to the Connected and Static routes...
...
Establish a standard L2MTU value and ensure everything meets it, test every single link incase there's a dumb switch in the way or misconfigured radio dropping packets. We use 1538 (1556 if you count ethernet header + FCS), we set all radios/switches to maximum. We set MPLS MTU to 1538 on all interfaces by default. VPLS L2MTU to 1516 to allow 2 VLANS + PPPoE header + 1500 byte packet without any fragmentation. This has given us a generally quite nice fast stable network. Certainly a much better design than EoIP or VLANs
Most of my routers have 1598 or 1600 L2MTU
...
Back on the topic of OSPF area's. How big the network? It may not be worth using a multi area design if its less than 100 routers with 500 routes. I've heard of people going much larger without a problem. You do avoid some issues like VPLS not working across summarized boundaries
Another possibility is to make every router with customers attached a PPPoE server to move the PPPoE boundary as close as possible to the customer, use OSPF for your router-router links and iBGP to carry the customer /32 routes. Let their traffic be normal IP through your network instead of PPPoE, then you can get rid of VPLS entirely and have no problems with multi-area OSPF or summarization. This is where i'd like to move our network design to
We have less than routers on the network and 800 routes !
 
millenium7
Member Candidate
Member Candidate
Posts: 205
Joined: Wed Mar 16, 2016 6:12 am

Re: WISP OSPF Multi Area optimum configuration

Thu Jul 04, 2019 9:26 am

As present I don't have any PTP failover links ! So not sure if should still use BFD!

Unfortunately MikroTik just isn't as stable as Cisco/Juniper etc. In an ideal world BFD would just work flawlessly all the time and then its an an easy answer of enable it everywhere in the network
Sometimes its not the MikroTik routers that are at fault, some radio's don't play well with BFD either
You just need to look at your network and determine the best course of action. Another big reason for us moving off PPPoE inside our backhaul is to massively reduce the downtime that can happen when a link is interrupted. PPPoE tunnels seem ok with a short interruption but say 5+ seconds nope I find they'll drop rather than resume. MikroTik routers are quick to re-establish, adding around 2-5 seconds but other vendors can be up to 2 minutes. This totally kills productivity, everyones sessions die, PBX/VoIP systems have to time out and re-register etc. Removing PPPoE means the tunnel doesn't need to come down and re-establish itself so the time to restore customer traffic is the same as the OSPF reconvergence time. Plus if the core devices fail and flip over to backup there's no need to re-establish connectivity unlike with PPPoE since it can't transfer the session from 1 device to another, but IP can just reroute (or have VRRP in place)

- At the moment we generally don't run BFD where there's a single link and use OSPF timers of 2/1/1/4 by default
- Wherever there's a backup link in place I enable BFD on the primary and leave the secondary without and default 5/1/10/40 (since the failover to the much slower backup can over saturate the link until TCP sessions slow down, 2/1/1/4 may result in missed OSPF packets and an adjacency drop)
- If there's no direct backup link but there is an alternate path through the network around to the core, i'll make a decision if fast failover or highest reliability is more appropriate and go from there

If you can't make up your mind, leave BFD on and if you havn't had an adjacency drop in 30 days you're probably fine to leave it in place. We've had some drop every few days hence we don't run it wherever its not needed


Most of my routers have 1598 or 1600 L2MTU


I don't just mean the routers, also check your radio links and switches between the routers
You can test L2MTU by making a new VLAN on the interface (i.e.ether2.50) and set L3MTU to the size you want to test -4bytes (to account for the extra VLAN header), then do a ping between them with the same packet size and tick don't fragment
I.e. we test 1538 so i'll make VLAN50 with L3MTU of 1534 and run a 1534 byte ping from BOTH sides (you may have different MTU in 1 direction). If it succeeds this means the link can carry a full 1556 L2 packet (including ethernet and FCS). Different vendors have different classification for MTU. MikroTik for instance does not include the the ethernet header (+14 bytes) or FCS trailer (+4 bytes). So a 1542 byte L2MTU in MikroTik actually means 1560 = big enough
Netonix does, so 1542 byte L2MTU means exactly 1542 = not big enough
This is why you need to test (but there is no harm in setting L2MTU as high as possible)


We have less than routers on the network and 800 routes !

Probably still fine. What's your congervence time when you stop OSPF and start the instance again? If the time is low enough, don't worry about it. HEX has more than enough grunt

or PTP to PTP (OSPF point-to-point BFD /30)
( PTP to HEXPOE port1 (OSPF broadcast /30)
(on HEXPOE port 2 AP1( OSPF area=area1111 /30 )
(on HEXPOE port 3 AP2( OSPF area=area1111 /30 )
(on HEXPOE port 4 AP2( OSPF area=area1111 /30 )
not sure what the correct OSPF Instance setting on HEXPOE should be?

'Instance' is a copy of the OSPF software running. When you make another instance they are totally separate from each other. 99% of the time you never need to make another instance, you can run as many area's as you want in the same instance thats fine. Reason you may want to run a different instance is when you have separate networks with overlapping area numbers i.e. 2 separate Area0 networks. You can run a separate instance for each, then redistribute some routes between them rather than joining them together. Or if you have a customer/company with doing OSPF routing with you, you can specify a different routing table (CustomerA) so their routes don't end up in your main routing table, and you aren't leaking your routes to them
 
Gombeen666
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Tue Jun 25, 2019 5:33 pm

Re: WISP OSPF Multi Area optimum configuration

Fri Jul 05, 2019 4:33 pm

@millenium7 Can I once again thank you for taking time to answer and offer very good advise!
Another big reason for us moving off PPPoE inside our backhaul is to massively reduce the downtime that can happen when a link is interrupted. PPPoE tunnels seem ok with a short interruption but say 5+ seconds nope I find they'll drop rather than resume. MikroTik routers are quick to re-establish, adding around 2-5 seconds but other vendors can be up to 2 minutes. This totally kills productivity, everyones sessions die, PBX/VoIP systems have to time out and re-register etc. Removing PPPoE means the tunnel doesn't need to come down and re-establish itself so the time to restore customer traffic is the same as the OSPF reconvergence time. Plus if the core devices fail and flip over to backup there's no need to re-establish connectivity unlike with PPPoE since it can't transfer the session from 1 device to another, but IP can just reroute (or have VRRP in place)
What did you use instead of PPPoE or did you move PPPoE to each tower?
- At the moment we generally don't run BFD where there's a single link and use OSPF timers of 2/1/1/4 by default
- Wherever there's a backup link in place I enable BFD on the primary and leave the secondary without and default 5/1/10/40 (since the failover to the much slower backup can over saturate the link until TCP sessions slow down, 2/1/1/4 may result in missed OSPF packets and an adjacency drop)
- If there's no direct backup link but there is an alternate path through the network around to the core, i'll make a decision if fast failover or highest reliability is more appropriate and go from there
Must check this it out!
I don't just mean the routers, also check your radio links and switches between the routers
You can test L2MTU by making a new VLAN on the interface (i.e.ether2.50) and set L3MTU to the size you want to test -4bytes (to account for the extra VLAN header), then do a ping between them with the same packet size and tick don't fragment
I.e. we test 1538 so i'll make VLAN50 with L3MTU of 1534 and run a 1534 byte ping from BOTH sides (you may have different MTU in 1 direction). If it succeeds this means the link can carry a full 1556 L2 packet (including ethernet and FCS). Different vendors have different classification for MTU. MikroTik for instance does not include the the ethernet header (+14 bytes) or FCS trailer (+4 bytes). So a 1542 byte L2MTU in MikroTik actually means 1560 = big enough
Netonix does, so 1542 byte L2MTU means exactly 1542 = not big enough
This is why you need to test (but there is no harm in setting L2MTU as high as possible)
Excellent advice!
 
Gombeen666
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Tue Jun 25, 2019 5:33 pm

Re: WISP OSPF Multi Area optimum configuration

Fri Jul 05, 2019 6:20 pm

......OSPF on MikroTik is generally stable with some caveats, i've found point-to-point network type to be the most reliable so I always use that. Even when its a multipoint (use separate VLAN's for each connecting router and make them point-to-point links).....
Does this mean creating creating VLAN's from a port of the HexPOE to an AP and setting it as OSPF PTP ?,

Already I have the radio PTP as OSPF PTP + now from ether side of PTP to HexPOE as OSPF PTP ?

What priority setting be used and where? should the number be be increasing from AP back to PTP at backhaul site , example

Backhaul ( Priority 10) - Ether PTP (Priority 9) Wireless (Priority 8 ) ----PTP----Wireless (Priority 7) Ether (Priority 6) ----HexPOE --- Ether port 2 (Priority 5)
Ether3 (Priority 4) ---AP1 (Priority 3),
Ether4 (Priority 4).....AP2 (Priority 3),
Ether5 (Priority 4).....AP3 (Priority 3)
 
millenium7
Member Candidate
Member Candidate
Posts: 205
Joined: Wed Mar 16, 2016 6:12 am

Re: WISP OSPF Multi Area optimum configuration

Sun Jul 28, 2019 11:27 am

Late response and I'm on a phone so I won't quote specific sections of text but....

PPPoE is staying in place in our network. Because Mikrotik doesnt support /32 DHCP address assignment for customers properly. Or rather the actual assignment of the address works ok but the router doesn't add a route to the IP address that got assigned to the customer into the routing table
The short version is you can't use DHCP over a routed network. You'd either have to use /30s everywhere for each customer (wasting a huge amount of IP addresses) or tunnel everything back into a common layer2 bridge somewhere which ultimately just doesn't make any sense and provides zero benefit over PPPoE
The biggest selling point of using DHCP customer assignment is to have a fully routed network and you just can't do it properly. Lot of headache for no benefit. Hence were still going to use PPPoE for customers, but move that PPPoE session as close to the customer as possible. I.e. previously if the topology was

Customer->RouterA->RouterB->RouterC->Internet

All customers would terminate their PPPoE session at RouterC and get it there via VPLS. Now that customer would have their PPPoE session terminate at RouterA. That usually means just 1 radio link to go through to reach A. Customers connected to RouterB would terminate there, etc
From them on it'll just be normal routed traffic. The big benefit for us is if the link from B to C goes down for say 5 seconds then comes back up. The PPPoE session remains connected since it terminates at A/B so as soon as it comes back up traffic keeps flowing, instead of having to wait for VPLS to connect then PPPoE to timeout and reconnect again (which can be minutes for some customer routers). As well as if there's a redundant internet link out i.e. B then customer traffic can dynamically route out there
 
Gombeen666
Frequent Visitor
Frequent Visitor
Topic Author
Posts: 54
Joined: Tue Jun 25, 2019 5:33 pm

Re: WISP OSPF Multi Area optimum configuration

Sun Jul 28, 2019 3:38 pm

........................ or tunnel everything back into a common layer2 bridge somewhere which ultimately just doesn't make any sense and provides zero benefit over PPPoE
The reason we moved back to L2 bridged was because we reckon our OSPF configuration was not correct and was causing intermittent AP lockup's and when compared to one site
that was L2 bridged that site had very little service interruptions?
Also maybe our OSPF interface like the management VLAN should be "Passive"
....Hence were still going to use PPPoE for customers, but move that PPPoE session as close to the customer as possible.....
Like having a PPPoE server at each mast site!
All customers would terminate their PPPoE session at RouterC and get it there via VPLS. Now that customer would have their PPPoE session terminate at RouterA. That usually means just 1 radio link to go through to reach A. Customers connected to RouterB would terminate there, etc
From them on it'll just be normal routed traffic. The big benefit for us is if the link from B to C goes down for say 5 seconds then comes back up. The PPPoE session remains connected since it terminates at A/B so as soon as it comes back up traffic keeps flowing, instead of having to wait for VPLS to connect then PPPoE to timeout and reconnect again (which can be minutes for some customer routers). As well as if there's a redundant internet link out i.e. B then customer traffic can dynamically route out there
I am starting to like and understand the benefit of having the PPPoE servers close to the customers also would I be correct in that once customers terminate their session to the PPPoE server at 1480 MTU that then a higher MTU size can be used back the network to increase throughput?
 
millenium7
Member Candidate
Member Candidate
Posts: 205
Joined: Wed Mar 16, 2016 6:12 am

Re: WISP OSPF Multi Area optimum configuration

Mon Jul 29, 2019 1:28 am

MTU is end to end. If the customers session is 1480 they can never send or receive anything larger without fragmentation anyway. There's no reason you can't use 1500 byte PPPoE sessions though, as long as the equipment to the customer supports large enough L2MTU. You set your PPPoE server to Max-MTU=1500 and Max-MRU=1500
Then if customer router supports it (like Mikrotik) you set the MTU of the PPPoE client to 1500
Packets sent to/from the customer directly will be 1508 but the PPPoE header will be stripped off and become 1500 after the first router

With OSPF it's good practice to manually add the interface 'all' and then tick passive. Then for all other interfaces you manually add them without passive. You don't want to be advertising OSPF towards potentially unknown links like VPN/PPPoE/random ethernet ports
Always use link type point to point to avoid issues. Priority value doesn't matter on PTP links
If a router is a multipoint to other routers on 1 port, create 2 VLANs for that port i.e. ether2.201 - RouterB and ether2.202 - RouterC
And the same VLAN on the other router, and again use point to point.
I'm actually using VLANs everywhere now, even just for point to point radio links because it allows for L2 QoS tags to be applied

Who is online

Users browsing this forum: No registered users and 5 guests