AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

maxslug · Fri Sep 18, 2020 7:05 am

EDIT: 2024-04-13 - Upgrade to RouterOS 7.x and wireless-qcom-ac.

Hi All,

I'm new at Mikrotik gear and thought it would be good to post my full configuration of my network. Hopefully this will help the next person along, and maybe I'll receive some sage advise from the gurus on here. All feedback welcome.

The config files are here https://github.com/maxslug/mikrotik_maxslug where I'll try to keep them up to date with the changes as I learn more from you all.

Router
https://github.com/maxslug/mikrotik_max ... mikro1.rsc

Switches
https://github.com/maxslug/mikrotik_max ... er/sw1.rsc
https://github.com/maxslug/mikrotik_max ... er/sw2.rsc

WAPs
https://github.com/maxslug/mikrotik_max ... r/wap1.rsc
https://github.com/maxslug/mikrotik_max ... r/wap2.rsc
https://github.com/maxslug/mikrotik_max ... r/wap3.rsc

Migration from RouterOS 6.x to RouterOS 7.x
The older RouterOS 6.x / old CapsMAN files are at https://github.com/maxslug/mikrotik_max ... outer-os-6.

This is the diff of changes need to upgrade form RouterOS 6.x to RouterOS 7.x, as well as migrate to wireless-qcom-ac and the new CapsMAN:
https://github.com/maxslug/mikrotik_max ... 55d040f262

mermaid-diagram-2024-04-13-171315.png

(SSIDs only shown on one AP and one Switch, not on all, just to not clutter the diagram)

Thank you to all the mikrotik forum posters for all this knowledge and hard work, especially `pcunite`!

Design Goals

WAN

Dual ISPs with auto-failover
Complete removal of AT&T router ("residential gateway") from the picture

L1

Disaggregation of routing and wifi into separate solutions
Switched managed Ethernet
Redundancy
Power over Ethernet to allow centralized UPS

L2 / L3

VLAN separation of Guest, Primary, IOT, Neighbor, and VOIP networks

Wireless

Centrally managed access points
Roaming / Hand-off improvements
Higher overlapping coverage at lower radio power rates

L4+

Port Forwarding over VLAN
Secure DNS

Network Design

These are notes to go along with the config files

Inventory

1 x Mikrotik RB5009UPr+s+IN Router using RouterOS 7.15beta8 [arm64]
2 x Mikrotik CRS109-8G-1S-2HnD Router/Switch/APs running RouterOS 7.15beta8 [mips]
3 x Mikrotik cAP AC (RBcAPGi-5acD2nD) using RouterOS 7.15beta8 + wireless-qcom-ac [arm]

VLANs

VLAN  |IP                |Usage
------|------------------|-----------------
 100  |192.168.100.0/24  |Base / Management
 200  |192.168.120.0/24  |Normal LAN 
 300  |192.168.130.0/24  |Guest / IOT 
 400  |192.168.140.0/24  |VOIP
 500  |192.168.150.0/24  |Neighbor

For each subnet addresses `.1` through `.39` are reserved for static IP assignment. `.1` is the router.
The WAN ports are not on VLANs
Once configured, you will need to make a port be on `VLAN 100` to use WinBox.

maxslug · Fri Sep 18, 2020 7:06 am

Update: 2024-04-13 for wireless-qcom-ac

Router

`192.168.100.1`
The EAP Authentication protocol requires a set system clock. DHCP requires EAP. NTP requies DHCP. This means you can't set the clock over the internet because of a chicken-n-egg problem. Make sure `mikro1.rsc` is modified with the current time before programming it. Or, if you have a local NTP server, use that.
You will need to coax your authentication keys out of your AT&T gateway so you can run in `supplicant mode`.
DNS is setup to use DNS over HTTP (DOH) which requires some certificates and hurdles.

Switches

I was really only interested in an 8-port managed GigE switch, but for the same price these units include a 2G WiFi radio.

`192.168.100.2`
`192.168.100.3`
The radio in the switches are not part of CapsMAN
I create a "backup" SSID out of these that should work if I need to hookup the old router, or if for some other reason CapsMAN fails.
One of the APs is chained off of `sw1` due to physical topology'
Uplink is ether0 to because that is the POE IN port

Access Points

`192.168.100.11`
`192.168.100.12` (config not included)
`192.168.100.13` (config not included)

Despite what the Mikrotik documentation says, you cannot fully remotely provision these. You will need to create a config file and add it to the AP.
After that, the wireless definitions will be automatic, but not the base config and security!

`/system reset-configuration run-after-reset=wap.rsc` does not seem to work. I still had to manually load the file after reset
Resetting into CAP mode (hold reset button till it gets to it's second mode after blinking) is a better starting point
Certificates will be auto-provisioned by CapsMAN
Whenever you do a `/system reset-configuration` on the router, it doesn't have the ability of saving the certificate keys, so unless you are managing your certificates outside of RouterOS, you'll need to clear the certs on EACH access point
- `/interface wireless cap set enabled=no`
- `/certificate print`
- `/certificate remove numbers=1,0`
- `/interface wireless cap set enabled=yes`
I scripted the mode button so that it will toggle the LEDs between "always on" and "turn off after 1h"

cAP AP Wireless Features

cAP (and other Mikrotik Qualcomm-based 802.11ac products) recently (as of 2024) got full support for
WiFi 5 Wave 2! YOU ROCK MIKROTIK! Thanks for improving an existing product instead of just
moving on. They now support software-based features like MIMO, DFS, Beam Forming,
Handoff Protocols, Spectral Scan etc. However in order to use these features, and to make them
compatible with 802.1ax devices, you need to run a new driver, and a new capsman. The new driver
is called `wifi-qcom-ac` and the new capsman is in `/interface wifi capsman` and `/interface wifi`.

One hiccup I did find is that the new CapsMAN does not play nice with the older capAC devices when
it comes to VLANs. You have to statically config some things, and it's all a bit kludgy. I could
not get three SSIDs on three VLANs working like I had in RouterOS 6.x. So instead I removed one and
settled for two with the work-arounds.

Please see router-os-6/ for scripts using the older driver.

maxslug · Fri Sep 18, 2020 7:07 am

802.11ac Band Planning

maxslug_802.11ac_5G_channels.png

Here is a diagram I put together to understand the 802.11ac channel assignment

DFS is the middle part of the spectral sandwich which requires fancy driver support and regulatory signoff
802.11ac requires 80MHz channels, made up of 4 x 20MHz channels
For any given 80 MHz chunk, there are 4 possible assignments, depending on which one you make the control channel
- This is what gives you the `Ceee` `eCee` `eeCe` `eeeC` "walking ones" pattern. I tried to depict this above
- I only defined the channels that worked for my region
I use WiFi analyzer (Windows, Android) to do a survey of least-busy bands at each AP physical location

Here were some other helpful diagrams I found from the references below

802.11ac channels2.png

802.11ac channels.png

802.11ac channels different primaries.png

ac1200.png

mcs modes ac1200 867.png

maxslug · Fri Sep 18, 2020 7:08 am

References

These are not in any particular order, but all my knowledge came from these, so
Thank You!!

2024 RouterOS 7.x upgrade / wireless-qcom-ac / new CAPsMAN

config

Security

https://mum.mikrotik.com/presentations/ ... 151116.pdf

CAPSman

Wifi Channel Planning / 802.11ac / CapsMAN

band steering

vlans

maxslug · Fri Sep 18, 2020 7:08 am

Software

This install required 3 versions of RouterOS:

`arm64` - `RB5009` Router. No wireless driver
`arm` - `capAC` WAP. `wifi-qcom-ac` driver
`mips` - `crs109`. `wireless` driver

winbox on MacOS
I like this way of using / installing it better than the suggested:

brew tap homebrew/cask-versions                                                                                                                                
brew install --cask --no-quarantine wine-devel
killall wineserver                                                                                                                                                       
wine64 winbox64.exe

maxslug · Fri Sep 18, 2020 7:26 am

Remaining mysteries and TODO

wireless-qcom-ac on a cAPac, new CapsMAN, 3 SSIDs on 3 VLANs. The 2 VLAN solution is already a kludge.

maxslug · Fri Sep 18, 2020 7:05 pm

and already one pull request, thanks Alibloke!

anav · Sat Sep 19, 2020 5:49 pm

I know that on the bridge vlan config, the untagged are automatically added but I like to see them in the config so that I can track them visually. To me they appear to be missing and its mildly confusing LOL. OP preference I suppose, no argument.

On the input chain I do have some points.
add chain=forward? action=fasttrack-connection connection-state=established,related
add chain=input action=accept connection-state=established,related,untracked
add chain=input action=accept in-interface-list=VLAN comment="Allow VLANs"
add chain=input action=accept in-interface-list=BASE comment="Allow Base_Vlan Full Access"
add chain=input action=accept protocol=icmp
add chain=input action=drop in-interface-list=!VLAN
add chain=input action=drop connection-state=invalid
add chain=input action=drop

I would change to the following order and content (not sure why your forward chain fastrack rule is in this set but so be it)
TO
add chain=input action=accept connection-state=established,related,untracked
add chain=input action=drop connection-state=invalid
add chain=input action=accept protocol=icmp
add chain=input action=accept in-interface-list=LAN source-address-list=adminaccess ***
add chain=input action=accept any services that users need such as DNS services. @@@
add chain=input action=drop

Discussion: There is absolutely no need on security principles to let any user on any vlan full access to the router. Limit it only to the admin.
Any services that users may need, provide access to the router for those services.
As for the last rule, thats what I use, so your third last rule - drop anything not coming from the LAN, is useless as the last rule already covers that and more.

***
/ip firewall address list
add ipaddress of admin desktop list=adminaccess
add ipaddress of admin laptop list=adminaccess
etc.......

@@@ example......
add action=accept chain=input comment="Allow LAN DNS queries-UDP" \
connection-state="" dst-port=53 in-interface-list=LAN protocol=udp
add action=accept chain=input comment="Allow LAN DNS queries - TCP" dst-port=\
53 in-interface-list=LAN protocol=tcp

Okay I do note that you dont have a LAN list, needs to be added.
options are to identify the bridge as being on the LAN or
each VLAN on the LAN (what I would do).

On the forward chain..........

FROM
add chain=forward action=accept connection-state=established,related,untracked
add chain=forward action=accept ipsec-policy=in,ipsec
add chain=forward action=accept ipsec-policy=out,ipsec
add chain=forward action=accept connection-state=new in-interface-list=VLAN out-interface-list=WAN
add chain=forward action=accept connection-state=new in-interface-list=BASE out-interface-list=WAN
add chain=forward action=accept connection-nat-state=dstnat comment="For port forwarding to VLANs"
add chain=forward action=drop connection-state=invalid
add chain=forward action=drop connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add chain=forward action=drop in-interface=ether1 log=yes log-prefix=!public src-address-list=not_in_internet comment="Drop incoming from internet which is not public IP"
#add chain=forward action=drop in-interface=ether2 log=yes log-prefix=!public src-address-list=not_in_internet comment="Drop incoming from internet which is not public IP"
add chain=forward action=drop

TO
add chain=forward action=accept ipsec-policy=in,ipsec
add chain=forward action=accept ipsec-policy=out,ipsec
add chain=forward action=fasttrack-connection connection-state=established,related (for accuracy place it here)
add chain=forward action=accept connection-state=established,related,untracked
add chain=forward action=drop connection-state=invalid
add chain=forward action=accept connection-state=new in-interface-list=VLAN out-interface-list=WAN
add chain=forward action=accept connection-state=new in-interface-list=BASE out-interface-list=WAN
add chain=forward action=accept connection-nat-state=dstnat connection-state=new in-interface-list=WAN
add chain=forward action=drop

Discussion, ipsec rules should go before fastrack. The port forwarding rules and extra bloat blocking were confusing at best and way to complex. Simplify to a clear allow port forwarding rule.
The drop all rule covers everything else not explicitly permitted so you are done!! :-)

maxslug · Sun Sep 20, 2020 8:55 am

Thanks anav, I really appreciate the review and feedback!

I know that on the bridge vlan config, the untagged are automatically added but I like to see them in the config so that I can track them visually. To me they appear to be missing and its mildly confusing LOL. OP preference I suppose, no argument.

I went back and forth on this one. Future proof the config files but allowing for new default RouterOS behavior, or future proof the config files by being explicit against future RouterOS default behavior changes? In the end I went with "fewest lines possible".

On the input chain I do have some points.
Discussion: There is absolutely no need on security principles to let any user on any vlan full access to the router. Limit it only to the admin.
Any services that users may need, provide access to the router for those services.
As for the last rule, thats what I use, so your third last rule - drop anything not coming from the LAN, is useless as the last rule already covers that and more.

I understand your advise, and appreciate it, but chose not to take it. I have the management VLAN (100) that provides the access control. When I attach to it with a laptop or my desktop (through a second NIC) I'm not going to have static IPs. So it's easier to not have to deal with an IP list. Instead I have an SSID (hidden) for the management, or switch ports with elevated priviledge that are not normally used. I'm not worried about physical security. I also am aware hidden SSID is obfuscation at best, I'll be relying on strong passwords.

Discussion, ipsec rules should go before fastrack. The port forwarding rules and extra bloat blocking were confusing at best and way to complex. Simplify to a clear allow port forwarding rule. The drop all rule covers everything else not explicitly permitted so you are done!! :-)

Done, thanks! I wasn't sure at all where to put the fastrack and IP rules, so I stuck the fast track at the beginning. I didn't add a "LAN" interface name group, but stuck with the example "VLAN" and "BASE" groups, where "VLAN" really is the LAN group and "BASE" is the Adminstrative VLAN.

Here's where I ended up, hopefully I covered everything.

ip firewall filter
# input services the local router only at all the .1 addresses
add chain=input   action=accept               connection-state=established,related,untracked
add chain=input   action=drop                 connection-state=invalid
add chain=input   action=accept               protocol=icmp
add chain=input   action=accept               dst-port=53 in-interface-list=VLAN protocol=udp
add chain=input   action=accept               dst-port=53 in-interface-list=VLAN protocol=tcp
add chain=input   action=accept               in-interface-list=BASE comment="Allow Management Vlan Full Access"
add chain=input   action=drop

# forward services the NAT / Routing
add chain=forward action=accept               ipsec-policy=in,ipsec
add chain=forward action=accept               ipsec-policy=out,ipsec
add chain=forward action=fasttrack-connection connection-state=established,related
add chain=forward action=accept               connection-state=established,related,untracked
add chain=forward action=drop                 connection-state=invalid
add chain=forward action=accept               connection-state=new in-interface-list=VLAN out-interface-list=WAN
add chain=forward action=accept               connection-state=new in-interface-list=BASE out-interface-list=WAN
add chain=forward action=accept               connection-nat-state=dstnat comment="For port forwarding to VLANs"
add chain=forward action=drop

Thanks!
-m

anav · Sun Sep 20, 2020 2:28 pm

Looks great!

maxslug · Tue Sep 22, 2020 7:11 pm

added routing / firewall handling of connections to the plex server from the guest and neighbor vlans.

maxslug · Wed Sep 23, 2020 12:39 am

added printing between vlans. Looks like bonjour / zeroconf / airprint will not forward across subnets / vlans, so only one vlan will get auto-discovery of the printer.

anav · Wed Sep 23, 2020 5:12 am

Dont know about proprietary crap but I use firewall rules to allow users on vlans to access a printer on another vlan.

xvo · Wed Sep 23, 2020 9:52 am

Dont know about proprietary crap but I use firewall rules to allow users on vlans to access a printer on another vlan.

Firewall won't help you in case of discovery protocols that rely on broadcasts (and are supposed to work inside one broadcast domain), no matter if they are open or proprietary.
Only some kind of "proxy" or "reflector".

pcunite · Wed Sep 23, 2020 4:46 pm

Appreciate the great effort here. You have put the work into this. The information about how to configure these devices needs to be more open, more clear, and easily digestible. This will help to move that forward. An entire topic should be spent on Service Discovery between VLANs, I should think.

maxslug · Tue Oct 06, 2020 7:08 am

Thanks pcunite!
I figured out the Hairpin NAT settings and added it here : viewtopic.php?f=2&t=162287&p=820788#p820788

maxslug · Sun Nov 01, 2020 1:07 am

pushed the hairpin NAT stuff to the git repo

maxslug · Sun Nov 15, 2020 11:35 pm

Switched to Google DOH, Cloudflare was buggin' out on me. Also added auto-update of cert and dns cache flush scripts from IYARINDRA.

maxslug · Sun Jan 17, 2021 2:09 am

Updates:

DOH is a lot more stable w/ 6.48
Moved switch port uplink to ethernet0 since that's the POE in port. Recently bought a big UPS, so moved everything over to POE to centralize the power point.

maxslug · Tue Mar 19, 2024 9:39 pm

Welp... my RB4011 bit the dust :/

I got myself a RB5009 and will be attempting to reproduce this setup. I imagine some things will change as I migrate to RouterOS 7 and update the switches and APs too.

Stay tuned...

Amm0 · Tue Mar 19, 2024 10:15 pm

I recall from @pcunite's thread on AT&T FTTH that V7 worked with the 802.1x stuff. On the AT&T front with the RB5009... the mystery is there some GPON SFP that can replace the AT&T one.

But... it's the CAPsMAN stuff where your decisions get tougher. You can keep APs at V6 and use RB5009 as the controller – which be recommendation at this point in time.

If you upgrade upgrade cAPac to V7... then you have more choice:
1. to keep using current "ac" drivers and "old" CAPsMAN...
2. or, using wifi-qcom-qc drivers* on them but that entails re-doing all the Wi-Fi stuff and using "new" CAPsMAN

* which with 16MB flash, the recent stable are awful close to not fitting on flash

maxslug · Sun Apr 14, 2024 3:35 am

Well, I bit the bullet and finally did the upgrade. It was a serious pain in the ass and took a lot of hours. Good news is the EAP supplicant stuff just worked, no fiddling.

The new CapsMAN with the old 802.11ac devices and new wireless-qcom-ac drivers are a very brittle combo.The documentation is pretty good, but I couldn't replicate my RouterOS 6.x setup of 3 SSIDs on 3 VLANs w/ CapsMAN.

See https://github.com/maxslug/mikrotik_max ... 55d040f262 for a diff of what I changed to upgrade, or see https://github.com/maxslug/mikrotik_maxslug for the full config.

-m

Amm0 · Sun Apr 14, 2024 5:03 am

It was a serious pain in the ass and took a lot of hours. Good news is the EAP supplicant stuff just worked, no fiddling.

Oh I'm sure. But the new drivers likely worth the effort. Used the wifiwave2 (now wifi-qcom-ac) on Audiences for long while, it was night-and-day difference.

Welp... my RB4011 bit the dust :/

How did it die?

anav · Sun Apr 14, 2024 4:29 pm

If you dont have AX wifi, then adding capsman IMHO is a waste of time and config space. Capacs take minutes to setup and dont change very often, so capsman is overhead and complexity for in reality very little if any gain.

Amm0 · Sun Apr 14, 2024 4:43 pm

Capacs take minutes to setup and dont change very often

Most Wi-Fi APs have some central manager, so it's not unreasonable to want that on Mikrotik. It's the view of all APs in one place that's useful of CAPsMAN.

Or if you want to make a change to SSID/password, etc. are all easier down the road too.

No one is arguing it's complex & PITA to setup

, but once configured, it works. OP did not claim it was easy & documented it all on GitHub as a recipe.

anav · Sun Apr 14, 2024 4:56 pm

You are forgetting the point, the capacs I set manually, are RARELY CHANGED, so central managment overhead and config is wasted.
Plus its far easier to setup.
I can guarantee you, that the time he took to setup capsman on the controller and on each capac, was far longer than anytime I will ever spend managing my capacs manually.

The only reason to use capsman is two fold.
a. You have all AX devices and thus can really benefit from all standards to make handover smoother.
b. You need to access wifi settings to block users from each other within the same WLAN or to block wifi users VLAN to access other wired same vlan users.

erlinden · Mon Apr 15, 2024 1:49 pm

The only reason to use capsman is two fold.
a. You have all AX devices and thus can really benefit from all standards to make handover smoother.

Can you change that to You have all wifi-qcom(-ac) devices? I love the roaming experience with my cAP XL ac and wAP ac!

anav · Mon Apr 15, 2024 2:18 pm

Sounds like a cult LOL.

AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

Re: AT&T FTTH, VLANs, CapsMAN Full Config

Re: AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

Re: AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

Re: AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

Re: AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

Re: AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

Who is online