AT&T FTTH, VLANs, CapsMAN Full Config (RouterOS 7 Updated)

maxslug · September 18, 2020, 4:05am

EDIT: 2024-04-13 - Upgrade to RouterOS 7.x and wireless-qcom-ac.

Hi All,

I’m new at Mikrotik gear and thought it would be good to post my full configuration of my network. Hopefully this will help the next person along, and maybe I’ll receive some sage advise from the gurus on here. All feedback welcome.

The config files are here https://github.com/maxslug/mikrotik_maxslug where I’ll try to keep them up to date with the changes as I learn more from you all.

Router
https://github.com/maxslug/mikrotik_maxslug/blob/master/mikro1.rsc

Switches
https://github.com/maxslug/mikrotik_maxslug/blob/master/sw1.rsc
https://github.com/maxslug/mikrotik_maxslug/blob/master/sw2.rsc

WAPs
https://github.com/maxslug/mikrotik_maxslug/blob/master/wap1.rsc
https://github.com/maxslug/mikrotik_maxslug/blob/master/wap2.rsc
https://github.com/maxslug/mikrotik_maxslug/blob/master/wap3.rsc

Migration from RouterOS 6.x to RouterOS 7.x
The older RouterOS 6.x / old CapsMAN files are at https://github.com/maxslug/mikrotik_maxslug/tree/master/router-os-6.

This is the diff of changes need to upgrade form RouterOS 6.x to RouterOS 7.x, as well as migrate to wireless-qcom-ac and the new CapsMAN:
https://github.com/maxslug/mikrotik_maxslug/commit/fdb2b360eb1f53512e3792e139eefa55d040f262

(SSIDs only shown on one AP and one Switch, not on all, just to not clutter the diagram)

Thank you to all the mikrotik forum posters for all this knowledge and hard work, especially pcunite!

Design Goals

WAN

Dual ISPs with auto-failover
Complete removal of AT&T router (“residential gateway”) from the picture

L1

Disaggregation of routing and wifi into separate solutions
Switched managed Ethernet
Redundancy
Power over Ethernet to allow centralized UPS

L2 / L3

VLAN separation of Guest, Primary, IOT, Neighbor, and VOIP networks

Wireless

Centrally managed access points
Roaming / Hand-off improvements
Higher overlapping coverage at lower radio power rates

L4+

Port Forwarding over VLAN
Secure DNS

Network Design

These are notes to go along with the config files

Inventory

1 x Mikrotik RB5009UPr+s+IN Router using RouterOS 7.15beta8 [arm64]
2 x Mikrotik CRS109-8G-1S-2HnD Router/Switch/APs running RouterOS 7.15beta8 [mips]
3 x Mikrotik cAP AC (RBcAPGi-5acD2nD) using RouterOS 7.15beta8 + wireless-qcom-ac [arm]

VLANs

VLAN  |IP                |Usage
------|------------------|-----------------
 100  |192.168.100.0/24  |Base / Management
 200  |192.168.120.0/24  |Normal LAN 
 300  |192.168.130.0/24  |Guest / IOT 
 400  |192.168.140.0/24  |VOIP
 500  |192.168.150.0/24  |Neighbor

For each subnet addresses .1 through .39 are reserved for static IP assignment. .1 is the router.
The WAN ports are not on VLANs
Once configured, you will need to make a port be on VLAN 100 to use WinBox.

maxslug · September 18, 2020, 4:06am

Update: 2024-04-13 for wireless-qcom-ac

Router

192.168.100.1
The EAP Authentication protocol requires a set system clock. DHCP requires EAP. NTP requies DHCP. This means you can’t set the clock over the internet because of a chicken-n-egg problem. Make sure mikro1.rsc is modified with the current time before programming it. Or, if you have a local NTP server, use that.
You will need to coax your authentication keys out of your AT&T gateway so you can run in supplicant mode.
DNS is setup to use DNS over HTTP (DOH) which requires some certificates and hurdles.

Switches

I was really only interested in an 8-port managed GigE switch, but for the same price these units include a 2G WiFi radio.

192.168.100.2
192.168.100.3
The radio in the switches are not part of CapsMAN
I create a “backup” SSID out of these that should work if I need to hookup the old router, or if for some other reason CapsMAN fails.
One of the APs is chained off of sw1 due to physical topology’
Uplink is ether0 to because that is the POE IN port

Access Points

192.168.100.11
192.168.100.12 (config not included)
192.168.100.13 (config not included)

Despite what the Mikrotik documentation says, you cannot fully remotely provision these. You will need to create a config file and add it to the AP.
After that, the wireless definitions will be automatic, but not the base config and security!

/system reset-configuration run-after-reset=wap.rsc does not seem to work. I still had to manually load the file after reset
Resetting into CAP mode (hold reset button till it gets to it’s second mode after blinking) is a better starting point
Certificates will be auto-provisioned by CapsMAN
Whenever you do a /system reset-configuration on the router, it doesn’t have the ability of saving the certificate keys, so unless you are managing your certificates outside of RouterOS, you’ll need to clear the certs on EACH access point
/interface wireless cap set enabled=no
- /certificate print
- /certificate remove numbers=1,0
- /interface wireless cap set enabled=yes
I scripted the mode button so that it will toggle the LEDs between “always on” and “turn off after 1h”

cAP AP Wireless Features

cAP (and other Mikrotik Qualcomm-based 802.11ac products) recently (as of 2024) got full support for
WiFi 5 Wave 2! YOU ROCK MIKROTIK! Thanks for improving an existing product instead of just
moving on. They now support software-based features like MIMO, DFS, Beam Forming,
Handoff Protocols, Spectral Scan etc. However in order to use these features, and to make them
compatible with 802.1ax devices, you need to run a new driver, and a new capsman. The new driver
is called wifi-qcom-ac and the new capsman is in /interface wifi capsman and /interface wifi.

One hiccup I did find is that the new CapsMAN does not play nice with the older capAC devices when
it comes to VLANs. You have to statically config some things, and it’s all a bit kludgy. I could
not get three SSIDs on three VLANs working like I had in RouterOS 6.x. So instead I removed one and
settled for two with the work-arounds.

Please see router-os-6/ for scripts using the older driver.

maxslug · September 18, 2020, 4:07am

802.11ac Band Planning
maxslug_802.11ac_5G_channels.png
Here is a diagram I put together to understand the 802.11ac channel assignment

DFS is the middle part of the spectral sandwich which requires fancy driver support and regulatory signoff
802.11ac requires 80MHz channels, made up of 4 x 20MHz channels
For any given 80 MHz chunk, there are 4 possible assignments, depending on which one you make the control channel
This is what gives you the Ceee eCee eeCe eeeC “walking ones” pattern. I tried to depict this above
- I only defined the channels that worked for my region
I use WiFi analyzer (Windows, Android) to do a survey of least-busy bands at each AP physical location

Here were some other helpful diagrams I found from the references below

802.11ac channels different primaries.png
802.11ac channels.png
802.11ac channels2.png
mcs modes ac1200 867.png

maxslug · September 18, 2020, 4:08am

References

These are not in any particular order, but all my knowledge came from these, so
Thank You!!

2024 RouterOS 7.x upgrade / wireless-qcom-ac / new CAPsMAN

config

Security

https://mum.mikrotik.com/presentations/UK18/presentation_6165_1539151116.pdf

CAPSman

Wifi Channel Planning / 802.11ac / CapsMAN

band steering

vlans

maxslug · September 18, 2020, 4:08am

Software

This install required 3 versions of RouterOS:

arm64 - RB5009 Router. No wireless driver
arm - capAC WAP. wifi-qcom-ac driver
mips - crs109. wireless driver

winbox on MacOS
I like this way of using / installing it better than the suggested:

brew tap homebrew/cask-versions                                                                                                                                
brew install --cask --no-quarantine wine-devel
killall wineserver                                                                                                                                                       
wine64 winbox64.exe

maxslug · September 18, 2020, 4:26am

Remaining mysteries and TODO

wireless-qcom-ac on a cAPac, new CapsMAN, 3 SSIDs on 3 VLANs. The 2 VLAN solution is already a kludge.

maxslug · September 18, 2020, 4:05pm

and already one pull request, thanks Alibloke!

anav · September 19, 2020, 2:49pm

I know that on the bridge vlan config, the untagged are automatically added but I like to see them in the config so that I can track them visually. To me they appear to be missing and its mildly confusing LOL. OP preference I suppose, no argument.

On the input chain I do have some points.
add chain=forward? action=fasttrack-connection connection-state=established,related
add chain=input action=accept connection-state=established,related,untracked
add chain=input action=accept in-interface-list=VLAN comment=“Allow VLANs”
add chain=input action=accept in-interface-list=BASE comment=“Allow Base_Vlan Full Access”
add chain=input action=accept protocol=icmp
add chain=input action=drop in-interface-list=!VLAN
add chain=input action=drop connection-state=invalid
add chain=input action=drop

I would change to the following order and content (not sure why your forward chain fastrack rule is in this set but so be it)
TO
add chain=input action=accept connection-state=established,related,untracked
add chain=input action=drop connection-state=invalid
add chain=input action=accept protocol=icmp
add chain=input action=accept in-interface-list=LAN source-address-list=adminaccess ***
add chain=input action=accept any services that users need such as DNS services. @@@
add chain=input action=drop

Discussion: There is absolutely no need on security principles to let any user on any vlan full access to the router. Limit it only to the admin.
Any services that users may need, provide access to the router for those services.
As for the last rule, thats what I use, so your third last rule - drop anything not coming from the LAN, is useless as the last rule already covers that and more.

/ip firewall address list
add ipaddress of admin desktop list=adminaccess
add ipaddress of admin laptop list=adminaccess
etc…

@@@ example…
add action=accept chain=input comment=“Allow LAN DNS queries-UDP”
connection-state=“” dst-port=53 in-interface-list=LAN protocol=udp
add action=accept chain=input comment=“Allow LAN DNS queries - TCP” dst-port=
53 in-interface-list=LAN protocol=tcp

Okay I do note that you dont have a LAN list, needs to be added.
options are to identify the bridge as being on the LAN or
each VLAN on the LAN (what I would do).

On the forward chain…

FROM
add chain=forward action=accept connection-state=established,related,untracked
add chain=forward action=accept ipsec-policy=in,ipsec
add chain=forward action=accept ipsec-policy=out,ipsec
add chain=forward action=accept connection-state=new in-interface-list=VLAN out-interface-list=WAN
add chain=forward action=accept connection-state=new in-interface-list=BASE out-interface-list=WAN
add chain=forward action=accept connection-nat-state=dstnat comment=“For port forwarding to VLANs”
add chain=forward action=drop connection-state=invalid
add chain=forward action=drop connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add chain=forward action=drop in-interface=ether1 log=yes log-prefix=!public src-address-list=not_in_internet comment=“Drop incoming from internet which is not public IP”
#add chain=forward action=drop in-interface=ether2 log=yes log-prefix=!public src-address-list=not_in_internet comment=“Drop incoming from internet which is not public IP”
add chain=forward action=drop

TO
add chain=forward action=accept ipsec-policy=in,ipsec
add chain=forward action=accept ipsec-policy=out,ipsec
add chain=forward action=fasttrack-connection connection-state=established,related (for accuracy place it here)
add chain=forward action=accept connection-state=established,related,untracked
add chain=forward action=drop connection-state=invalid
add chain=forward action=accept connection-state=new in-interface-list=VLAN out-interface-list=WAN
add chain=forward action=accept connection-state=new in-interface-list=BASE out-interface-list=WAN
add chain=forward action=accept connection-nat-state=dstnat connection-state=new in-interface-list=WAN
add chain=forward action=drop

Discussion, ipsec rules should go before fastrack. The port forwarding rules and extra bloat blocking were confusing at best and way to complex. Simplify to a clear allow port forwarding rule.
The drop all rule covers everything else not explicitly permitted so you are done!!

maxslug · September 20, 2020, 5:55am

Thanks anav, I really appreciate the review and feedback!

I went back and forth on this one. Future proof the config files but allowing for new default RouterOS behavior, or future proof the config files by being explicit against future RouterOS default behavior changes? In the end I went with “fewest lines possible”.

I understand your advise, and appreciate it, but chose not to take it. I have the management VLAN (100) that provides the access control. When I attach to it with a laptop or my desktop (through a second NIC) I’m not going to have static IPs. So it’s easier to not have to deal with an IP list. Instead I have an SSID (hidden) for the management, or switch ports with elevated priviledge that are not normally used. I’m not worried about physical security. I also am aware hidden SSID is obfuscation at best, I’ll be relying on strong passwords.

Done, thanks! I wasn’t sure at all where to put the fastrack and IP rules, so I stuck the fast track at the beginning. I didn’t add a “LAN” interface name group, but stuck with the example “VLAN” and “BASE” groups, where “VLAN” really is the LAN group and “BASE” is the Adminstrative VLAN.

Here’s where I ended up, hopefully I covered everything.

ip firewall filter
# input services the local router only at all the .1 addresses
add chain=input   action=accept               connection-state=established,related,untracked
add chain=input   action=drop                 connection-state=invalid
add chain=input   action=accept               protocol=icmp
add chain=input   action=accept               dst-port=53 in-interface-list=VLAN protocol=udp
add chain=input   action=accept               dst-port=53 in-interface-list=VLAN protocol=tcp
add chain=input   action=accept               in-interface-list=BASE comment="Allow Management Vlan Full Access"
add chain=input   action=drop

# forward services the NAT / Routing
add chain=forward action=accept               ipsec-policy=in,ipsec
add chain=forward action=accept               ipsec-policy=out,ipsec
add chain=forward action=fasttrack-connection connection-state=established,related
add chain=forward action=accept               connection-state=established,related,untracked
add chain=forward action=drop                 connection-state=invalid
add chain=forward action=accept               connection-state=new in-interface-list=VLAN out-interface-list=WAN
add chain=forward action=accept               connection-state=new in-interface-list=BASE out-interface-list=WAN
add chain=forward action=accept               connection-nat-state=dstnat comment="For port forwarding to VLANs"
add chain=forward action=drop

Thanks!
-m

anav · September 20, 2020, 11:28am

Looks great!

maxslug · September 22, 2020, 4:11pm

added routing / firewall handling of connections to the plex server from the guest and neighbor vlans.

maxslug · September 22, 2020, 9:39pm

added printing between vlans. Looks like bonjour / zeroconf / airprint will not forward across subnets / vlans, so only one vlan will get auto-discovery of the printer.

anav · September 23, 2020, 2:12am

Dont know about proprietary crap but I use firewall rules to allow users on vlans to access a printer on another vlan.

xvo · September 23, 2020, 6:52am

Firewall won’t help you in case of discovery protocols that rely on broadcasts (and are supposed to work inside one broadcast domain), no matter if they are open or proprietary.
Only some kind of “proxy” or “reflector”.

pcunite · September 23, 2020, 1:46pm

Appreciate the great effort here. You have put the work into this. The information about how to configure these devices needs to be more open, more clear, and easily digestible. This will help to move that forward. An entire topic should be spent on Service Discovery between VLANs, I should think.

maxslug · October 6, 2020, 4:08am

Thanks pcunite!
I figured out the Hairpin NAT settings and added it here : http://forum.mikrotik.com/t/hairpin-nat-is-there-a-simple-solution/140385/25

maxslug · October 31, 2020, 11:07pm

pushed the hairpin NAT stuff to the git repo

maxslug · November 15, 2020, 9:35pm

Switched to Google DOH, Cloudflare was buggin’ out on me. Also added auto-update of cert and dns cache flush scripts from IYARINDRA.

maxslug · January 17, 2021, 12:09am

Updates:

DOH is a lot more stable w/ 6.48
Moved switch port uplink to ethernet0 since that’s the POE in port. Recently bought a big UPS, so moved everything over to POE to centralize the power point.

maxslug · March 19, 2024, 7:39pm

Welp… my RB4011 bit the dust

I got myself a RB5009 and will be attempting to reproduce this setup. I imagine some things will change as I migrate to RouterOS 7 and update the switches and APs too.

Stay tuned…