Functional partitioning between CR310 and hAP AX3

RouterOS Beginner Basics
1

Greetings. I’m experienced with Cisco routers and switches from corporate life but am new to RoS. I’ve purchased three identical CR310-8G-2S-IN switches and one hAP AX3. All of these have RouterOS 7.18, and I do not plan to convert the switches to SwitchOS.

I’ve read the RoS and hardware docs (a couple of times, actually) and done some experimentation on an air-gapped network, but I have some general “best practices” questions as I begin my SOHO network design in earnest. Most of my questions aren’t how to design the network, but rather, how to design the network in a way that is sensible for Mikrotik devices/software.

Environment

There are two ISP premises devices: “isp1” is a wired DSL gateway (hopefully soon replaced by fiber-to-premises), and “isp2” is an LTE gateway.

Physically, the equipment will be located as follows:

  • isp1 (DSL/FTTP): Telecom closet in basement
  • switch1 (CR310): Telecom closet in the basement
  • switch2 (CR310): My office, main work area
  • switch3 (CR310): My office, lab bench
  • isp2 (LTE): My office, lab bench (required due to LTE coverage)
  • router1 (hAP AX3): Location TBD (prefer basement but not “in” telecom closet)

I currently have multiple Cat5e (but not Cat6, as it didn’t exist yet) runs through flex conduit from the basement to my office, and in the semi-near term I plan to pull some LC-terminated fiber for the inter-floor runs.

The network topology is flat right now except for an isolated segment between my current Linux firewall/router and the isp1 premises device. That Linux box is also an OpenVPN endpoint to my company. My LAN is 192.168.X.0/24 (“X” redacted for security), and the isolated ISP-to-Linux is 192.168.0.0/24. The isp2 premises device isn’t installed at the moment.

Design Goals

  • Five data VLANs (excluding VLAN 1, reserved for device internals). The VLANs will be for private LAN (has VPN routing), guest LAN (no access to VPN), isp1 DMZ, isp2 DMZ, and devlab. For private and guest LANs the same VLAN covers wired and wireless.
  • Separate IPv4 subnets and IPv6 prefixes for each VLAN.
  • Connect switch1 and switch2, switch1 and switch3, with LACP over Cat5e (existing wiring), two physical ports per connection. Each of these LACP groups is to be a VLAN trunk passing all VLANs. The intention is that if I need to troubleshoot, I can put a laptop directly on any VLAN from my desk. This will also make it easier for me to test security.
  • When I get the fiber run between floors, single LC runs at 10 Gb/sec will connect switch1 and switch2, switch1 and switch3. I’ll pull redundant fiber but leave the spares dark, since I don’t have enough SFP+ ports on switch1 to do otherwise.
  • router1 (the hAP AX3) needs to have different SSIDs and secrets for private and guest wifi, and those should be on different VLANs and IP subnets, as noted. I want to connect this device’s single 2 Gb/sec port (ether1) to switch1, which means using one of the other ports for emergency management access.

I’d post a diagram, but that’s what I’m making now and ran into the “best practice” questions. Finally, here are those questions:


  • Since the hAP AX3 doesn’t have hardware offload for switching, am I hurting performance by having multiple VLANs on that device? What I’ve tried to do is keep it on the edge rather than at the core, to let it focus on the wireless.
  • Is there anything “wrong” with letting each of the CR310 devices also be an OSPF router that advertises the segments to which it directly connects? In Cisco-land you generally don’t want to burden a switch with Layer 3 functions, but with the Mikrotik devices it seems the situation is reversed – don’t burden the hAP AX3 with Layer 2 functions.
  • The hAP AX3 may not have a UPS, depending on where I have to install it for coverage. For this reason, I’m also inclined to put my OpenVPN endpoint, local DNS, and local DHCP on switch1, which is in the telecom closet and has the best UPS protection. Again, is that a terrible idea? My thinking is that making one of the switches my core device also means the one on my lab bench could be an emergency spare for the core.
  • In a general sense, I’m still a little confused about the relationship between VLANs on a bridge, VLANs on a port, and ports on a bridge. I’ve read a lot of forum posts about this, and I think what I need is a single bridge in each device, VLANs on the bridge, ports on the bridge, and not VLANs on the ports. Then I do my routing and firewall config with respect to the bridge VLANs, mostly without reference to the ports. Is that a correct understanding?

I’m sorry this is so long. I really did do my homework first, quite a lot of reading, so maybe I’m just overthinking this by putting it into my corporate frame of reference. I appreciate any comments and suggestions the wizards of RouterOS care to offer.

2

Yes, the modern way of configuring MikroTik routers is with single bridge where you add VLAN interfaces on top of it (optionally) and set up VLAN numbers in Bridge Ports (PVID) and Bridge VLANs menus.

  • Bridge VLANs
    Here you define what VLANs the bridge will handle and set tagged ports. No need to set untagged ports.
  • Bridge Ports
    Allows you to define an access port for a specific VLAN by setting PVID (Port VLAN ID). This will automatically add untagged port to a VLAN in Bridge VLANs.
  • VLAN interface on the bridge (optional)
    You will need it for inter-VLAN routing, firewalling, etc. If it is a switch and you just want to pass VLAN traffic from one port to another, you do not need to create a VLAN interface.

Once you add the interface as a bridge port, the interface becomes slave and IP configuration must be done on the master interface (bridge). And in case of VLANs you add VLAN interface to the bridge and not an individual port. Adding VLAN interfaces to individual ports makes sense for off-bridge ports, like ether1 - WAN port in the default config.

Consider using Interface Lists instead of Interfaces in the Firewall rules. The default config uses WAN and LAN lists and you may define your own lists for VLANs. Later on if you decide for example to provide VPN service for your guests you’ll just add guest VPN interface into the GUEST list and all firewall rules will apply to VPN connections. Maybe not the best example, but you get the point.

3

Besides the ( BTW surely interesting and useful) discussione on the details of a VLAN and OSPF (complex) configuration It seems to me that you are (seriously) over-estimating the capabilities of a single access point for the wi-fi part.
There Is no way an Ax3 in the basement will cover anything besides - maybe - the basement itself.
With 2.4GHz a very good single AP on the first floor can - in some cases, i.e. very small houses with not very thick walls and decks - cover also the ground floor and the second floor.
With 5GHz even one or two thin walls will be enough to reduce the signal dramatically.
And the Ax3 , while a very good router, Is not among the best/more powerful AP’s around.
Of course It has to be seen in your specific case, but in a very first approximation an AP for each floor of a house Is needed and very often a single one Is not enough to give good coverage, especially in 5 GHz.

4

I’m responding to these two excellent replies out of order due to anticipated complexity of my responses.


I agree that VLAN and OSPF are, shall we say, nontrivial topics, I’m fortunate to have done both in a corporate setting. OSPF isn’t as bad as eBGP, in my opinion. :slight_smile:


… It seems to me that you are (seriously) over-estimating the capabilities of a single access point for the wi-fi part. There Is no way an Ax3 in the basement will cover anything besides - maybe - the basement itself.

You’re probably right, but it costs me very little to try it. Our current 5 GHz WAP is in the half-basement under the family room, which is the room from which it’s used 95% of the time. My office is up one floor and slightly offset from that, and I get a fairly good signal from the office.

I’ve considered the possibility that you’re right and it won’t cover well (thanks, BTW, for letting me know not to expect it to be among the more powerful radios). I’m going to first try it in the telecom closet, just for the heck of it, but I expect that will be a failure. Second choice will be to put it where the current WAP is, under the family room. If it’s at least as good as what we have, it’s a keeper. Third choice is to put it in the family room, replacing one of two cheap unmanaged switches I have there now. (*) And if all else fails, I’ll get another one to cover the areas this one can’t.

(*) In humor mode, two points:

  • If you are thinking, “This person has a lot of stuff on that LAN!”, you are absolutely correct.
  • Among Mikrotik customers, I am willing to bet there are plenty of others who have more nodes than I do. :laughing:
5

Thanks – that helps a great deal. Let me see if I understand correctly, assuming I have just one bridge called “bridge”.

I add VLANs to the bridge, then since I want to do routing and firewalls between VLANs, I will need a VLAN interface for each of the bridge VLANs. All of my routing and firewall configs will refer to those bridge VLAN interfaces, not to physical ports, with the exception of an out-of-band management port if I choose to IP address it.

To set up trunk ports, I add the trunk interfaces (physical ports or LACP groups, as the case may be) as tagged ports on the bridge VLANs. I assume – and please correct me if I’m wrong – that this implies any given interface can be a tagged port on an arbitrary number of VLANs.

To set up access ports, I assign a pvid to the interface, which in this case will be a physical port since I don’t plan to have any non-trunk LACP groups.

Is that correct, or did I misunderstand something?


Consider using Interface Lists instead of Interfaces in the Firewall rules. The default config uses WAN and LAN lists and you may define your own lists for VLANs. Later on if you decide for example to provide VPN service for your guests you’ll just add guest VPN interface into the GUEST list and all firewall rules will apply to VPN connections. Maybe not the best example, but you get the point.

You don’t even have to persuade me on that point! When I set up the company’s new Cisco routers, I was building a new config from scratch to replace an extremely old and crufty one. I used Cisco’s “object groups” to symbolically represent address lists, interface lists, and TCP and UDP port numbers. As you can imagine, our corporate firewall is a lot more complex than anything I’ll need for the SOHO network, and the object groups make it much more readable. This has repaid my labor many, many times over, because it’s so much easier to maintain the configuration. It’s also allowed me to delegate routine changes like moving an access port to a different VLAN, instead of having to do every change personally.

I’ll RTFM to see if I can use Interface Lists for other things in RoS than just firewall rules.

Again, thanks to both of you for a pair of extremely helpful replies. I’ll make a preliminary design and do some testing. I may be back with further questions, but hopefully at that point will have configs to post.

6

The WAPs (It may depend on the exact model) are rather different from “generic” AP’s, they tend to be rather directional with signal projected mainly in a sector 60 to 90 degrees as opposed to the omni-directional (360 degrees) emissions of the internals antennas of most AP’s and of the standard antennas the Ax3 comes with.
This would explain the good penetration you report, so it Is possible that you can replicate it by using different antennas on the Ax3, but there is no real way to known if not with field testing.
It seems to me that you have however the right attitude and flexibility :slight_smile: to experiment and adapt the layout as needed.

7

Hmmmm… I hadn’t considered the possibility of swapping the antennas. Depending on testing, I may look at what Mikrotik offers.

8

This is slightly off-topic for the thread, but it might benefit others designing networks for small businesses. The IETF (in RFC 1918) designated three IPv4 address blocks that are reserved for private networks and not routed on the public Internet: 192.168.0.0/24, 172.16.0.0/12, and 10.0.0.0/8. When our company was young, I was in charge of the IT Department and decided to define the last one, 10.0.0.0/8, as “out of bounds” for the company intranet except on isolated segments. That guarantees employees (most of whom are developers) can freely use those addresses on their home LAN without worrying about routing precedence making part of the company network inaccessible. The same logic goes for developer teams within the company who need isolated segments for testing or inside hypervisors. Of course, none of this is a problem after migration from IPv4 to IPv6.

The moral of the story is that sometimes instead of specifying how things will be used, it’s worthwhile to mark a subset as never being used.

I mention this in the context of this thread because that’s how I plan to handle the aforementioned guest LAN and the DMZ segments between me and the two ISPs.

9

Correct!

In the default configuration management is allowed only from LAN interface list. If you want to have dedicated management interface, that is not a bridge member port, make sure it is added to the LAN list.
If you don’t set any IP address for the management interface, you’ll still be able to connect to the router by MAC using WinBox, as such connections are allowed for LAN list interfaces (see Tools / MAC Server / MAC WinBox Server).


To set up trunk ports, I add the trunk interfaces (physical ports or LACP groups, as the case may be) as tagged ports on the bridge VLANs. I assume – and please correct me if I’m wrong – that this implies any given interface can be a tagged port on an arbitrary number of VLANs.

I am not sure about “any”, but I would say yes.
Please note that the bridge itself must be a tagged interface of VLANs you created interfaces for. It won’t work otherwise. Don’t ask me why :slight_smile:
For example, in router-on-a-stick scenario where ether5 is a trunk port, the config may look like:
Bridge VLANs

vlan-ids=10,20,30 tagged=bridge,ether5

You can also split it by VLAN:

vlan-ids=10 tagged=bridge,ether5
vlan-ids=20 tagged=bridge,ether5
vlan-ids=30 tagged=bridge,ether5

If you configure a switch, no need to include bridge as tagged interface, unless you created VLAN interface for management purposes and set an IP address on it. If your mgmt VLAN is 10, the config could be like:

vlan-ids=10 tagged=bridge,ether5
vlan-ids=20,30 tagged=ether5



To set up access ports, I assign a pvid to the interface, which in this case will be a physical port since I don’t plan to have any non-trunk LACP groups.

Not necessarily physical port, it can also be wifi, vxlan, etc.

Give it a try!

10

Another easy option be MVRP to “share” the VLANs, as that keep them in “sync”. Now still need to add /interface/vlan and perhaps DHCP client/server for any MVRP VLANs on any of the routers (which add the VLAN subnet as connected routes thus routable). But at least MVRP save the VLAN config on the switches.

11

Somehow I just lost a lot of what I had typed. So summary. CR310 is primarily a switch (with some limited L3HW and slow L3 routing, similar to a hEX RB750Gr3) based on Marvel 98DX226S see https://mikrotik.com/product/crs310_8g_2s_in#fndtn-specifications. hAP ax3 is primarily a router with some limited switching based on the IPQ-6010 see https://mikrotik.com/product/hap_ax3#fndtn-specifications,

If routing is being done by hAP ax3, including inter-vlan routing, then whether vlans are involved or not isn’t going to make much difference, because the traffic will already be going to the CPU anyway, and once it is there the incremental amount of extra processing for vlan operations is trivial. But you probably won’t want to use any of the extra ports on the hAP ax3 for high volume vlan access ports (i.e. don’t put a file server on it that is serving devices in the same vlan).

In the case where the switches are just doing L2 (other than their management), then I can’t see any advantage of running ospf on the switches, because ospf is for dynamic routing updates.

Disclaimer: I don’t have any MikroTik CRS switches; only the CSS106-5G-1S and the switches built into RB760iGS and RB5009. So this info is not hands on, just spec sheet reading.

12

Design.
Both ISP connections must reach router ( can be through an existing switch )
Router connected to each switch
Switch configured as required to each device.
Done.

What more do you need!

State Requirements ( identify all users/devices, include external,internal, admin / then identify all the traffic the need )
Draw network diagram to reflect above and all equipment proposed and connections and vlans
Then a coherent config can be formulated.

13

The main reason for wanting OSPF on the switches is due to proximity. The primary ISP (isp1) has its premises gateway (from the ISP) in my basement telecom closet. The secondary ISP (isp2) is LTE and I have to locate that premises gateway in my upper-floor office in order have a decent LTE signal. If I let the nearest switch also handle the L3 I can assure that each border router is on the same switch and UPS as the telco path it supports. OSPF dynamic routing should let me have decent failover between the ISPs, and later I might even do some preferential adjustment of traffic by protocol, port, QoS, etc., to load balance when both ISPs are available but fail over to the remaining ISP if either is down.

The other alternative would be to make sure the hAP ax3 is on a reliable UPS, let it be the One True Router (no edge routers, just a collapsed core/edge) and pass all the VLANs to that device using its 2.5 Gbit port as the trunk. OTOH, if I do let the switches be edge routers, then once I have that SFP+ fiber installed it would mean the entire network infrastructure except for the wifi is interconnected on 10 Gbit backbone.

That’s not such an issue now, but I just yesterday learned that a company is finally bringing FTTP (fiber to the premises) to our neighborhood in the next few months. When that happens, I will be hurling my wallet in their direction. I telecommute full time, and much of my job involves bidirectional data transfers to the company. The company data center has symmetric Gbit plus to redundant ISPs, but the best I’m able to get at the home office right now is 54 Mbit/sec download and 18 Mbit/sec upload. That asymmetry bites hard when pushing new project builds to the company’s repository.

14

Which speed do you expect from the LTE connection?
And which kind of reliability does the primary connection you have?
As always I may be wrong, but It seems to me like you are a bit overthinking it.
I mean, it is not like your two connections are equivalent, if we (for the sake of the example) say that the primary connection works just fine 360 days out of 365 or 98.6% of the time with speed 500 and the LTE Is used for the remaining 1.4% and tops off at speed 100, which amount of optimization do you think you can do on the latter and what impact can this optimization have on your activities?
If you had to guarantee a given bandwidth 24/7 for (say) the command center of a cluster of nuclear plants :open_mouth: it would be another thing, and you would have also a different, much faster second connection, probably topped by a third and a fourth one, each connected to different cables.

15

I think you’re right about overthinking the QoS weighting. Most likely I’ll just assign a higher administrative distance (cost) to the secondary ISP so it’s ignored as long as the primary is active, and call it a day. The redundant connectivity matters, but the small benefit from offloading selected traffic is probably not worth it.

16

I’ve got a partly-working config but ran into a problem. I moved the devices from one room to another to have a more convenient workspace. I did an orderly shutdown before unplugging. After powering back up, at least one of the switches shows up in WinBox as connectable, but even using the MAC address to connect, it gives SynTimeout errors. I tried cycling power, tried multiple different ports, and isolated it from other devices in case it was some sort of RSTP or other low-level loop problem. The last time I had this error I “resolved” it by netinstalling RouterOS from scratch. Hopefully that’s not something that’s required often. Is there something I could be doing wrong that causes this?

A couple of days ago I also thought I had posted my configs to this thread, but today the post is gone, so I’m not sure what happened. Did I break a forum rule that caused moderators to take it down? I had redacted the serial numbers out of the files and replaced them with a string of “ZZZZZZ” and replaced a couple of other fields with dummy values.

17

Winbox3 or Winbox4?
(the latter has been reported as having sometimes issues with MAC connections)
IP access ( if available) should be more reliable.

Time to get familiar with the rules of the Mikrotik Club?
http://forum.mikrotik.com/t/the-twelve-rules-of-mikrotik-club/182164/1
Namely #7.

18

Argh! I had a port like that initially, and in the process of working through the bridge part I accidentally added it to the bridge. That’s likely my problem.

19

Yep, being locked out Is one of the common mistakes when starting with Mikrotik devices, it is part of the learning experience.
But there should not be the need of doing a netinstall, a simpler reset should be enough.

20

I got connected to switch03, which was still unconfigured, and the first thing I did was to change the password. The second thing was to turn ether1 into a dedicated management port with a hardwired IP address.

Now I’ll have to reset the other two devices and see what went sideways.