I have a network consisting of a few ipv4 networks separated into their own vlans. Only 1 network uses ipv6. I dont need routing between ipv4 and ipv6.
Theres both a router, L3 switch and im planning on adding another router as a switch. All routers/switches are mikrotik. What i’d like to do is figure out how to set up routing efficiently. For example if 2 devices between different networks communicate, they should be able to between either the same router/switch or through different ones. How would i go about doing this to avoid saturating the link between the router and switch? The routers act as switches too with several devices. Should each router/switch be giving out their own DHCP entries? Is there a way to use a single DHCP server with relays and different entries?
Each router/switch has the same networks connected to it, both multiple vlans and multiple ip networks.
Lets go step by step, from the obvious to the actual subject of your question.
By design, the traffic takes the shortest path within the domain, unless you forcifully change that. So two devices in the same subnet and VLAN connected to the same switch tak to each other via the switch alone; if they are connected to different switches in an L2 network, the traffic between them takes the shortest path through the tree organized by the spanning tree protocol, which may not be the shortest physical path if some links have a better bandwidth than others and you set the preferences accordingly in the spanning tree configuration.
For L3, the actual path between two endpoints may not be even symmetric - let’s say there are two routers, each connected to the same pair of subnets, and two endpoints, each in another one of those two subnets. As a gateway to the other subnet than its own, each endpoint may have another router configured, so the request from endpoint A to endpoint B may be forwarded from subnet A to subnet B by one router, whereas the response may be forwarded from subnet B to subnet A by the other one - it depends on how you set the routes on the endpoints.
How to do it “best” depends on your overall network topology, which you haven’t described enough. From the context, it seems you are interested in the case where there are two routers, each with their own internet uplink, on two geographic locations interconnected using some private transport channel, and you want the clients on each geographical location to use the local router to access internet but still be able to talk from one location to the other via that private transport. Is this a correct understanding?
no not with their own uplink. Theres only 1 internet uplink. Basically if the router is the route to all, then a device wanting to communicate with another on another subnet has to cross the router due to the default route.
IF each router/switch had its own dhcp then traffic wouldnt cross the router to cross subnets but then the routes would have to be manually added.
I've not done real tests, but something like this should be possible, assuming all your switching devices are running RouterOS and are either real routers or CRS with L3HW (because otherwise it's better to let the main router to do all the routing):
You need to be able to "poison" the ARP tables of the client devices, so that the ARP entries for all the gateways of each subnet (normally those . 1 addresses) have the MAC address of the nearest CRS. One way is to have those CRS sending out Gratuitous ARP on all of their non-uplink ports:
Of course, for that you have to use TrafficGen to build the packets to be sent on the individual ethernet ports (do not send on the VLAN or bridge interfaces), which means you have to build the individual VLAN header for each VLAN in the template too.
If that works, clients still use the normal gateway IP addresses announced by DHCP, but when they send packets to the gateways to be forwarded, the destination MAC addresses are the one of the nearest switches, and those will do the routing duty.
I’d still like to see a concise problem description. Is the task indeed to make sure that if two devices in different subnets/VLANs are connected to the same router&switch combo, that combo device will route the traffic between them, as @CGGXANNX solution proposal suggests?
I am not able to see what problem is solved by adding an additional device from the OP. I only see the way the OP will deal with a situation that is not described, hence the request.
Is it port count? Is it different location/room? Is it for high availability?
Knowing this information might result in different answer and approach.
no, the problem is different. The first is routing, i run many VMs and containers and some subnets span across routers. Basically think of each router/switch having the same vlans and subnets across. With the way that routing works, by default DHCP gives a default gateway. So even to communicate in LAN but to a different subnet, traffic must cross that gateway.
This indeed becomes a problem when devices need to communicate. Lets say a device from subnet 1 wants to communicate with a device on subnet 3. They are both connected to the same switch but the router itself is a different device as the gateway. This means that the traffic has to cross the link to the router and come back. This saturates the same link used for the internet.
The number of ports dont matter, as again theres a lot of virtualisation that happens. Some containers also run on mikrotik too, some on other small arm SBCs. There are a lot of VMs as well. Different subnets are meant for different reasons because of the firewall rules as well i.e. hosting or managing and separating networks i.e. home from management or development from everything else. High availability isn’t a concern here as most devices dont support connecting to 2 different switches and 2 different switches interconnected into high availability.
This is just a home lab network with multiple vlans and subnets for different purposes. I’ve researched and found one way around this is to use a special DHCP entry on adding static routes, which would designate the switch as a gateway to a different subnet other than 0.0.0.0/0 as the destination however im not sure how to do this and having multiple DHCP servers can work, however i would also need to figure out how do i deal with having multiple DHCP servers/relays on the same network to make sure they give DHCP only to the attached device rather than to one across another switch/router.
My setup is simple. i have vlans and subnets on a single physical network of switches and routers that contain virtual devices. The number of ports, routers and switches or devices isnt the problem here and i cannot just connect up every cable and use up every port either between every switch or router. When 2 devices are on the same subnet thats not a problem when the gateway is a different device to travel to but it is when across different subnets on the same switch.
Switch routing does work, but it creates issues with internet pathing for some subnets. Having a good DHCP might be the solution but i cant figure out how to implement it in mikrotik. Adding static routes and making sure each device gets a DHCP entry with static routes from the closest physical device is one solution but i would need to figure out how to implement it. On the router however, you cannot selectively not deploy a DHCP server on a slave port as routerOS DHCP is based on CPU connectivity and how the ports are set up as you still need to bridge L2. On routerOS routers and switches have slightly different configurations. For routers L3 switching happens on the CPU and is hardware offloaded onto the switch chip when possible. For switches you configure the switch chip for the routes instead. However configuring the layout of the ports isnt an issue, only that i want local traffic not to have to unnecessarily cross links between devices.
I understood your requirements like you explain in this post when I wrote my answer above.
What you want to avoid is for traffic crossing VLANs having to go way up to the central gateway and back down the uplink, isn't it? You want traffic between two devices located in two different VLANs but physically connected to ports of the same switch to be routed by that switch without being sent up to the main gateway router.
My previous post had the (untested) workaround that doesn't require multiple DHCP servers or DHCP relays. Think about it like the gateway address of each subnet acting like anycast, because it's mapped to the MAC address of the nearest L3 capable switch.
Each L3 switch should of course have the routing table with all subnets it has L3 access to (default route can still point to the main uplink router) and repetition of any firewall filter rules between subnets.
There is an alternative solution but with the downside that Fasttrack will become ineffective on your RouterOS devices and should also only be used if your devices are the one using these models/switch chips:
Because those allow DHCP Snooping and Bridge VLAN Filtering while keeping L2 hardware offload.
You can then enable DHCP Snooping as well as the setting Add DHCP Option 82 on the bridges of the devices. Don't forget to also properly configure the trusted flag for the relevant ports.
Once correctly configured, The lease entries in of the DHCP server instances on the main router will have additional information in the Agent Circuit Id and Agent Remote Id fields that can be used to identify the path from the DHCP client to the DHCP server. Please note that in 7.21 the format of these fields will change, the example below are from 7.20.6.
You can then create DHCP Option Matchers, that matches on the Agent Circuit Id or Agent Remote Id string containing certain substring (added by the intermediate switches) to override the value included in Option 3 (gateway address) sent to the client.
Here in the screenshot below clients with eth 0/3 in Agent Circuit Id (which is DHCP Option 82) will get assign a custom Option Set that has the customized Option 3 value, setting the gateway to 192.168.88.2 instead of the default 192.168.88.1.
By creating matchers on the main router's DHCP server, matching the option 82 parts inserted by the downlink switches/routers to different option sets containing different values for Option 3, you can customize the gateway address received by the DHCP client devices based on their nearest switches.
But be aware that turning on DHCP Snooping on the bridges will make Fasttrack becoming ineffective (because it disables the bridge fast path capability).
