Need some routing experts to help me figure out my setup

RouterOS General
1

Hi All,

I normally try very hard not to needlessly ask questions without trying all my ideas first, but this one has me a bit lost.

Let me explain my setup:

I have 2 business-class ISP WAN connections, one through Comcast cable broadband, and one through a local fiber carrier (Fidium). (I could explain why I want two ISPs but in a nutshell there’s certain traffic from certain devices I want going through one, and the remainder through the other.)

I will refer to these as the “cable” and “fiber” connections.

So we have two dynamic IP gateways, one is the fiber ONT, the other is the DOCSIS 3.1 modem. (Yes, there is a path to obtaining static IPs but there’s cost-tiering that makes it preferable to avoid, as this would result in a leap in monthly costs.)

These are interfaced with a CCR2004-1G-12S+2XS router.

What I need is to allow only certain physical interfaces on the CCR2004 to get routed to the cable service, and all other traffic to route to the fiber service.

Default routes for each gateway are handled by a DHCP client script, since ROS can’t handle two dynamic (DHCP) default routes at once automatically (as far as I know).

Anything going to the cable service will be in the “BRIDGE-GUEST” ip/pool. This is a subnet assigned to 10.2.2.x. All other traffic will be in bridge/subnet “BRIDGE-LAN” 10.2.0.x or “BRIDGE-WORK” 10.2.1.x, and that will go to the fiber service.

I had followed a tutorial on the older wiki to enable PCC (per-connection-classifier) (article: https://wiki.mikrotik.com/Manual:PCC), and using VRF as well, but when I attempted to upgrade to a newer ROS this setup broke entirely. (I mentioned it in this reply going from 7.15.2 to 7.16.1)

Despite my best efforts, I could not figure out how to get this setup working under the newer version and had to downgrade to regain functionality. The fact remains that something in this setup is not done correctly and I need to resolve it to keep the router updated. Mikrotik support was unhelpful entirely. I’m turning to the community for a desperate plea for guidance.

If there are any tutorials on how to accomplish this cleanly and in compliance with the newer ROS versions, I’m all ears.

The configuration is very lengthy on this router, please forgive me, I tried to clean it up so I can post it. I’m not entirely sure it’s even worth posting, I suspect I will be starting from scratch to get this correct… but I know someone will ask. If I missed a section please let me know.

Thank you for any help you can offer, I’m grateful for this community as a resource.

/ip dhcp-client
add add-default-route=no comment="XFINITY (CABLE) WAN ISP" interface=\
    sfp11-CBL-WAN script=":global \"xfi-gateway-address\" \$\"gateway-address\
    \"\r\
    \n:global \"xfi-gateway-bound\" \$bound\r\
    \n\r\
    \n/system script run dhcp-client-xfi\r\
    \n" use-peer-dns=no use-peer-ntp=no
add add-default-route=no comment="FIDIUM (FIBER) WAN ISP" interface=\
    sfp12-ONT-WAN script=":global \"ont-gateway-address\" \$\"gateway-address\
    \"\r\
    \n:global \"ont-gateway-bound\" \$bound\r\
    \n\r\
    \n/system script run dhcp-client-ont" use-peer-dns=no use-peer-ntp=no
/ip firewall nat
add action=masquerade chain=srcnat comment="MAIN Internet Out" out-interface=\
    sfp12-ONT-WAN
add action=masquerade chain=srcnat comment="XFINITY Internet Out" \
    out-interface=sfp11-CBL-WAN
/ip firewall mangle
add action=mark-connection chain=prerouting comment="(Policy Routing) FIDIUM I\
    nbound - MARK CONNECTION -> WAN-FID-CONN ### See https://wiki.mikrotik.com\
    /wiki/Manual:PCC For Dual-WAN info ###" connection-mark=no-mark \
    in-interface=sfp12-ONT-WAN new-connection-mark=WAN-FID-CONN passthrough=\
    yes
add action=mark-connection chain=prerouting comment=\
    "(Policy Routing) XFINITY Inbound - MARK CONNECTION -> WAN-XFI-CONN" \
    connection-mark=no-mark in-interface=sfp11-CBL-WAN new-connection-mark=\
    WAN-XFI-CONN passthrough=yes
add action=mark-connection chain=prerouting comment="(Policy Routing) XFINITY \
    Outbound (PCC 2/1) - MARK CONNECTION -> WAN-XFI-CONN" connection-mark=\
    no-mark dst-address-type=!local in-interface=BRIDGE-GUEST \
    new-connection-mark=WAN-XFI-CONN passthrough=yes \
    per-connection-classifier=both-addresses:2/1
add action=mark-connection chain=prerouting comment="(Policy Routing) FIDIUM O\
    utbound from MAIN (PCC 2/0) - MARK CONNECTION -> WAN-FID-CONN" \
    connection-mark=no-mark dst-address-type=!local in-interface=BRIDGE-LAN \
    new-connection-mark=WAN-FID-CONN passthrough=yes \
    per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting comment="(Policy Routing) FIDIUM O\
    utbound from WORK (PCC 2/0) - MARK CONNECTION -> WAN-FID-CONN" \
    connection-mark=no-mark dst-address-type=!local in-interface=BRIDGE-WORK \
    new-connection-mark=WAN-FID-CONN passthrough=yes \
    per-connection-classifier=both-addresses:2/0
add action=mark-routing chain=prerouting comment=\
    "(Policy Routing) XFINITY connections - MARK ROUTING -> xfi" \
    connection-mark=WAN-XFI-CONN in-interface=BRIDGE-GUEST new-routing-mark=\
    xfi passthrough=yes
add action=mark-routing chain=prerouting comment=\
    "(Policy Routing) FIDIUM connections - MARK ROUTING -> main" \
    connection-mark=WAN-FID-CONN in-interface=BRIDGE-WORK new-routing-mark=\
    main passthrough=yes
add action=mark-routing chain=prerouting comment=\
    "(Policy Routing) FIDIUM connections - MARK ROUTING -> main" \
    connection-mark=WAN-FID-CONN in-interface=BRIDGE-LAN new-routing-mark=\
    main passthrough=yes
add action=mark-routing chain=output comment=\
    "(Policy Routing) XFINITY connections - MARK ROUTING -> xfi" \
    connection-mark=WAN-XFI-CONN new-routing-mark=xfi passthrough=no
add action=mark-routing chain=output comment=\
    "(Policy Routing) FIDIUM connections - MARK ROUTING -> main" \
    connection-mark=WAN-FID-CONN new-routing-mark=main passthrough=no
/ip route
add check-gateway=ping comment="Default route to SFP12-ONT-WAN" disabled=no \
    distance=1 dst-address=0.0.0.0/0 gateway=[public IP redacted]@main routing-table=\
    main scope=10 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment="Default route to SFP11-XFI-WAN" disabled=no \
    distance=1 dst-address=0.0.0.0/0 gateway=[public IP redacted]@xfi routing-table=\
    xfi scope=10 suppress-hw-offload=no target-scope=10
/ip vrf
add interfaces=sfp10-STRICT,sfp11-CBL-WAN name=xfi
/routing rip interface-template
add disabled=no instance=rip-instance-main interfaces=BRIDGE-LAN mode=passive
add disabled=no instance=rip-instance-main interfaces=BRIDGE-WORK mode=\
    passive
add disabled=no instance=rip-instance-guest interfaces=BRIDGE-GUEST mode=\
    passive
/routing rule
add action=lookup disabled=no interface=BRIDGE-LAN src-address=10.2.0.0/24 \
    table=main
add action=lookup disabled=no interface=BRIDGE-WORK src-address=10.2.1.0/24 \
    table=main
add action=lookup disabled=no interface=BRIDGE-GUEST src-address=10.2.2.0/24 \
    table=xfi
2

I to have two WANS, one fiber and one cable docsis. I only need a script for the fiber due to the fact when it changes IP address, the new gateway fails to be changed in my manual routes.
For some reason my cable modem and router seem to have no issues dealing with the changing gateway…

As to the approach,
since this is USERS X to one WAN1 and USERS Y to WAN2, you dont need to worry about PCC.
Simply use ECMP approach which treats each WAN equally with recursive routing to ensure connection to the www is good.

Use routing rules to move subnet A out WAN2 special table
Use routing rules to move subnet B out WAN1 special table
Any other subnets not involved in being forced will go out either WAN1 or WAN2 in a roughly equal manner ( regardless of the throughput of each connection, in other words the higher throughput WAN would not be used more than the lower throughput WAN )

The more difficult questions will be.
a.. is there any need for out of subnet traffic ( non-layer2), in other words between subnets on the router or any external access to the subnets

  • admin to all vlans
  • external users to servers
  • admin remote VPN to vlans

b. What happens if one or the other WAN connection becomes unavailable.
Do you want the other WAN to be a failover option or simply no internet?

3

Thanks, I should have said my intentions more simply as “I want it to act as if there’s two physically separated networks with two NAT routers,” but both piped through the same router instead to reduce extra hardware and energy costs. That CCR2004 should handle both with ease.

No need for access from the GUEST subnet (Cable for WAN routes) to any of the other subnets (that would use fiber for WAN routes).

No failover needed.

4

No need for mangles!
/table
add fib name=useWANFiber
add fib name=useWANCable

/ip route
add check-gateway=ping dst-address=0.0.0.0/0 gateway=(current)fiber-gateway-IP routing-table=main comment=Fiber1
add check-gateway=ping dst-address=0.0.0.0/0 gateway=(current)cable-gateway-IP routing-table=main comment=Cable2
add dst-address=0.0.0.0/0 gateway=(current)fiber-gateway-IP routing-table=useWANFiber comment=Fiber1
add dst-address=0.0.0.0/0 gateway=(current)cable-gateway-IP routing-table=useWANCable main comment=Cable2

/ip routing rule
add min-prefix=0 lookup-only-in table table=main
add src-address=10.2.0.0/24 action=lookup-only-in-table table=useWANFiber
add src-address=10.2.1.0/24 action=lookup-only-in-table table=useWANFiber
add src-address=10.2.2.0/24 action=lookup-only-in-table table=useWANCable

DONE!

Since you dont show the ip dhcp client script its hard to say what else to put in.
However, you can do it all from the IP DHCP script and dont need to point to a separate script… more efficient.

Do this for both as applicable:

DHCP
Set default route=yes peer dns=no peer NTP=no default route=yes
Advanced
set distance=255

:if ($bound=1) do={
:local gw $“gateway-address”
/ip route set [ find comment=“Fiber1” gateway!=$gw ] gateway=$gw
}

What it would look like in export config:
/ip dhcp-client
add comment=MyFiber default-route-distance=255 interface=vlanName script=“:if ($bound=1) do={\r
\n:local gw $"gateway-address"\r
\n/ip route set [ find comment="Fiber1" gateway!=$gw ] gateway=$gw\r
\n}\r
\n” use-peer-dns=no use-peer-ntp=no

5

What for using: add min-prefix=0 lookup-only-in table table=main ?

6

It ensures any local traffic that is allowed, by firewall rules, has a path, otherwise all traffic will head out WANS.

7

Some more details on the meaning and some horror stories (about needing to delete the routing rule and recreate it to really emove the min-prefix if needed) are here:
http://forum.mikrotik.com/t/cant-get-wireguard-client-to-work/179987/1

8

Thanks everyone for the feedback, I’ll give these ideas a try!