Combining two routers

k6ccc · September 26, 2021, 7:09pm

I currently have two routers serving various stuff at home. One has a cable based internet and the other has a DSL based internet. I want to combine the two routers into one. I tried this once before and all the LANs and VLANs worked fine, but external routing was all screwed up. I think I know what I messed up and need to do to get the DSL based LANs to route via the DSL and the cable based LANs to route via the cable, but as Anav is fond of saying, tell us what you are trying to accomplish, not configuration (or something along that line). Besides, there are people here that know far more about this than I do. Now the details.

First of all, here is an extract of my LAN drawing of how it is today (and working fine).

In case you can’t see the image, here is a link to it: http://extraphotos.info/mikrotik/2021-09-26a_extract.png

Router #1 gets a DHCP from my cable, and is used by the 100 series of LANs and VLANs (and .209). Router #2 gets a DHCP from my DSL and is used by the 200 series LANs and VLANs. The 211 LAN is used to get between the two routers. For now, these is no intent of automatic failover. There will likely be some of that in the future, but not today. Neither router has a bridge configured and since I am only using the routers as routers and all switch functions are done in the CSS326, I don’t believe there is any advantage to adding a bridge to the router. If I’m wrong about that, please explain why.

For now what I want to do is move ports 1 through 5 of router #2 into ports 6 through 10 of Router #1. Obviously the 211 LAN currently on port 5 of each router will not be needed as an external link. Setting up the interfaces, and firewall rules is easy. I just need to make sure that traffic on the 100 series LANs and VLANs uses the DSL and traffic on the 200 series LANs uses the cable internet. As I said, I think I know what I need to do to make it work right this time, but want to hear from the experts…

sindy · September 26, 2021, 7:31pm

There isn’t. Just loosely related (i.e. not a reason to add a bridge), I don’t know why you use separate cables between the 750 and the CSS rather than moving LANs 201&203 to the trunk, but I suppose it’s for bandwidth reasons.

Since there are no extras like routing based on destination port etc., I’d say use routing rules (and I anticipate @anav will suggest the same):

/ip route add
routing-mark=via-Extreme gateway=ex.tre.me.gw

/ip route rule add
interface=lan200 action=lookup-only-in-table table=via-Extreme
interface=lan201 action=lookup-only-in-table table=via-Extreme
interface=vlan202 action=lookup-only-in-table table=via-Extreme
interface=lan203 action=lookup-only-in-table table=via-Extreme
interface=vlan204 action=lookup-only-in-table table=via-Extreme

If the subnet, and therefore gateway IP, assigned by Extreme changes from time to time, you’ll need to set up a script on the DHCP client to update the gateway in the route.

It would be even easier to use VRF to assign the routing-mark rather than routing rules, but if I remember well, there was some issue with the DHCP client not adding the default route into the proper routing table, so you’d have to try first. And if you think about doing some failover magic later on, there’s no point in trying the VRF path at all.

k6ccc · September 26, 2021, 9:11pm

Thanks Sindy!

Old history mostly. 101 and 103 on the cable and 201 and 203 on the DSL have the bulk of the traffic. so I put them on their own ports. The router has the physical ports, might as well use them… Yes, I could (and may) move more of the LANs onto the trunk. That would free up some ports on the CSS (which is getting quite full).

That is similar to what I had in mind. Slightly different implementation. I had assumed using a routing mark in firewall mangle. Do I need to do similar set of rules for the 100 series LANs to make sure they only use the Spectrum cable internet?

Understand that. Has not changed in years.

I looked at VRF as well.

Now one part I did not mention (because it should not make any difference). Today I make this work with the DSL, but on Thursday my DSL goes away and is replaced with fiber (WooHoo!)

sindy · September 27, 2021, 7:33am

Whatever doesn’t explicitly get a routing-mark will use routing table main. So no, you don’t need a similar set of rules for (V)LANs 1xx.

Assigning the routing-mark using mangle rules cannot coexist with fasttracking whereas assigning them using routing rules can. OK, it’s a 4011 so you shouldn’t need fasttracking.

Congratulations. Yes, no difference from the routing point of view.

k6ccc · September 27, 2021, 1:59pm

Whatever doesn’t explicitly get a routing-mark will use routing table main. So no, you don’t need a similar set of rules for (V)LANs 1xx.

I had figured out the main table part shortly after my previous post.

Assigning the routing-mark using mangle rules cannot coexist with fasttracking whereas assigning them using routing rules can. OK, it’s a 4011 so you shouldn’t need fasttracking.

OK, makes sense (I think).

Congratulations. Yes, no difference from the routing point of view.

Thanks. The fiber to my neighborhood has been a long story (something like 7 years). Delighted to finally have it available.

Thank you for your time. Sindy.

k6ccc · September 30, 2021, 11:28pm

Not working. The original cable internet on this router is working fine, but the new fiber internet is not.. Can’t even ping it via the fiber. Torch shows the ICMP packets arriving on the interface, and an added rule to specifically allow ICMP packets from the fiber is counting the packets. Outbound pings and traceroute from the router when I specify the fiber interface work fine. I have not even tried traffic through the router on the 201 - 204 LANs that will use the fiber. I figure that if I can’t even ping it and get a response, I’m missing something…
Here is a cleaned up export.

# sep/30/2021 15:22:42 by RouterOS 6.48.3
# software id = <redacted>
#
# model = RB4011iGS+
# serial number = <redacted>
/interface ethernet
set [ find default-name=ether1 ] comment="Spectrum cable internet" name=\
    E01-p2_Cable_Internet speed=100Mbps
set [ find default-name=ether2 ] comment="Main cable home LAN" name=\
    E02-p4_101 speed=100Mbps
set [ find default-name=ether3 ] comment="Cable Private WiFi" name=E03-p6_103 \
    speed=100Mbps
set [ find default-name=ether4 ] comment="802.1Q trunk" name=E04-p8_802.1Q \
    speed=100Mbps
set [ find default-name=ether5 ] comment="Router to Router tie" name=\
    "E05 to Router 2" speed=100Mbps
set [ find default-name=ether8 ] disabled=yes
set [ find default-name=ether10 ] name=E10-Fiber poe-out=off
set [ find default-name=ether7 ] disabled=yes
set [ find default-name=ether9 ] disabled=yes
set [ find default-name=sfp-sfpplus1 ] disabled=yes
/interface vlan
add comment="AREDN WAN" interface=E04-p8_802.1Q name=VLAN_001 vlan-id=1
add comment="AREDN LAN" interface=E04-p8_802.1Q name=VLAN_005 vlan-id=5
add comment="AREDN 3GHz at Johnstone to Pleasants Peak LAN interface" \
    interface=E04-p8_802.1Q name=VLAN_011 vlan-id=11
add comment="AREDN 5GHz at Johnstone SW sector LAN interface" interface=\
    E04-p8_802.1Q name=VLAN_012 vlan-id=12
add comment="AREDN 5GHz at Johnstone SE sector LAN interface" interface=\
    E04-p8_802.1Q name=VLAN_013 vlan-id=13
add comment="Cable Public WiFi" interface=E04-p8_802.1Q name=VLAN_102 \
    vlan-id=102
add comment="Cactus" interface=E04-p8_802.1Q name=VLAN_104 vlan-id=104
add comment="VOIP phones" interface=E04-p8_802.1Q name=VLAN_105 vlan-id=105
add comment="Internet of Things" interface=E04-p8_802.1Q name=VLAN_106 \
    vlan-id=106
add comment=".131 E1.31 LAN" interface=E04-p8_802.1Q name=VLAN_131 vlan-id=\
    131
add comment="Main FO home LAN" interface=E04-p8_802.1Q name=VLAN_201 \
    vlan-id=201
add comment=".202 Public WiFi LAN" interface=E04-p8_802.1Q name=VLAN_202 \
    vlan-id=202
add comment=".203 Private WiFi LAN" interface=E04-p8_802.1Q name=VLAN_203 \
    vlan-id=203
add comment=".204 Cactus" interface=E04-p8_802.1Q name=VLAN_204 vlan-id=204
add comment=".209 HARPUSA LAN" interface=E04-p8_802.1Q name=VLAN_209 vlan-id=\
    209
/interface ethernet switch port
set 0 default-vlan-id=0
set 1 default-vlan-id=0
set 2 default-vlan-id=0
set 3 default-vlan-id=0
set 4 default-vlan-id=0
set 5 default-vlan-id=0
set 6 default-vlan-id=0
set 7 default-vlan-id=0
set 8 default-vlan-id=0
set 9 default-vlan-id=0
set 10 default-vlan-id=0
set 11 default-vlan-id=0
/interface list
add exclude=dynamic name=discover
add name=mactel
add name=mac-winbox
add name=LAN
add name=Router-access
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip firewall layer7-protocol
add name=local.mesh regexp=local.mesh
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=aes-128-cbc
/ip pool
add name=".101 DHCP pool" ranges=192.168.101.100-192.168.101.119
add name=".102 DHCP pool" ranges=192.168.102.100-192.168.102.119
add name=".103 DHCP pool" ranges=192.168.103.100-192.168.103.119
add name=".104 DHCP pool" ranges=192.168.104.100-192.168.104.109
add name=".106 DHCP pool" ranges=192.168.106.100-192.168.106.109
add name=".105 DHCP pool" ranges=192.168.105.100-192.168.105.109
add name=".151 DHCP pool" ranges=192.168.151.100-192.168.151.119
add name=".001 DHCP pool" ranges=192.168.1.100-192.168.1.109
add name=".131 DHCP pool" ranges=192.168.131.100-192.168.131.109
add name=".201 DHCP pool" ranges=192.168.201.100-192.168.201.119
add name=".202 DHCP pool" ranges=192.168.202.100-192.168.202.119
add name=".203 DHCP pool" ranges=192.168.203.100-192.168.203.119
add name=".204 DHCP pool" ranges=192.168.204.100-192.168.204.109
add name=".209 DHCP pool" ranges=192.168.209.100-192.168.209.109
add name=".206 DHCP pool" ranges=192.168.206.100-192.168.206.119
/ip dhcp-server
add address-pool=".101 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=E02-p4_101 lease-time=3h name=".101 DHCP server"
add address-pool=".102 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_102 lease-time=3h name=".102 DHCP server"
add address-pool=".103 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=E03-p6_103 lease-time=3h name=".103 DHCP server"
add address-pool=".104 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_104 lease-time=3h name=".104 DHCP server"
add address-pool=".106 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_106 lease-time=3h name=".106 DHCP server"
add address-pool=".105 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_105 lease-time=3h name=".105 DHCP server"
add address-pool=".151 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_151 lease-time=3h name=".151 DHCP server"
add address-pool=".001 DHCP pool" disabled=no interface=VLAN_001 lease-time=\
    3h name=".001 DHCP server"
add address-pool=".131 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_131 lease-time=3h name=".131 DHCP server"
add address-pool=".201 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_201 lease-time=3h name=".201 DHCP server"
add address-pool=".202 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_202 lease-time=3h name=".202 DHCP server"
add address-pool=".204 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_204 lease-time=3h name=".204 DHCP server"
add address-pool=".203 DHCP pool" authoritative=after-2sec-delay disabled=no \
    interface=VLAN_203 lease-time=3h name=".203 DHCP server"
add address-pool=".209 DHCP pool" disabled=no interface=VLAN_209 lease-time=\
    3h name=".209 DHCP server"
add address-pool=".206 DHCP pool" authoritative=after-2sec-delay interface=\
    E08-pXX_206 lease-time=3h name=".206 DHCP server"
/ipv6 dhcp-server
add address-pool=pool1 interface=E02-p4_101 name=server1
/ipv6 pool
add name=pool1 prefix-length=56
/queue simple
add burst-limit=256k/512k burst-time=10s/10s limit-at=128k/256k max-limit=\
    128k/256k name="Test queue" target=192.168.103.182/32
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/system logging action
set 3 remote=192.168.101.11 src-address=192.168.101.251
/user group
set full policy="local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,pas\
    sword,web,sniff,sensitive,api,romon,dude,tikapp"
#error exporting /interface bridge calea
/ip neighbor discovery-settings
set discover-interface-list=all
/ip settings
set allow-fast-path=no
/interface list member
add interface=E02-p4_101 list=mactel
add interface=E02-p4_101 list=mac-winbox
add interface=VLAN_201 list=LAN
add interface=VLAN_202 list=LAN
add interface=VLAN_203 list=LAN
add interface=VLAN_204 list=LAN
add interface=E02-p4_101 list=LAN
add interface=E03-p6_103 list=LAN
add interface=VLAN_102 list=LAN
add interface=VLAN_104 list=LAN
add interface=VLAN_105 list=LAN
add interface=VLAN_106 list=LAN
add interface=VLAN_131 list=LAN
add interface=VLAN_151 list=LAN
add interface=E02-p4_101 list=Router-access
add interface=VLAN_201 list=Router-access
add interface=E01-p2_Cable_Internet list=WAN
add interface=E10-Fiber list=WAN
/ip accounting
set account-local-traffic=yes enabled=yes
/ip accounting web-access
set accessible-via-web=yes address=192.168.101.0/26
/ip address
add address=192.168.101.251/24 interface=E02-p4_101 network=192.168.101.0
add address=192.168.102.251/24 interface=VLAN_102 network=192.168.102.0
add address=192.168.103.251/24 interface=E03-p6_103 network=192.168.103.0
add address=192.168.211.251/24 interface="E05 to Router 2" network=\
    192.168.211.0
add address=192.168.104.251/24 interface=VLAN_104 network=192.168.104.0
add address=192.168.105.251/24 interface=VLAN_105 network=192.168.105.0
add address=192.168.106.251/24 interface=VLAN_106 network=192.168.106.0
add address=192.168.151.251/24 interface=VLAN_151 network=192.168.151.0
add address=192.168.204.251/24 interface=VLAN_204 network=192.168.204.0
add address=192.168.1.251/24 interface=VLAN_001 network=192.168.1.0
add address=192.168.201.251/24 interface=VLAN_201 network=192.168.201.0
add address=192.168.202.251/24 interface=VLAN_202 network=192.168.202.0
add address=192.168.203.251/24 interface=VLAN_203 network=192.168.203.0
add address=192.168.209.251/24 interface=VLAN_209 network=192.168.209.0
add address=192.168.131.251/24 interface=VLAN_131 network=192.168.131.0
/ip cloud
set update-time=no
/ip dhcp-client

< Removed simply because it's a really long list that should serve no
  purpose in this discussion >

/ip dhcp-server network
add address=192.168.1.0/24 comment="AREDN WAN network" dns-server=\
    192.168.1.251 gateway=192.168.1.251 netmask=24
add address=192.168.101.0/24 comment=".101 network" dns-server=\
    192.168.101.11,192.168.101.251 gateway=192.168.101.251 netmask=24
add address=192.168.102.0/24 comment=".102 network" dns-server=\
    192.168.102.251,8.8.8.8,4.2.2.2 gateway=192.168.102.251 netmask=24
add address=192.168.103.0/24 comment=".103 network" dns-server=\
    192.168.103.251,8.8.8.8,4.2.2.3 gateway=192.168.103.251 netmask=24
add address=192.168.104.0/24 comment=".104 network" dns-server=\
    192.168.104.251,8.8.8.8,4.2.2.4 gateway=192.168.104.251 netmask=24
add address=192.168.105.0/24 comment=".105 network" dns-server=\
    192.168.105.251,8.8.8.8,4.2.2.2 gateway=192.168.105.251 netmask=24
add address=192.168.106.0/24 comment=".106 network" dns-server=\
    192.168.106.251,8.8.8.8,4.2.2.4 gateway=192.168.106.251 netmask=24
add address=192.168.131.0/24 comment=".131 network" dns-server=\
    192.168.131.251,8.8.8.8,4.2.2.1 gateway=192.168.131.251 netmask=24
add address=192.168.151.0/24 comment=".151 network" dns-server=\
    192.168.151.251,8.8.8.8,4.2.2.1 gateway=192.168.151.251 netmask=24
add address=192.168.201.0/24 comment=".201 network" dns-server=\
    192.168.201.251,8.8.8.8,4.2.2.1 gateway=192.168.201.251 netmask=24
add address=192.168.202.0/24 comment=".202 network" dns-server=\
    192.168.202.251,8.8.8.8,4.2.2.2 gateway=192.168.202.251 netmask=24
add address=192.168.203.0/24 comment=".203 network" dns-server=\
    192.168.203.251,8.8.8.8,4.2.2.3 gateway=192.168.203.251 netmask=24
add address=192.168.204.0/24 comment=".204 network" dns-server=\
    192.168.204.251,8.8.8.8,4.2.2.4 gateway=192.168.204.251 netmask=24
add address=192.168.209.0/24 comment=".209 network" dns-server=\
    192.168.209.251,8.8.8.8,4.2.2.5 gateway=192.168.209.251 netmask=24
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,4.2.2.1
/ip dns static
add address=192.168.101.251 name=router
add address=192.168.101.11 name=k6ccc.org
/ip firewall address-list
add address=131.107.13.100 list="NTP servers"
add address=64.62.190.177 list="NTP servers"
add address=50.22.155.163 list="NTP servers"
add address=8.8.8.8 comment="Google #2" list="DNS servers"
add address=8.8.4.4 comment="Google #1" list="DNS servers"
add address=107.167.0.0/16 list="Manual Blacklist"
add address=10.9.60.81 comment="Mikrotik hAP-at-Home for AREDN" list=\
    hAP-at-Home
add address=192.73.242.152 list="NTP servers"
add address=132.163.97.4 list="NTP servers"
#error exporting /ip firewall calea

/ip firewall filter
add action=passthrough chain=output out-interface=E01-p2_Cable_Internet
add action=passthrough chain=output out-interface=VLAN_106
add action=passthrough chain=input in-interface=VLAN_106
add action=jump chain=input comment="Jump to Attack chain to prevent Port scan\
    \_and DoS attacks from WAN interfaces" in-interface-list=WAN jump-target=\
    Attack
add action=jump chain=input comment=\
    "Jump to ICMP chain to prevent being ping flooded from WAN interfaces" \
    in-interface-list=WAN jump-target=ICMP protocol=icmp
add action=accept chain=input comment="Allow PING on all LAN interfaces." \
    in-interface-list=LAN protocol=icmp
add action=drop chain=input comment=\
    "Drop TCP DNS packets from WAN interfaces" dst-port=53 in-interface-list=\
    WAN protocol=tcp
add action=drop chain=input comment=\
    "Drop UPD DNS packets from WAN interfaces" dst-port=53 in-interface-list=\
    WAN protocol=udp
add action=drop chain=input comment=\
    "Drop FTP attempts to router from internet via \"normal\" port 21" \
    dst-port=21 in-interface=E01-p2_Cable_Internet log-prefix=\
    "Invalid FTP p21" protocol=tcp
add action=drop chain=input comment=\
    "Drop SSH attempts to router from internet via \"normal\" port 22" \
    dst-port=22 in-interface=E01-p2_Cable_Internet log-prefix=\
    "Invalid SSH p22" protocol=tcp
add action=drop chain=input comment=\
    "Drop Telnet attempts to router from internet via \"normal\" port 23" \
    dst-port=23 in-interface=E01-p2_Cable_Internet log-prefix=\
    "Invalid Telnet p23" protocol=tcp
add action=drop chain=input comment=\
    "Drop WinBox attempts to router from internet via \"normal\" port 8291" \
    dst-port=8291 in-interface=E01-p2_Cable_Internet log-prefix=\
    "Invalid WinBox p8291" protocol=tcp
add action=accept chain=input comment=\
    "Allow established and related connections to router" connection-state=\
    established,related
add action=drop chain=input comment=\
    "Drop any other input packets that get this far" log-prefix=\
    "Dropped connection"

add action=drop chain=forward comment="Drop invalid packets" \
    connection-state=invalid
add action=accept chain=forward comment="Allow AREDN VLAN 11 access to .101" \
    disabled=yes in-interface=VLAN_011 out-interface=E02-p4_101
add action=accept chain=forward comment="Allow AREDN VLAN 12 access to .101" \
    disabled=yes in-interface=VLAN_012 out-interface=E02-p4_101
add action=accept chain=forward comment="Allow AREDN VLAN 13 access to .101" \
    disabled=yes in-interface=VLAN_013 out-interface=E02-p4_101

< Deleted a bunch of passthrough rules used a packet counters - normally disabled
  except when needed >

add action=accept chain=forward comment="Accept all that is DST NATed" \
    connection-nat-state=dstnat connection-state=new
add action=accept chain=forward comment="Accept all that is Source NATed" \
    connection-nat-state=srcnat connection-state=new
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" \
    connection-state=established,related
add action=accept chain=forward comment=\
    "Accept established and related packets" connection-state=\
    established,related
add action=accept chain=forward comment=\
    "Allow outbound traffic from .101 to Cable internet" in-interface=\
    E02-p4_101 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment=\
    "Allow outbound traffic from .102 to Cable internet" in-interface=\
    VLAN_102 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment=\
    "Allow outbound traffic from .103 to Cable internet" in-interface=\
    E03-p6_103 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment=\
    "Allow outbound traffic from .104 to Cable internet" in-interface=\
    VLAN_104 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment=\
    "Allow outbound traffic from .105 to Cable internet" in-interface=\
    VLAN_105 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment=\
    "Allow outbound traffic from .106 to Cable internet" in-interface=\
    VLAN_106 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment=\
    "Allow outbound traffic from .209 to Cable internet" in-interface=\
    VLAN_209 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment=\
    "Allow outbound traffic from AREDN WAN to Cable internet" in-interface=\
    VLAN_001 out-interface=E01-p2_Cable_Internet
add action=accept chain=forward comment="Allow internet to AREDN WAN" \
    in-interface=E01-p2_Cable_Internet out-interface=VLAN_001
add action=drop chain=forward comment=\
    "Drop any forward packets that get this far"

add action=drop chain=Attack comment=\
    "Drop all packets from IPs on the Manual Blacklist" log=yes log-prefix=\
    "Manual Blacklist" src-address-list="Manual Blacklist"
add action=drop chain=Attack comment=\
    "Detect and drop TCP port scan connections" protocol=tcp psd=21,3s,3,1
add action=drop chain=Attack comment=\
    "Detect and drop UDP port scan connections" protocol=udp psd=21,3s,3,1
add action=return chain=Attack comment=\
    "Prevent safe IPs from getting tarpitted." src-address-list=Safe
add action=tarpit chain=Attack comment="Suppress DoS attackby tarpitting" \
    connection-limit=3,32 protocol=tcp src-address-list=Black_list
add action=add-src-to-address-list address-list=Black_list \
    address-list-timeout=1d chain=Attack comment="Detect DoS attack" \
    connection-limit=10,32 log=yes log-prefix="Black list" protocol=tcp
add action=return chain=Attack comment="Return from Attack chain"

add action=accept chain=ICMP comment=\
    "Accept ICMP type 0:0 (Echo reply) and limit to 5 packets / sec" \
    icmp-options=0:0-255 limit=5,5 protocol=icmp
add action=accept chain=ICMP comment="Accept ICMP type 3:3 (Destination host u\
    nreachable) and limit to 5 packets / sec" icmp-options=3:3 limit=5,5 \
    protocol=icmp
add action=accept chain=ICMP comment="Accept ICMP type 3:4 (Fragmentation requ\
    ired) and limit to 5 packets / sec" icmp-options=3:4 limit=5,5 protocol=\
    icmp
add action=accept chain=ICMP comment="Accept ICMP type 8:0 (Echo request) and \
    limit to 5 packets / sec via Fiber" icmp-options=8:0-255 in-interface=\
    E10-Fiber limit=5,5:packet protocol=icmp
add action=accept chain=ICMP comment=\
    "Accept ICMP type 8:0 (Echo request) and limit to 5 packets / sec" \
    icmp-options=8:0-255 limit=5,5 protocol=icmp
add action=accept chain=ICMP comment=\
    "Accept ICMP type 11:0 (Time exceeded) and limit to 5 packets / sec" \
    icmp-options=11:0-255 limit=5,5 protocol=icmp
add action=drop chain=ICMP comment="Drop all other ICMP packets" protocol=\
    icmp
add action=return chain=ICMP comment="Return from ICMP chain"

/ip firewall mangle
add action=mark-connection chain=prerouting dst-address=192.168.101.251 \
    dst-port=53 layer7-protocol=local.mesh new-connection-mark=\
    local.mesh-forward passthrough=yes protocol=tcp
add action=mark-connection chain=prerouting dst-address=192.168.101.251 \
    dst-port=53 layer7-protocol=local.mesh new-connection-mark=\
    local.mesh-forward passthrough=yes protocol=udp
add action=mark-connection chain=prerouting in-interface=VLAN_201 \
    new-connection-mark=DSL passthrough=yes

/ip firewall nat
add action=masquerade chain=srcnat comment="Masquerade for cable" \
    out-interface=E01-p2_Cable_Internet
add action=masquerade chain=srcnat comment="Masquerade for Fiber" \
    out-interface=E10-Fiber
add action=masquerade chain=srcnat comment=\
    "Outbound masquerade for traffic to VLAN 11" dst-address=10.113.6.64/29
add action=masquerade chain=srcnat comment=\
    "Outbound masquerade for traffic to VLAN 12" dst-address=10.115.242.96/29
add action=masquerade chain=srcnat comment=\
    "Outbound masquerade for traffic to VLAN 13" dst-address=10.115.244.80/29
add action=masquerade chain=srcnat comment="Masquerade for AREDN LAN" \
    out-interface=VLAN_005
add action=dst-nat chain=dstnat comment="Web Server on Jupiter." \
    dst-address-type=local dst-port=80 in-interface=!VLAN_005 protocol=tcp \
    to-addresses=192.168.101.11 to-ports=80
add action=dst-nat chain=dstnat comment="Web Server on Jupiter from AREDN." \
    dst-address-type=local dst-port=80 in-interface=VLAN_005 protocol=tcp \
    to-addresses=192.168.101.11 to-ports=80
add action=dst-nat chain=dstnat comment="AREDN Tunnel server" dst-port=5525 \
    in-interface=E01-p2_Cable_Internet protocol=tcp to-addresses=192.168.1.71 \
    to-ports=5525
add action=dst-nat chain=dstnat connection-mark=local.mesh-forward \
    to-addresses=10.9.60.81
add action=masquerade chain=srcnat connection-mark=local.mesh-forward
/ip firewall service-port
set ftp disabled=yes
set tftp disabled=yes
set irc disabled=yes
set h323 disabled=yes
set sip disabled=yes
set pptp disabled=yes
set udplite disabled=yes
set dccp disabled=yes
set sctp disabled=yes

/ip route
add distance=1 gateway=Fiber.GW.IP.Address routing-mark=via-FO
add comment="AREDN hAP-at-Home" distance=1 dst-address=10.0.0.0/8 gateway=\
    10.9.60.81
add distance=1 dst-address=172.16.0.0/12 gateway=10.9.60.81
add distance=1 dst-address=192.168.0.96/28 gateway=192.168.211.250
add distance=1 dst-address=192.168.202.0/24 gateway=192.168.211.252
add disabled=yes distance=1 dst-address=192.168.203.0/24 gateway=\
    192.168.211.252
add distance=1 dst-address=192.168.204.0/24 gateway=192.168.211.252
add distance=1 dst-address=192.168.205.0/24 gateway=192.168.211.252

/ip route rule
add action=lookup-only-in-table interface=E10-Fiber table=via-FO
add action=lookup-only-in-table interface=VLAN_201 table=via-FO
add action=lookup-only-in-table interface=VLAN_202 table=via-FO
add action=lookup-only-in-table interface=VLAN_203 table=via-FO
add action=lookup-only-in-table interface=VLAN_204 table=via-FO

/ip service
set telnet disabled=yes
set ftp port=<redacted>
set www port=<redacted>
set ssh port=<redacted>
set www-ssl disabled=no port=<redacted>
set api disabled=yes
set winbox port=<redacted>
set api-ssl disabled=yes
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/ipv6 dhcp-client
add add-default-route=yes interface=E01-p2_Cable_Internet pool-name=pool1 \
    pool-prefix-length=56 request=address,prefix
/ipv6 firewall filter
add action=accept chain=input comment=\
    "Accept Established and Related packets" connection-state=\
    established,related
add action=drop chain=input comment="Drop all ICMPv6 packets from cable" \
    disabled=yes in-interface=E01-p2_Cable_Internet protocol=icmpv6
add action=accept chain=input comment="Accept all ICMPv6 packets from cable." \
    in-interface=E01-p2_Cable_Internet protocol=icmpv6
add action=accept chain=input comment=\
    "Accept all ICMPv6 packets from LAN interface list." in-interface-list=\
    LAN protocol=icmpv6
add action=drop chain=input comment="Drop all IPV6 packets from cable" \
    in-interface=E01-p2_Cable_Internet log-prefix=IPV6
add action=drop chain=input comment=\
    "Drop all IPV6 packets from all interfaces - optionally log them" \
    disabled=yes log-prefix=IPV6
add action=drop chain=input comment="Drop all input IPV6 packets"
add action=accept chain=forward comment=\
    "Accept Established and Related packets" connection-state=\
    established,related
add action=accept chain=forward comment=\
    "Accept outbound IPv6 packets from .101 LAN" in-interface=E02-p4_101
add action=drop chain=forward comment="Drop all forwarded IPV6 packets"
/ipv6 nd
set [ find default=yes ] advertise-dns=no disabled=yes
/system clock
set time-zone-name=America/Los_Angeles
/system identity
set name=RB4011iGS+
/system logging
set 1 disabled=yes
set 2 disabled=yes
add action=remote topics=info
add action=remote topics=error
add action=remote topics=critical
add action=disk topics=critical
add action=disk topics=error
add action=remote topics=warning
add action=disk topics=warning
add disabled=yes topics=critical
/system ntp client
set enabled=yes primary-ntp=192.73.242.152 secondary-ntp=132.163.97.4 \
    server-dns-names=pool.ntp.org
/system resource irq rps
set E01-p2_Cable_Internet disabled=no
set E02-p4_101 disabled=no
set E03-p6_103 disabled=no
set E04-p8_802.1Q disabled=no
set "E05 to Router 2" disabled=no
/tool graphing interface
add allow-address=192.168.101.11/32
add allow-address=192.168.101.43/32
add allow-address=192.168.201.11/32
add allow-address=192.168.201.43/32
add allow-address=192.168.101.171/32
/tool graphing resource
add allow-address=192.168.101.11/32
add allow-address=192.168.101.43/32
add allow-address=192.168.201.11/32
add allow-address=192.168.201.43/32
add allow-address=47.44.2.90/32
add allow-address=192.168.101.171/32
/tool mac-server
set allowed-interface-list=mactel
/tool mac-server mac-winbox
set allowed-interface-list=mac-winbox
/tool sniffer
set file-name="Echo 2" filter-interface=VLAN_106 filter-ip-address=\
    192.168.106.12/32

I think I cleaned up anything that would be sensitive without deleting anything that is needed. Ask if something does not make sense…
See anything I did stupid?

anav · October 1, 2021, 1:23am

This looks funny to me.
/ip route rule
add action=lookup-only-in-table interface=E10-Fiber table=via-FO

why would you put the wan interface as the interface you want to route out a wan interface??
probably completely legit, but seems weird.

as for the rest way to complex for me to comment responsibly.

k6ccc · October 1, 2021, 1:33am

It’s what Sindy recommended above.
Didn’t make a lot of sense to me either. I tried it both with and without this one

sindy · October 1, 2021, 9:03am

I actually haven’t recommended that, but it needs a deeper explanation.

In firewall rules, you can match on both in-interface(-list) and on out-interface(-list), because some firewall chains handle the packets after routing has been done and thus the out-interface has already been determined. But the routing rules are used before the routing takes place, so the interface there actually means in-interface. Hence a route rule matching on interface=E10-Fiber only handles packets that came in via E10-Fiber, and if you send them to table via-FO, they get hairpinned back to the fibre so they never reach anything on LAN. The other rules are correct - “if it came in via vlan-xxx, send it via fibre” is what you actually want. So remove the wrong rule, and try pinging something from a host in one of the VLANs mentioned in the other rules - it should work allright, and sniffer/torch should show you these packets on E10-fiber.
packets sent by the router itself are a separate discipline. First, for these packets, there is no in-interface, so only routing rules not matching on any interface can handle them. Second, when an initial request packet (such as a DNS query one) is sent by the router itself, the router first determines the route based on the destination address, and based on that, it chooses the source address for that packet - the one attached to the out-interface determined by the routing, or the one specified as pref-src on the route. For some services (:ping, /system ssh, /system telnet), you can override both by specifying src-address as a parameter of the command.
So if you want to try the fiber connection from the router itself, you have to add a routing rule
src-address=the.ip.on.E10-fibre action=lookup-only-in-table interface=E10-Fiber table=via-FO
and then use ping 8.8.8.8 src-address=the.ip.on.E10-fibre to test that the idea works.
The same routing rule will also make the router start responding to pings coming from outside to the.ip.on.E10-fibre

anav · October 1, 2021, 11:40am

The short version…
So remove the wrong rule
A unique situation can describe something similar but you will never need it. )

(Q. I have never used both source address and interface at the same time in a route rule, is that legal??)
(src-address=the.ip.on.E10-fibre action=lookup-only-in-table interface=E10-Fiber table=via-FO)

Sounds like I need to explore the ways one can use src-address/dst-address/routingmark/interface/ in combo for example… I have only used it to identify a LAN or some portion of a LAN to direct out a specific WAN, to avoid the dreaded mangling.

sindy · October 1, 2021, 12:20pm

@anav, I’ve come across a tutorial on cultural differences (on LinkedIn I suppose, but don’t remember exactly) - in some cultures, people expect first the explanation of the reasons and then a suggestion of the solution, whilst on others, they want to hear the solution first and then the reasons that lead to such a suggestion. If you choose the “wrong” order, the audience may get so disappointed they won’t even listen further.

But even in this tutorial, the short version of only suggesting the solution and not giving the reasons at all has been mentioned

To the Q, sure you can use both interface and src-address in the same routing rule, but such a rule will never match if the src-address is an own address of the router attached to the interface. But there’s nothing wrong in saying that if host x.x.x.x (src-address) wants to connect somewhere from vlan-201 (interface), it should use routing table yyy, whereas if the packet from the same source address x.x.x.x comes in via another interface, it should use routing table zzz. This is the essence of the virtual router functionality.

anav · October 1, 2021, 3:02pm

Thanks much Sindy,
I am now wondering of the differences between using dstnat or routes to move packets around within the router (besides off the router which is usually routing and port forwarding which is usually dstnat).

k6ccc · October 1, 2021, 4:25pm

I fully admit that I know enough to be dangerous on some of this stuff. Simple routing I understand, but more complex is harder. I’m trying to understand Sindy’s recommendation in his first post so I can complete troubleshooting this and make it all work right.
In very general terms, I understand the concept of routing marks. At some point a routing mark gets applied, and later on, some routing decision is made based on the presence of the routing mark.
So looking at Sindy’s original recommendation…

I did create a confusion that got brought up earlier in that LAN 200 IS actually the DSL (now fiber) WAN interface. Obviously Sindy assume (reasonably) that LAN 200 was just another one of the LANs. My drawing extract in my first post created that confusion as I did not make any distinction of the 200 LAN (and 100 for the cable internet) were in fact the internet connections. I knew what they meant, but the drawing did not explain it. Sorry about that.

BTW, I’m going to largely write this based on the 2nd WAN still being on the DSL because that is how the recommendation was written.
So looking at Sindy’s recommendation, the first line adds a route for packets that have a routing mark of “via-Extreme” to use a gateway of the IP address of the DSL Extreme gateway..
I get that part. Route packets with the “via-Extreme” routing mark so that they will use the DSL extreme internet WAN interface.

It’s these lines I don’t quite get: interface=vlan202 action=lookup-only-in-table table=via-Extreme
Does that mean that packets that arrive into the router on in this case VLAN202 are treated as if they have the routing mark (even though they don’t) or does that add the routing mark, or something I did not think of?

One of the things I did confirm last night is that packets coming in via the fiber are being received and processed by the router, but responses are not. I have a port knock routine that uses the fiber for one of the knocks, and that worked just fine.

I am at work today so won’t be able to do much with this until this evening. Although I do have remote access to the router, I don’t want to do something that breaks my access (and worse kills the web pages served by my web server) when I am not there to fix it…

sindy · October 1, 2021, 6:03pm

Exactly. Conclusion: you’re much better in reading my mind than I am in reading yours

It’s the second option, this is the rule to “add the routing mark”, or to “choose the routing table”, which is more or less the same thing - it’s yet again more of just a vernacular mess. A “routing mark” assigned by mangle rules is an instruction for the routing engine which “routing table” to use; a “routing mark” assigned to a route in configuration is an indication to which routing table that route belongs.

But the routing rules have one extra capability - they can translate one routing mark, assigned by the previous step of packet processing (mangle rules in prerouting or output chain, or VRF) to another one (or another ones depending on the additional match conditions), or they can be used to prevent the “fallback to main routing table if there is no route for the packet in the specified table”. So they have one routing mark as a match parameter, and another one as an output parameter. And whereas mangle rules address this by using routing-mark as the match parameter name and new-routing-mark as the output parameter name, routing rules use table as the output parameter instead.

The router actually did process them (even while the “wrong” routing rule was in place, because packets for the router itself are not handled by routing rules), but the responses used routing table main. So you would have seen them if you ran torch on the other WAN. BTW, when analysing routing/nat/firewall mysteries, I find /tool sniffer much more useful than /tool torch.

k6ccc · October 1, 2021, 6:38pm

Sorry for creating the confusion on the 200 LAN. Although I understand it, I may edit my drawing a bit in case something like this comes up again.

Thanks for the explanation. I think I knew a good part of what you said about routing marks, but you filled in a few blanks.

Torch is very useful, but correct me if I’m wrong, it only can see packets coming into the router on a specified interface, and not packets leaving the router on a specified interface.
The packet sniffer can see inbound or outbound or both on a specified interface, but you only get very limited real-time information - most if the data is only available after you stop the packet capture - whereas Torch is real time.

sindy · October 1, 2021, 6:54pm

Torch shows both directions on an interface, but its notion of “in” and “out” may be confusing.

It depends on what you’re hunting for.
/tool sniffer quick ip-address=9.9.9.9 ip-protocol=icmp will show you, in real time, how the ping requests leave towards 9.9.9.9 and ping responses arrive from there, on any interface, or even on multiple interfaces if there are bridges and/or vlans and/or bonds and/or if the traffic is forwarded.

k6ccc · October 1, 2021, 11:23pm

I am making a bunch of progress! I can now ping the router via the fiber interface from the internet. I can also reach the one device on my .203 LAN that has a web server on the .203 LAN from the internet. I had one minor issue. Router 2 (the old one for the DSL) had an address on the .203 LAN of 192.168.203.252 whereas router 1 (the combined) has an address on that LAN of 192.168.203.251. That would not be an issue with a remote device using DHCP, but that environmental monitor has a static IP and gateway. So it was trying to respond via a GW on 192.168.203.252 which does not exist. As a stopgap until the person who manages that device can change the gateway to 192.168.203.251, I simply added a second address for the router on the .203 LAN. Working!

Thanks for the help guys.
I’m learning!