Thanks to some great help here from Anav I’ve gotten something partially working now. My use case is that I’m on Comcast business and switching to Xfinity because its cheaper. I have to leave some traffic via Comcast for a while (mail primarily) and migrate the rest to Xfinity so I’m using routing marks to do it. I’m not a network guru by any stretch but have done network programming a long time.
To begin testing the routing table approach, I set up a test VM with a fixed IP 192.168.4.149 and mark its outgoing connections and assign routing in the mangle table such that it goes via XFinity instead of Comcast. All other traffic is defaulting through Comcast.
Here is the weirdness. Traffic routed through XFinity (ether3.xfinity interface) from the .149 VM is super slooow. wget a basic html page takes like 15-20 seconds. So I try to figure out what is going on. I open up Torch to listen on ether3.xfinity and all of a sudden the traffic is zooming. The wget tests take less than a second. Stop torch and it slows back down to 15 seconds! Start torch again and it gets fast again.
This same behavior happens if I use packet sniffer on ether3.xfinity instead of torch. With packet sniffer capturing it runs really fast, Turn off packet sniffer and it takes 15-20 seconds again for a single page.
Any ideas? This is mind blowing to me 
I know it must be something simple, a check box or something, but I’ve been trying to figure this out for a few days with no luck.
Below is my config (export terse) with some interspersed comments by me.
Any help would be greatly appreciated. I may go broke paying two ISP bills while trying to learn the ins and outs of Mikrotik, although I love the flexibility
# jun/28/2024 15:41:50 by RouterOS 7.8
#
# model = RB760iGS
/interface bridge add admin-mac=18:FD:74:DE:7F:89 auto-mac=no comment=defconf name=mainbridge
/interface ethernet set [ find default-name=ether1 ] name=ether1.comcast
/interface ethernet set [ find default-name=ether2 ] name=ether2.lan
/interface ethernet set [ find default-name=ether3 ] name=ether3.xfinity
/interface list add comment=defconf name=WAN
/interface list add comment=defconf name=LAN
/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik
/ip hotspot profile set [ find default=yes ] html-directory=hotspot
/ip pool add name=dot4pool ranges=192.168.4.150-192.168.4.250
/ip dhcp-server add address-pool=dot4pool interface=mainbridge name=dot4dhcpsrvr
/port set 0 name=serial0
/routing table add fib name=ComcastRoutes
/routing table add disabled=no fib name=XFinityRoutes
Here are my routes to include dynamics
/ip route print
Flags: D - DYNAMIC; A - ACTIVE; c, s, y - COPY
Columns: DST-ADDRESS, GATEWAY, DISTANCE
# DST-ADDRESS GATEWAY DISTANCE
0 s 0.0.0.0/0 <xfinity-gateway> 3
1 As 0.0.0.0/0 <comcast-gateway> 2
DAc <comcast-network>/30 ether1.comcast 0
DAc 71.199.144.0/23 ether3.xfinity 0
DAc 192.168.4.0/24 mainbridge 0
2 As 0.0.0.0/0 <comcast-gateway> 1
3 As 0.0.0.0/0 <xfinity-gateway> 1
/system logging action add name=rsyslog remote=192.168.4.60 target=remote
/interface bridge port add bridge=mainbridge comment=defconf interface=ether2.lan
/interface bridge port add bridge=mainbridge comment=defconf interface=ether4
/interface bridge port add bridge=mainbridge comment=defconf interface=ether5
/interface bridge port add bridge=mainbridge comment=defconf interface=sfp1
/ip neighbor discovery-settings set discover-interface-list=LAN
/ipv6 settings set disable-ipv6=yes
/interface list member add comment=defconf interface=mainbridge list=LAN
/interface list member add comment=defconf interface=ether1.comcast list=WAN
/interface list member add interface=ether3.xfinity list=WAN
/ip address add address=<my-public-comcast-ip>/30 interface=ether1.comcast network=<comcast-network>
/ip address add address=192.168.4.254/24 interface=mainbridge network=192.168.4.0
/ip dhcp-client add comment=defconf disabled=yes interface=ether1.comcast use-peer-dns=no use-peer-ntp=no
/ip dhcp-client add add-default-route=no interface=ether3.xfinity use-peer-dns=no use-peer-ntp=no
# This is the vm I'm testing to send traffic out via Xfinity instead of Comcast
/ip dhcp-server lease add address=192.168.4.149 comment="Kubntu test VM for debugging traffic delays" mac-address=00:0C:29:AB:36:C4 server=dot4dhcpsrvr
/ip dhcp-server network add address=192.168.4.0/24 dns-server=192.168.4.12,192.168.4.11 gateway=192.168.4.254 ntp-server=192.168.4.12,192.168.4.18,192.168.4.11
/ip dns set allow-remote-requests=yes servers=192.168.4.12,192.168.4.11
/ip dns static add address=192.168.88.1 comment=defconf name=router.lan
/ip firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=input comment=INPUTDROP1 connection-state=invalid log=yes log-prefix=INPUTDROP1
/ip firewall filter add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
/ip firewall filter add action=accept chain=input comment="defconf: accept to local loopback (for CAPsMAN)" dst-address=127.0.0.1
/ip firewall filter add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN log=yes log-prefix=INPUTDROP2
/ip firewall filter add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
/ip firewall filter add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
/ip firewall filter add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related hw-offload=yes
/ip firewall filter add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
/ip firewall filter add action=drop chain=forward comment=INPUTDROP3 connection-state=invalid log=yes log-prefix=FORWARDDROP3
/ip firewall filter add action=drop chain=forward comment=INPUTDROP4 connection-nat-state=!dstnat connection-state=new in-interface-list=WAN log=yes log-prefix=FORWARDDROP4
/ip firewall mangle add action=mark-connection chain=prerouting comment="Mark mail server connections going external" connection-mark=no-mark connection-state=new dst-address-type=!local dst-port=25 in-interface-list=LAN new-connection-mark=SMTPOutbound passthrough=yes protocol=tcp src-address=192.168.4.12
#Here is where I mark the connection coming from my test VM
/ip firewall mangle add action=mark-connection chain=prerouting comment="Mark TEST outbound connections" connection-mark=no-mark connection-state=new dst-address-type=!local in-interface-list=LAN new-connection-mark=TestConnOutbound passthrough=yes src-address=192.168.4.149
/ip firewall mangle add action=mark-connection chain=prerouting comment="Mark smtp connections coming in from Comcast" connection-mark=no-mark connection-state=new dst-port=25 in-interface=ether1.comcast new-connection-mark=SMTPConnFromComcast passthrough=yes protocol=tcp src-port=""
/ip firewall mangle add action=mark-connection chain=prerouting comment="Mark all connections coming in from Comcast" connection-mark=no-mark connection-state=new in-interface=ether1.comcast new-connection-mark=ConnFromComcast passthrough=yes
/ip firewall mangle add action=mark-connection chain=prerouting comment="Mark connections from LAN going out" connection-mark=no-mark connection-state=new dst-address-type=!local in-interface-list=LAN new-connection-mark=ConnFromLanToExternal passthrough=yes
/ip firewall mangle add action=mark-routing chain=prerouting comment="Route smtp connections initiated from mail server" connection-mark=SMTPOutbound in-interface-list=LAN new-routing-mark=ComcastRoutes passthrough=no
# Here is where I set the outbound traffic from my test VM to use the XFinity routes
/ip firewall mangle add action=mark-routing chain=prerouting comment="Route phone outbound connections" connection-mark=TestConnOutbound in-interface-list=LAN new-routing-mark=XFinityRoutes passthrough=no routing-mark=!XFinityRoutes
# These pass-throughs are just where I'm marking stuff to understand flow better
/ip firewall mangle add action=passthrough chain=prerouting comment="Route smtp connections initiated from Comcast" connection-mark=SMTPConnFromComcast connection-nat-state=dstnat in-interface=ether1.comcast
/ip firewall mangle add action=passthrough chain=forward comment="Route smtp connections initiated from Comcast" connection-mark=SMTPConnFromComcast
/ip firewall mangle add action=passthrough chain=output comment="Route smtp connections initiated from Comcast" connection-mark=SMTPConnFromComcast
/ip firewall mangle add action=passthrough chain=output comment="Catch all - Route connections established inbound from Comcast" connection-mark=ConnFromComcast
/ip firewall mangle add action=passthrough chain=output comment="Catch all - Route LAN connections going out" connection-mark=ConnFromLanToExternal dst-address-type=!local
/ip firewall mangle add action=mark-routing chain=output new-routing-mark=XFinityRoutes out-interface=ether3.xfinity passthrough=yes
/ip firewall mangle add action=mark-routing chain=output new-routing-mark=ComcastRoutes out-interface=ether1.comcast passthrough=yes
/ip firewall nat add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
# This is to support existing services on my old Comcast connection until I get them switched over
/ip firewall nat add action=dst-nat chain=dstnat comment="SMTP From Internet" dst-address=<my-public-comcast-ip> dst-port=25 protocol=tcp to-addresses=192.168.4.12
/ip firewall nat add action=dst-nat chain=dstnat comment="Plex From Internet" dst-address=<my-public-comcast-ip> dst-port=30103 protocol=tcp to-addresses=192.168.4.12 to-ports=32400
/ip firewall nat add action=dst-nat chain=dstnat comment="IMAPS From Internet" dst-address=<my-public-comcast-ip> dst-port=993 protocol=tcp to-addresses=192.168.4.12
/ip firewall nat add action=dst-nat chain=dstnat comment="HTTPS From Internet" dst-address=<my-public-comcast-ip> dst-port=443 protocol=tcp to-addresses=192.168.4.12
/ip firewall nat add action=dst-nat chain=dstnat comment="DS Cloud From Internet" dst-address=<my-public-comcast-ip> dst-port=6690 protocol=tcp to-addresses=192.168.4.12 to-ports=32400
/ip firewall nat add action=dst-nat chain=dstnat comment="SUBMISSION (mail protocol 587) From Internet" dst-address=<my-public-comcast-ip> dst-port=587 protocol=tcp to-addresses=192.168.4.12
/ip route add disabled=no distance=2 dst-address=0.0.0.0/0 gateway=<comcast-gateway> pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
/ip route add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=<comcast-gateway> pref-src="" routing-table=ComcastRoutes scope=30 suppress-hw-offload=no target-scope=10
/ip route add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=<xfinity-gateway> pref-src="" routing-table=XFinityRoutes scope=30 suppress-hw-offload=no target-scope=10
/ip route add disabled=no distance=3 dst-address=0.0.0.0/0 gateway=<xfinity-gateway> pref-src="" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
# I have IPV6 disabled but left these rules in here for now so I could study them later
/ipv6 firewall address-list add address=::/128 comment="defconf: unspecified address" list=bad_ipv6
/ipv6 firewall address-list add address=::1/128 comment="defconf: lo" list=bad_ipv6
/ipv6 firewall address-list add address=fec0::/10 comment="defconf: site-local" list=bad_ipv6
/ipv6 firewall address-list add address=::ffff:0.0.0.0/96 comment="defconf: ipv4-mapped" list=bad_ipv6
/ipv6 firewall address-list add address=::/96 comment="defconf: ipv4 compat" list=bad_ipv6
/ipv6 firewall address-list add address=100::/64 comment="defconf: discard only " list=bad_ipv6
/ipv6 firewall address-list add address=2001:db8::/32 comment="defconf: documentation" list=bad_ipv6
/ipv6 firewall address-list add address=2001:10::/28 comment="defconf: ORCHID" list=bad_ipv6
/ipv6 firewall address-list add address=3ffe::/16 comment="defconf: 6bone" list=bad_ipv6
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept ICMPv6" protocol=icmpv6
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept UDP traceroute" port=33434-33534 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept DHCPv6-Client prefix delegation." dst-port=546 protocol=udp src-address=fe80::/10
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept ipsec AH" protocol=ipsec-ah
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept ipsec ESP" protocol=ipsec-esp
/ipv6 firewall filter add action=accept chain=input comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
/ipv6 firewall filter add action=drop chain=input comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop packets with bad src ipv6" src-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop packets with bad dst ipv6" dst-address-list=bad_ipv6
/ipv6 firewall filter add action=drop chain=forward comment="defconf: rfc4890 drop hop-limit=1" hop-limit=equal:1 protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept ICMPv6" protocol=icmpv6
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept HIP" protocol=139
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept IKE" dst-port=500,4500 protocol=udp
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept ipsec AH" protocol=ipsec-ah
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept ipsec ESP" protocol=ipsec-esp
/ipv6 firewall filter add action=accept chain=forward comment="defconf: accept all that matches ipsec policy" ipsec-policy=in,ipsec
/ipv6 firewall filter add action=drop chain=forward comment="defconf: drop everything else not coming from LAN" in-interface-list=!LAN
/system clock set time-zone-name=America/New_York
/system identity set name=fw1
/system logging add action=remote topics=firewall
/system logging add action=remote topics=debug
/system logging add action=rsyslog topics=info
/system ntp client set enabled=yes
/system ntp server set enabled=yes
/system ntp client servers add address=0.us.pool.ntp.org
/system ntp client servers add address=1.us.pool.ntp.org
/system ntp client servers add address=2.us.pool.ntp.org
/tool mac-server set allowed-interface-list=LAN
/tool mac-server mac-winbox set allowed-interface-list=LAN
/tool romon set enabled=yes id=00:00:00:00:00:08
/tool sniffer set filter-interface=ether3.xfinity