2 PPPoE - 2 Networks - 1 RB - advice needed.

RouterOS General
1

Hi

I am usually quite sharp at solving these problems, but I simply cannot get around this one.

We have one RB2011

I have 13 Interfaces setup. 10 Physical LAN1-10, 1 Bridge (LAN 6 & 7) and 2 PPPoE.

LAN1 is connected to a 100/50 GPoN Fibre CPE that authenticates via PPPoE

PPPoE_1 uses a dynamic IP address, but we can turn on a static x.x.x.99 when we need it.
PPPoE_2 uses a static IP x.x.x.49 address.

PPPoE_1 is an uncapped account, but due to issues with Netflix and Playstore, we only use the Static IP when we need to complete a specific task. Then we go back to dynamic.
PPPoE_2 is a 20GB capped account that we use for a PBX only.

LAN2-5 and 8-10 are used for various segments on our network, and are required to use PPPoE_1.
LAN6-7 are for our PBX and Voip Phones and are on a Bridge brPBX. The IP Address range assigned to the bridge is 192.168.26.0/24.

I have Masquerading setup on both PPPoE and the Metric for PPPoE_1 is 1 and PPPoE_2 is 2. So all traffic defaults to PPPoE_1.

What I also did was setup NAT so that traffic coming in through x.x.x.49 points to the PBX. I can connect remote phones and also to the PBX GUI via this static IP, and we can make calls, but we have a small issue. The inbound packets are coming in through PPPoE_2 but then return to the client via PPPoE_1 which is causing hectic Jitter and Packet Loss of about 40-70% which makes call in audible on the client device.

So what I want to do is setup that any packets that come from brPBX need to go out PPPoE_2 and then everything else goes out PPPoE_1.

I have tried using routing tables, mangle, and NAT rules, but nothing seems to work in any way, or if it does have an effect it results in me driving out to the office to undo what I did as it kills all traffic.

Any ideas would welcome and appreciated.

2

Post the complete current config minus the sensitive info as suggested in my automatic signature below. Policy routing should work normally, so it must have been some minor mistake in the way you’ve set it up.

3 
# sep/29/2020 23:13:13 by RouterOS 6.something
# software id = nope-nope
#
# model = 2011iL
# serial number = 
/interface bridge
add name=br.PBX
/interface ethernet
set [ find default-name=ether1 ] name=1.WAN.PPPoE
set [ find default-name=ether3 ] name=3.Office
set [ find default-name=ether4 ] name=4.Thinclient.RemDesktop
set [ find default-name=ether5 ] name=5.Digit
set [ find default-name=ether6 ] name=6.PBX
set [ find default-name=ether7 ] name=7.PBX
set [ find default-name=ether9 ] name=9.PBX.PPPoE
set [ find default-name=ether10 ] name=10.BraaiFi poe-out=forced-on
set [ find default-name=ether2 ] disabled=yes
set [ find default-name=ether8 ] disabled=yes
/interface pppoe-client
add add-default-route=yes default-route-distance=10 disabled=no interface=\
    1.WAN.PPPoE name=pppoe-pbx service-name=PBX use-peer-dns=yes user=\
    randompppoeusername
add add-default-route=yes disabled=no interface=1.WAN.PPPoE name=\
    pppoe-main service-name=ESIR use-peer-dns=yes user=\
    anotherrandompppoeusername
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip firewall layer7-protocol
add name=torrentsites regexp="^.*(get|GET).+(torrent|thepiratebay|isohunt|ente\
    rtane|demonoid|btjunkie|mininova|flixflux|torrentz|vertor|h33t|btscene|bit\
    unity|bittoxic|thunderbytes|entertane|zoozle|vcdq|bitnova|bitsoup|meganova\
    |fulldls|btbot|flixflux|seedpeer|fenopy|gpirate|rarbg|commonbits).*\$"
add name=torrents regexp="^.*(get|GET).+(torrent|thepiratebay|isohunt|entertan\
    e|demonoid|btjunkie|mininova|flixflux|vertor|h33t|zoozle|bitnova|bitsoup|m\
    eganova|fulldls|btbot|fenopy|gpirate|commonbits|1337x|bitlord|rarbg|yts|ez\
    tv|piratebay|nyaa|zooqle|idope|kat|torlock|demoniod|monova|toorgle|seedpee\
    r|torrentz|rartv|ettv|bittorrent).*\$"
add name=torrent-dns regexp="^.+(torrent|thepiratebay|isohunt|entertane|demono\
    id|btjunkie|mininova|flixflux|torrentz|vertor|h33t|btscene|bitunity|bittox\
    ic|thunderbytes|entertane|zoozle|vcdq|bitnova|bitsoup|meganova|fulldls|btb\
    ot|flixflux|seedpeer|fenopy|gpirate|commonbits).*\$"
add name=torrent-wwws regexp="^.*(get|GET).+(torrent|thepiratebay|isohunt|ente\
    rtane|demonoid|btjunkie|mininova|flixflux|torrentz|vertor|h33t|btscene|bit\
    unity|bittoxic|thunderbytes|entertane|zoozle|vcdq|bitnova|bitsoup|meganova\
    |fulldls|btbot|flixflux|seedpeer|fenopy|gpirate|commonbits).*\$"
/ip pool
add name=pool.Digit ranges=192.168.66.2-192.168.66.254
add name=pool.Office ranges=192.168.1.100-192.168.1.149
add name=pool.BraaiFi ranges=192.168.99.2-192.168.99.254
add name=pool.PBX ranges=192.168.26.100-192.168.26.200
add name=pool.vpn ranges=192.168.13.100-192.168.13.199
add name=dhcp_remdesktop ranges=192.168.13.2-192.168.13.254
/ip dhcp-server
add address-pool=pool.Digit disabled=no interface=5.Digit lease-time=4w2d \
    name=dhcp.Digit
add address-pool=pool.Office disabled=no interface=3.Office lease-time=1d \
    name=dhcp.Office
add address-pool=pool.BraaiFi disabled=no interface=10.BraaiFi lease-time=6h \
    name=dhcp.BraaiFi
add address-pool=pool.PBX disabled=no interface=br.PBX lease-time=4w2d name=\
    dhcp.PBX
add address-pool=dhcp_remdesktop disabled=no interface=\
    4.Thinclient.RemDesktop lease-time=14w2d name=dhcp1
    own-routers,own-users,own-profiles,own-limits,config-payment-gw
/interface bridge port
add bridge=br.PBX interface=6.PBX
add bridge=br.PBX interface=7.PBX
/interface detect-internet
set detect-interface-list=all
/ip address
add address=192.168.66.1/24 interface=5.Digit network=192.168.66.0
add address=192.168.1.1/24 interface=3.Office network=192.168.1.0
add address=192.168.99.1/24 interface=10.BraaiFi network=192.168.99.0
add address=192.168.26.1/24 interface=br.PBX network=192.168.26.0
add address=192.168.13.1/24 interface=4.Thinclient.RemDesktop network=\
    192.168.13.0
/ip cloud
set ddns-enabled=yes ddns-update-interval=1m
/ip dhcp-server config
set store-leases-disk=1d6h
/ip dhcp-server network
add address=192.168.1.0/24 gateway=192.168.1.1
add address=192.168.13.0/24 gateway=192.168.13.1
add address=192.168.26.0/24 gateway=192.168.26.1
add address=192.168.66.0/24 gateway=192.168.66.1
add address=192.168.99.0/24 gateway=192.168.99.1
/ip firewall address-list
add address=192.168.66.247 disabled=yes list=192.168.66.247
add address=151.80.120.112/30 list=trackers
/ip firewall filter
add action=accept chain=forward connection-nat-state=dstnat disabled=yes \
    in-interface=*B protocol=tcp
add action=drop chain=forward disabled=yes dst-address-list=trackers
add action=drop chain=forward comment="block torrent wwws" disabled=yes \
    layer7-protocol=*3
add action=drop chain=forward comment="block torrent dns" disabled=yes \
    dst-port=53 layer7-protocol=torrent-dns protocol=udp
/ip firewall mangle
add action=mark-routing chain=prerouting in-interface=br.PBX \
    new-routing-mark=PBX passthrough=yes
add action=mark-routing chain=prerouting in-interface=!br.PBX \
    new-routing-mark=NotPBX passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat comment=WAN out-interface=\
    pppoe-main
add action=masquerade chain=srcnat comment=PBX out-interface=pppoe-pbx
add action=dst-nat chain=dstnat dst-port=5060 in-interface=pppoe-pbx \
    protocol=udp to-addresses=192.168.26.26 to-ports=5060
add action=dst-nat chain=dstnat dst-port=666 in-interface=pppoe-pbx protocol=\
    tcp to-addresses=192.168.26.26 to-ports=80
add action=dst-nat chain=dstnat dst-port=8111 in-interface=pppoe-pbx \
    protocol=udp to-addresses=192.168.26.26 to-ports=8111
add action=dst-nat chain=dstnat dst-port=10000-12000 in-interface=pppoe-pbx \
    protocol=udp to-addresses=192.168.26.26 to-ports=10000-12000
add action=dst-nat chain=dstnat comment="thinclient tcp" \
    dst-port=53435 in-interface=pppoe-main protocol=tcp \
    to-addresses=192.168.13.254 to-ports=53435
add action=dst-nat chain=dstnat comment="thinclient udp" \
    dst-port=53435 in-interface=pppoe-main protocol=udp \
    to-addresses=192.168.13.254 to-ports=53435
add action=dst-nat chain=dstnat comment="thinclient web" \
    dst-port=9999 in-interface=pppoe-pbx protocol=tcp to-addresses=\
    192.168.13.254 to-ports=9091
add action=dst-nat chain=dstnat dst-port=9999 in-interface=\
    pppoe-main protocol=tcp to-addresses=192.168.13.254 to-ports=80
/ip route
add disabled=yes distance=1 gateway=pppoe-pbx routing-mark=PBX
add disabled=yes distance=1 gateway=pppoe-main routing-mark=NotPBX
add distance=1 gateway=pppoe-main
add distance=2 gateway=pppoe-pbx
/ip service
set telnet address=192.168.0.0/16
set ftp address=192.168.0.0/16
set www address=192.168.0.0/16
set ssh address=192.168.0.0/16
set api disabled=yes
set api-ssl disabled=yes
/system clock
set time-zone-name=Nowhere/Middle
/system ntp client
set enabled=yes primary-ntp=91.189.91.157 secondary-ntp=91.189.89.199
/tool sniffer
set memory-limit=10000KiB
4

Eth9 - PBX-PPPOE is not being used. We were looking at getting a dedicated 10/10 fibre line for PBX but changed our minds

5

OK, I understand the issue. Enable just the route with routing-mark=PBX, but not the one with routing-mark=NotPBX, and you’ll be good.

The thing is that the mangle rule attaching the routing-mark NotPBX (which you can delete too once you test the above suggestion, as well as the route with routing-mark=NotPBX) attaches that routing mark also to packets coming in via any of the WAN interfaces (because the only match condition is in-interface=!br.PBX). And in the routing table with that routing-mark, there is just a default route, which sends the packets out via pppoe-main.

The routing-mark attribute of a route has precedence over distance, so the dynamically created routes to connected subnets do not supersede that routing-marked default route. Only if no route with the required routing-mark is found at all, the routing uses the routing table main (consisting of routes with no routing-mark attribute). So even though the mangle rule assigns the routing-mark NotPBX, if the default route bearing that routing-mark is disabled, everything works.

Other than that, your /ip firewall filter needs attention. There are rules which don’t do anything because the configuration items they refer to have been removed (so now there is *number instead, indicating an unresolvable link to an object), and it provides no protection of the devices in the LAN so you rely on just the NAT which is not enough (if someone knows the private IPs, they can talk to them via WAN).

6

Thanks Sindy

Yeah, the firewall rules are all disabled, as I was just testing something for a hotel that was having issues with guests killing their connection. Where they are based they can’t get faster than 4mbps and they have 40 rooms.

7

Thanks, it worked, but then all devices on the local network lost connectivity, so I added on the Mangle rule, DST ADD !192.168.0.0/16 and now everything works perfectly, thanks again.