Unable to Force IP Cloud Updates via a specific Interface

ronnielazarus · Mon Oct 17, 2022 7:31 pm

Hi good folks of the Mikrotik Community,

I am a newbie to Mikrotik but have tried my best to read, research and understand as much as I could before posting here. However, I'm currently having an ongoing issue which for the life of me I'm unable to get around and would really appreciate some help from the Community around the same. In the end, I believe my question is two folds.

So, my setup is as follows:

1. ISP1=BSNL & ISP2=NetPlus
1. ISP1-Modem (BridgedMode) and ISP2-Modem+Router (NAT) connected to Mikrotik RB750Gr3.
2. For ISP1 MT is connected via a PPPoE client whereas for ISP2 simply as a NATed Client with a static IP.
3. From the LAN Bridge of the MT one of the interfaces is connected to my main ASUS RTAX-88U which handles subsequent QOS + DHCP etc. for the end clients.
4. I have attempted to setup PCC with ISP2 getting higher priority by adding the relevant rule entry once more. This is done because ISP1=150MBPS whereas ISP2=350MBPS.
5. I believe that the PCC configuration is working, as I can see counters on both routing marked output chains incrementing. However, I'm unable to get aggregated speed while doing a speedtest.net for example. Although the MT does show upto 70% utilization when reaching max speeds of 300 MBPS and it never goes above that while I do see traffic flowing from both the interfaces in winbox. PS, In other round-robin like LB algos I was able to achieve upwards of 500MBPS with clients attemtping to do the same speedtest.net test when the LB was handled directly at my ASUS without the MT anywhere in the picture, this means that then the ISPs were terminated directly at the ASUS.
6. My ultimate intention is to setup IKEv2 VPN on my ASUS router by forwarding all the ports from MT to ASUS. However, I want to be able to use the IP Cloud functionality since I have a Dynamic Public IP on ISP1 and a Private IP on ISP2 and I want to be able to connect to an FQDN from a VPN client.
7. It was only while I tried forcing the Traffic for cloud/cloud2.mikrotik.com via the PPPoE gateway through multiple approaches is when I found things are not working as expected. By this I mean that when I put in the routing rules for the same, I can still see the traffic going out via ISP2 for some reason, and if I force it by using a routing rule, i'm simply unable to get a ping response back from my PC. All of this leads me to believe that there is a possibility of a deeper problem at work here. And since I'm new, it definitely seems like a possibility with every passing day.

So, in the interest of keeping this short, I would like to humbly request assitance in looking at my export conifg below and critiqueing my PCC config, as well as ways to optimise that.

Else if, everything with PCC is top-notch, I would love to know how can I force the IP Cloud traffic via ISP1 always. Here's my config below:

# oct/17/2022 21:33:08 by RouterOS 7.5
# software id = AU10-A1CY
#
# model = RB750Gr3
# serial number = 
/interface bridge
add name=bridge-LAN1andLAN2
/interface ethernet
set [ find default-name=ether1 ] l2mtu=1452 mac-address=F8:75:A4:AB:D1:AB \
    mtu=1452 name=ether1-BSNL
set [ find default-name=ether2 ] comment=NetPlus name=ether2-NetPlus
set [ find default-name=ether3 ] comment="LAN - To ASUS" name=ether3-LAN1
set [ find default-name=ether4 ] comment="LAN - Free" name=ether4-LAN2
set [ find default-name=ether5 ] comment="LAN - Management" name=\
    ether5-Management
/disk
set sd1 disabled=no
set sd1-part1 disabled=no name=disk1
/interface list
add comment="Contains all WAN Interfaces" name=WAN
add comment="Contains all LAN Interfaces" name=LAN
add comment="Contains Iterfaces with Internet" name=INTERNET
add comment="Contains all Management Interfaces" name=Management
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip pool
add name=dhcp_pool1 ranges=192.168.88.100-192.168.88.254
add name=dhcp_pool2 ranges=192.168.3.2-192.168.3.254
/ip dhcp-server
add address-pool=dhcp_pool1 interface=bridge-LAN1andLAN2 name=\
    Bridge_DCHP_Server
add address-pool=dhcp_pool2 interface=ether5-Management name=\
    Management_DCHP_Server
/port
set 0 name=serial0
/interface pppoe-client
add add-default-route=yes comment=BSNL disabled=no interface=ether1-BSNL \
    max-mru=1492 max-mtu=1452 name=PPPoE-BSNL profile=default-encryption \
    use-peer-dns=yes user=xxxxxxxxxx_xxx@xxxx.xxxx.xx
/routing table
add fib name=to_ISP_BSNL
add fib name=to_ISP_NetPlus
add disabled=no fib name=via_BSNL
/interface bridge port
add bridge=bridge-LAN1andLAN2 interface=ether3-LAN1
add bridge=bridge-LAN1andLAN2 interface=ether4-LAN2
/ip neighbor discovery-settings
set discover-interface-list=LAN
/interface detect-internet
set detect-interface-list=all internet-interface-list=INTERNET \
    lan-interface-list=LAN wan-interface-list=WAN
/interface list member
add interface=ether1-BSNL list=WAN
add interface=ether2-NetPlus list=WAN
add interface=ether3-LAN1 list=LAN
add interface=ether4-LAN2 list=LAN
add interface=PPPoE-BSNL list=WAN
add interface=PPPoE-BSNL list=INTERNET
add interface=ether2-NetPlus list=INTERNET
add interface=ether5-Management list=Management
/ip address
add address=192.168.88.1/24 comment=Bridge interface=bridge-LAN1andLAN2 \
    network=192.168.88.0
add address=192.168.1.12/24 interface=ether2-NetPlus network=192.168.1.0
add address=192.168.0.10/24 interface=ether1-BSNL network=192.168.0.0
add address=192.168.3.1/24 comment=Management interface=ether5-Management \
    network=192.168.3.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-server network
add address=192.168.3.0/28 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.3.1
add address=192.168.3.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.3.1
add address=192.168.88.0/24 dns-server=8.8.8.8,8.8.4.4,218.248.114.129 \
    gateway=192.168.88.1
add address=192.168.89.0/24 dns-server=8.8.8.8,8.8.4.4 gateway=192.168.89.1
/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4
/ip firewall address-list
add address=0.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=172.16.0.0/12 comment=RFC6890 list=not_in_internet
add address=192.168.0.0/16 comment=RFC6890 list=not_in_internet
add address=10.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=169.254.0.0/16 comment=RFC6890 list=not_in_internet
add address=127.0.0.0/8 comment=RFC6890 list=not_in_internet
add address=224.0.0.0/4 comment=Multicast list=not_in_internet
add address=198.18.0.0/15 comment=RFC6890 list=not_in_internet
add address=192.0.0.0/24 comment=RFC6890 list=not_in_internet
add address=192.0.2.0/24 comment=RFC6890 list=not_in_internet
add address=198.51.100.0/24 comment=RFC6890 list=not_in_internet
add address=203.0.113.0/24 comment=RFC6890 list=not_in_internet
add address=100.64.0.0/10 comment=RFC6890 list=not_in_internet
add address=240.0.0.0/4 comment=RFC6890 list=not_in_internet
add address=192.88.99.0/24 comment="6to4 relay Anycast [RFC 3068]" list=\
    not_in_internet
add address=255.255.255.255 comment=RFC6890 list=not_in_internet
add address=cloud.mikrotik.com list=mikrotik-cloud
add address=cloud2.mikrotik.com list=mikrotik-cloud
/ip firewall filter
add action=accept chain=input comment="Accept, Established & Related" \
    connection-state=established,related
add action=drop chain=input comment="Drop Invalid Connections" \
    connection-state=invalid
add action=accept chain=input comment="Allow ICMP" in-interface-list=WAN \
    protocol=icmp
add action=accept chain=input comment="Allow Winbox (1964)" \
    in-interface-list=WAN port=1964 protocol=tcp
add action=accept chain=input comment="Allow SSH (1963)" in-interface-list=\
    WAN port=1963 protocol=tcp
add action=drop chain=input comment="Block Everything Else" \
    in-interface-list=WAN
/ip firewall mangle
add action=mark-connection chain=prerouting comment="For Mikrotik Cloud" \
    connection-state=new dst-address-list=mikrotik-cloud log=yes \
    new-connection-mark=for_mikrotik_cloud passthrough=yes
add action=mark-routing chain=prerouting connection-mark=for_mikrotik_cloud \
    log=yes new-routing-mark=to_ISP_BSNL passthrough=no
add action=mark-connection chain=prerouting comment=\
    "For PCC between BSNL & NetPlus" connection-mark=no-mark in-interface=\
    PPPoE-BSNL new-connection-mark=ISP_BSNL passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark \
    in-interface=ether2-NetPlus new-connection-mark=ISP_NetPlus passthrough=\
    no
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_BSNL passthrough=yes \
    per-connection-classifier=both-addresses:2/0 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_NetPlus passthrough=yes \
    per-connection-classifier=both-addresses:2/1 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_NetPlus passthrough=yes \
    per-connection-classifier=both-addresses:2/2 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_BSNL passthrough=yes \
    per-connection-classifier=both-addresses:2/0 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_NetPlus passthrough=yes \
    per-connection-classifier=both-addresses:2/1 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_NetPlus passthrough=yes \
    per-connection-classifier=both-addresses:2/2 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_BSNL passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_NetPlus passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:2/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-list=!not_in_internet dst-address-type=!local \
    in-interface-list=LAN new-connection-mark=ISP_NetPlus passthrough=yes \
    per-connection-classifier=both-addresses-and-ports:2/2
add action=mark-routing chain=prerouting connection-mark=ISP_BSNL \
    in-interface-list=LAN new-routing-mark=to_ISP_BSNL passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP_NetPlus \
    in-interface-list=LAN new-routing-mark=to_ISP_NetPlus passthrough=no
add action=mark-routing chain=output connection-mark=ISP_BSNL \
    new-routing-mark=to_ISP_BSNL out-interface=PPPoE-BSNL passthrough=no
add action=mark-routing chain=output connection-mark=ISP_NetPlus \
    new-routing-mark=to_ISP_NetPlus out-interface=ether2-NetPlus passthrough=\
    no
/ip firewall nat
add action=masquerade chain=srcnat comment=\
    "General NAT for all WAN (includes Failover)" out-interface=PPPoE-BSNL
add action=masquerade chain=srcnat out-interface=ether2-NetPlus
add action=masquerade chain=srcnat comment="Specifically for Mikrotik Cloud" \
    disabled=yes dst-address-list=mikrotik-cloud log=yes
add action=dst-nat chain=dstnat comment="ASUS as a DMZ Host" in-interface=\
    PPPoE-BSNL log-prefix=Received_on_BSNL_ to-addresses=192.168.88.6
/ip route
add gateway=192.168.1.1
add check-gateway=ping comment="Load Balancing Route to ISP BSNL" disabled=no \
    distance=1 dst-address=0.0.0.0/0 gateway=PPPoE-BSNL pref-src="" \
    routing-table=to_ISP_BSNL scope=30 suppress-hw-offload=no target-scope=10
add check-gateway=ping comment="Load Balancing Route to ISP NetPlus" \
    distance=1 gateway=192.168.1.1 routing-table=to_ISP_NetPlus
add comment="For Cloud Update" disabled=no distance=1 dst-address=0.0.0.0/0 \
    gateway=PPPoE-BSNL pref-src="" routing-table=via_BSNL scope=30 \
    suppress-hw-offload=no target-scope=10
/ip service
set telnet disabled=yes
set ssh port=1963
set winbox port=1964
/routing rule
add action=lookup-only-in-table comment="For Mikrotik Cloud" disabled=no \
    dst-address=::/0 interface=PPPoE-BSNL routing-mark=via_BSNL table=\
    via_BSNL
/system clock
set time-zone-name=Asia/Kolkata
/system scheduler
add interval=1d name="Reboot everynight @ 4:30 AM" on-event="/system reboot" \
    policy=ftp,reboot,read,write,policy,test,password,sniff,sensitive,romon \
    start-date=oct/10/2020 start-time=04:30:00
/tool bandwidth-server
set enabled=no
/tool mac-server
set allowed-interface-list=LAN
/tool sniffer
set file-name=ron.pcap filter-interface=PPPoE-BSNL

Thanks in advance.

anav · Tue Oct 18, 2022 3:51 pm

You cannot aggregate speed on the hex, why dont you do that on your ASUS as well ??
Dont see the point to the ASUS really.

ronnielazarus · Tue Oct 18, 2022 5:51 pm

You cannot aggregate speed on the hex, why dont you do that on your ASUS as well ??
Dont see the point to the ASUS really.

Would you be able to elaborate please?

The ASUS is there to handle my fleet of > 50 clients and servers behind the LAN with a brick ton of data transfers amongst them everyday and along with the most important reason, adaptive QoS. I tried replacing the ASUS with the HEX and my network took a hit, especially my iSCSI targets, Media Servers, Remote VMs etc and even my internet cz now my clients are spoiled with adaotive QoS. All of this with Hardware Offloading enabled and still the CPU bottlenecking. It was only when I realized that the HEX's hardware and switching won't be able to handle my load is when I set it up specifically to act as a front-end LB for my existing ASUS. The Dual WAN implementation in ASUS is simple round robin which as we know, creates more problems then it solves in the day and age of HTTPs. TBH, I always predicted that the HEX won't be able to replace my main router and hence my main intention somewhat was always to make it work as a LB only like I mentioned.

So, when you say that the HEX can't do bandwidth aggregation, considering PCC is implemented and speedtest.net is multithreaded, what specifically did you mean?

ronnielazarus · Thu Oct 20, 2022 11:32 am

Hi, would anybody please be able to provide assiance? If I'm missed giving out any info please let me know so that I could collate and provide.

Many thanks,

ronnielazarus · Sat Oct 22, 2022 9:33 pm

Looks like I've figured it out myself over time. All it took was to create a new mangle rule with chain 'output' which is the chain that the routers internal traffic uses, and then finally use the dynamically created firewall list as the DST address with a new routing mark choosing a specific gateway:

/ip cloud
set ddns-enabled=yes

/ip firewall address-list
add address=cloud.mikrotik.com list=mikrotik-cloud
add address=cloud2.mikrotik.com list=mikrotik-cloud

/routing table
add disabled=no fib name=only_via_ISP_X

/ip firewall mangle
add action=mark-routing chain=output dst-address-list=mikrotik-cloud log=yes new-routing-mark=only_via_ISP_X passthrough=no

/ip route
add comment="For Cloud Update" disabled=no distance=1 dst-address=0.0.0.0/0 gateway=ISP_XGW pref-src="" routing-table=only_via_ISP_X scope=30 suppress-hw-offload=no target-scope=10

Seems to be working correctly to choose any interface for any domain now.

Unable to Force IP Cloud Updates via a specific Interface [SOLVED]

Unable to Force IP Cloud Updates via a specific Interface

Re: Unable to Force IP Cloud Updates via a specific Interface

Re: Unable to Force IP Cloud Updates via a specific Interface

Re: Unable to Force IP Cloud Updates via a specific Interface

Re: Unable to Force IP Cloud Updates via a specific Interface [SOLVED]

Who is online