{Urgent] Please look at my setting: experiencing unstable data transfer and laging

RouterOS General
1

Here is my setup in my CCR1009-7G-1C-1S+. The problem is that users behind Lan bridge experiencing unstable data download and upload. Sometimes the users test bandwidth and get 15Mbs and sometime get 3Mbs. Please note that

  1. at the time that users report the problem total bandwidth through WAN1-3 in total still have very large room for much more bandwidth.
  2. I have tried to limit user speed, but user have reported a much worst speed and lag.
    Port1-3: WAN1-3 (WAN1 = 200Mbs, WAN2 = 200Mbs, WAN3 = 50Mbs)
    Port4,5 (bridge1): Lan (through hotspot authen)
    Port6,7 (bridge2): Lan (no authen needed)

Would you please take a look at my setting below to see if I miss anything.

#RouterOS 6.39.2
#
/interface bridge
add arp=reply-only comment="bridge1: ether 4-5 as HOTSPOT" name=bridge1
add arp=reply-only comment="bridge2: ether 6-7 as LAN" name=bridge2

/interface ethernet
set [ find default-name=ether1 ] name=ether1-WAN
set [ find default-name=ether2 ] name=ether2-WAN
set [ find default-name=ether3 ] name=ether3-WAN
set [ find default-name=ether4 ] name=ether4-HOTSPOT
set [ find default-name=ether5 ] name=ether5-HOTSPOT
set [ find default-name=ether6 ] name=ether6-LAN
set [ find default-name=ether7 ] name=ether7-LAN

/ip firewall layer7-protocol
add name=layer7-bittorrent-exp regexp="^(\\x13bittorrent protocol|azver\\x01\$\
    |get /scrape\\\?info_hash=get /announce\\\?info_hash=|get /client/bitcomet\
    /|GET /data\\\?fid=)|d1:ad2:id20:|\\x08'7P\\)[RP]"

/ip hotspot profile
add dns-name=wifi.com hotspot-address=10.10.10.1 login-by=\
    http-chap,mac-cookie name=HOTSPOT use-radius=yes

/ip hotspot
add disabled=no idle-timeout=4h interface=bridge1 name=hotspot1 profile=\
    HOTSPOT

/ip hotspot user profile
add !idle-timeout keepalive-timeout=1d name=unlimited_user shared-users=\
    unlimited transparent-proxy=yes
add !idle-timeout keepalive-timeout=1d name=shared1user shared-users=1 transparent-proxy=yes

/ip pool
add name=pool_HOTSPOT ranges=10.10.10.2-10.10.10.254
add name=pool_LAN ranges=10.10.20.2-10.10.20.254

/ip dhcp-server
add add-arp=yes address-pool=pool_HOTSPOT authoritative=after-2sec-delay \
    disabled=no interface=bridge1 lease-time=1d name=HOTSPOT_dhcp
add add-arp=yes address-pool=pool_LAN authoritative=after-2sec-delay disabled=no \
    interface=bridge2 lease-time=4h name=LAN_dhcp

/tool user-manager customer
set admin access=\
    own-routers,own-users,own-profiles,own-limits,config-payment-gw password=\
    XXX time-zone=+07:00

/tool user-manager profile
add name=30day_shared1 name-for-users="" override-shared-users=1 owner=\
    admin starts-at=logon validity=4w2d

/tool user-manager profile limitation
add address-list="" download-limit=0B group-name=shared1user ip-pool="" name=\
    shared1user owner=admin transfer-limit=0B upload-limit=0B uptime-limit=0s

/interface bridge port
add bridge=bridge1 interface=ether4-HOTSPOT
add bridge=bridge1 interface=ether5-HOTSPOT
add bridge=bridge2 interface=ether6-LAN
add bridge=bridge2 interface=ether7-LAN

/ip address
add address=192.168.10.2/24 comment="WAN: ether1" interface=ether1-WAN \
    network=192.168.10.0
add address=192.168.20.2/24 comment="WAN: ether2" interface=ether2-WAN \
    network=192.168.20.0
add address=192.168.30.2/24 comment="WAN: ether3" interface=ether3-WAN \
    network=192.168.30.0
add address=10.10.10.1/24 comment="HOTSPOT: bridge ether4-5" interface=\
    bridge1 network=10.10.10.0
add address=10.10.20.1/24 comment="LAN: bridge ether 6-7" interface=bridge2 \
    network=10.10.20.0
add address=10.0.10.1/24 comment="Access point IP to HOTSPOT" interface=\
    bridge1 network=10.0.10.0
add address=10.0.20.1/24 comment="Access point IP to LAN" interface=bridge2 \
    network=10.0.20.0

/ip dhcp-server network
add address=10.10.10.0/24 comment="HOTSPOT pool" dns-server=10.10.10.1 \
    gateway=10.10.10.1
add address=10.10.20.0/24 comment="LAN pool" dns-server=10.10.20.1 gateway=\
    10.10.20.1

/ip dns
set allow-remote-requests=yes servers=8.8.8.8,8.8.4.4

/ip firewall address-list
add address=192.168.20.0/24 comment="WAN1: 192.168.20.0/24" list=WAN2
add address=192.168.30.0/24 comment="WAN1: 192.168.30.0/24" list=WAN3
add address=192.168.10.0/24 comment="WAN1: 192.168.10.0/24" list=WAN1
add address=10.10.10.0/24 comment="HOTSPOT: 10.10.10.0/24" list=LAN_HOTSPOT
add address=10.10.20.0/24 comment="LAN: 10.10.20.0/24" list=LAN_HOTSPOT

/ip firewall filter
add action=passthrough chain=unused-hs-chain comment=\
    "place hotspot rules here" disabled=yes
add action=add-src-to-address-list address-list=bit-list \
    address-list-timeout=1m chain=forward comment="Block Bittorrent" \
    dst-address-list=!bit-unblock-list layer7-protocol=layer7-bittorrent-exp \
    src-address-list=!bit-unblock-list
add action=add-src-to-address-list address-list=bit-list \
    address-list-timeout=1m chain=forward dst-address-list=!bit-unblock-list \
    layer7-protocol=layer7-bittorrent-exp src-address-list=!bit-unblock-list \
    src-address-type=local
add action=drop chain=forward dst-port=!80,443 protocol=tcp src-address-list=\
    bit-list
add action=drop chain=forward protocol=udp src-address-list=bit-list

/ip firewall mangle
add action=accept chain=prerouting dst-address-list=WAN1
add action=accept chain=prerouting dst-address-list=WAN2
add action=accept chain=prerouting dst-address-list=WAN3
add action=accept chain=prerouting disabled=yes dst-address-list=LAN_HOTSPOT
add action=mark-connection chain=input comment="INPUT: mark connection" \
    connection-mark=no-mark in-interface=ether1-WAN new-connection-mark=\
    WAN1_conn passthrough=yes
add action=mark-connection chain=input connection-mark=no-mark in-interface=\
    ether2-WAN new-connection-mark=WAN2_conn passthrough=yes
add action=mark-connection chain=input connection-mark=no-mark in-interface=\
    ether3-WAN new-connection-mark=WAN3_conn passthrough=yes
add action=mark-routing chain=output comment="OUTPUT: mark connection" \
    connection-mark=WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN2_conn \
    new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=output connection-mark=WAN3_conn \
    new-routing-mark=to_WAN3 passthrough=yes

add action=mark-connection chain=prerouting comment=\
    "PPC HOTSPOT: WAN1 (200Mbs) + WAN2 (200Mbs) + WAN3 (50Mbs)" \
    connection-mark=no-mark dst-address-type=!local hotspot=auth \
    in-interface=bridge1 new-connection-mark=WAN1_conn passthrough=yes \
    per-connection-classifier=both-addresses:9/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/2
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN1_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/4
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/5
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/6
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN2_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/7
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local hotspot=auth in-interface=bridge1 \
    new-connection-mark=WAN3_conn passthrough=yes per-connection-classifier=\
    both-addresses:9/8

add action=mark-connection chain=prerouting comment=\
    "PPC LAN: WAN1 (200Mbs) + WAN2 (200Mbs) + WAN3 (50Mbs)" connection-mark=\
    no-mark dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN1_conn passthrough=yes per-connection-classifier=both-addresses:9/0
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN1_conn passthrough=yes per-connection-classifier=both-addresses:9/1
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN1_conn passthrough=yes per-connection-classifier=both-addresses:9/2
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN1_conn passthrough=yes per-connection-classifier=both-addresses:9/3
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN2_conn passthrough=yes per-connection-classifier=both-addresses:9/4
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN2_conn passthrough=yes per-connection-classifier=both-addresses:9/5
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN2_conn passthrough=yes per-connection-classifier=both-addresses:9/6
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN2_conn passthrough=yes per-connection-classifier=both-addresses:9/7
add action=mark-connection chain=prerouting connection-mark=no-mark \
    dst-address-type=!local in-interface=bridge2 new-connection-mark=\
    WAN3_conn passthrough=yes per-connection-classifier=both-addresses:9/8

add action=mark-routing chain=prerouting comment=\
    "Mark packets from each connection to to proper routes " connection-mark=\
    WAN1_conn new-routing-mark=to_WAN1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN2_conn \
    new-routing-mark=to_WAN2 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=WAN3_conn \
    new-routing-mark=to_WAN3 passthrough=yes

/ip firewall nat
add action=passthrough chain=unused-hs-chain comment=\
    "place hotspot rules here" disabled=yes
add action=masquerade chain=srcnat comment=\
    "NAT to make local IP to be able to go to extranet" out-interface=\
    ether1-WAN
add action=masquerade chain=srcnat out-interface=ether2-WAN
add action=masquerade chain=srcnat out-interface=ether3-WAN
add action=masquerade chain=srcnat comment="masquerade hotspot network" \
    src-address=10.10.10.0/24

/ip hotspot walled-garden
add dst-host=:^www.apple.com
add dst-host=:^www.airport.us
add dst-host=:^www.itools.info
add dst-host=:^www.appleiphonecell.com
add dst-host=:^captive.apple.com
add dst-host=:^www.thinkdifferent.us
add dst-host=:^www.ibook.info

/ip route
add check-gateway=ping distance=1 gateway=192.168.10.1 routing-mark=to_WAN1
add check-gateway=ping distance=1 gateway=192.168.20.1 routing-mark=to_WAN2
add check-gateway=ping distance=1 gateway=192.168.30.1 routing-mark=to_WAN3
add check-gateway=ping distance=1 gateway=192.168.10.1
add check-gateway=ping distance=2 gateway=192.168.20.1
add check-gateway=ping distance=3 gateway=192.168.30.1

/radius
add address=127.0.0.1 secret=1234 service=hotspot

/radius incoming
set accept=yes
/tool user-manager router
add coa-port=1700 customer=admin disabled=no ip-address=127.0.0.1 log=\
    auth-fail name=mikrotik shared-secret=1234 use-coa=yes
2

Hi,
I’m just a newbie myself, maybe of less knowledge than you.
But I also had similar problems and my research showed that it was because of the load balancing (3 unequal lines). I followed zaib’s guide, and now users have reasonable surfing experience.
Also, try to use your provider’s modems in bridge mode.
load-bal2.png

3

Hi Jimmy,

Thx for your reply. Believe me that i am just a newbie myself, and following others’ guide as well.

Could you please provide link to Zaib’s guide you said help to solve your problem? The surfing experience and complaint have driven me crazy.

Thx
Nick

4

https://aacable.wordpress.com/2011/06/04/mikrotik-4-wan-load-balance-pcc-complete-script-by-zaib/

5

Yeap, that’s the link.
Believe me, I know the feeling!

6

Thx Jimmy and Amt,

My curiousity is that in Zaib’s post. He mentioned that using src-address would solve the problem of broken link. However, I cannot see in his setting mentioning about scr-address or that has already been covered in using both address and port classifier.

I’m very new. Please help if my question is noob.

7

I don’t think your problem is caused by the PCC classifier, so don’t get confused by people pointing to that.
There are advantages and disadvantages to each of the classifiers (src-address, both-addresses, both-addresses-and-ports)
but choosing another one should not cause the issue you are facing. They are mainly related to problems you may have
with badly designed websites whose session identification my get confused by the changing source address.

I have no idea what the problem is right now. I see from your config that you basically understood the method for distributing
load across unequal WAN links (find least common denominator and distribute load according to speed relative to that).

Check if the CPU is not overloaded, maybe remove pointless things like L7 matches.

8

Hi Peich,

Thx for your reply. I did tried to change classifier from both address to src-address to see different as recommended from Jimmy and Zaib’s.
Will wait and see how user comments on this.

Regarding your point. CPU load is super low. It actually maintain low level at 0% and 1%. L7 matches I put it there intended to block p2p bittorrent. Please advice if I should remove it.

9

Well, you should understand that this makes the allocation of user to WAN link fixed. So 8/9 of the users will always get one of
the two 200 Mbit/s links and 1/9 of the users are always on the 50 Mbps link. Depending on what rate you give to the users, I
could understand that this makes some of then unhappy. With both-addresses the WAN link depends on the website someone is
visiting, so they will all get some 200 Mbit/s and some 50 Mbit/s experience.
However, when people complain they get only 13 Mbit/s this is not what is the cause of that.

Regarding your point. CPU load is super low. It actually maintain low level at 0% and 1%. L7 matches I put it there intended to block p2p bittorrent. Please advice if I should remove it.

Ok. On a CCR it is usual to see that figure. I am still not completely convinced that it is correct. At least it is the average over all CPUs so it is not relevant for the load on a single CPU.
Try to reproduce the issue and check if it occurs on a directly wired ethernet link to your router (can be via a switch), not only on WiFi.
(the real issue could be in the WiFi not in this router)

10

Ok. On a CCR it is usual to see that figure. I am still not completely convinced that it is correct. At least it is the average over all CPUs so it is not relevant for the load on a single CPU.
Try to reproduce the issue and check if it occurs on a directly wired ethernet link to your router (can be via a switch), not only on WiFi.
(the real issue could be in the WiFi not in this router)

Hi Peich,

I’ve check CPU on each core. The max. load on each core is 10%, which at this level I assume I would not be the cause of the problem.
You mentioned that this could occur eternally from the router. I thought it would be an issue as well. But the user complaint that after changing to new router they are unhappy.
Previously, the user has been surfing through mikrotik rb850x2 with previous networking guy. However, after upgrading to CCR and I am in charged of the configuration the user report an unhappy surfing situation. So I assume the problem would be one of this 1) configuartion 2) hardware malfunction (CCR) 3. Firmware