New Load Balanced Setup - Poor Performance

Using ROS 7.1.1
WAN1: 400mb/sec, 55.x.x.x
WAN2: 1000mb/sec 192.x.x.x

Simple load balance setup using PCC ‘both addresses’ across both WAN connections. No matter what I do (even disabling each WAN interface one at a time) I can’t get the speed above 450/mbs. If I test each link independently by removing them from the Mikrotik, I get the expected results. Any ideas?

/ip firewall mangle
add action=accept chain=prerouting comment=Accept dst-address=55.188.40.0/24
add action=accept chain=prerouting dst-address=192.168.1.0/24
add action=accept chain=prerouting dst-address=10.10.10.0/24
add action=mark-connection chain=input comment=Input in-interface=ether1-ISP1 \
    new-connection-mark=ISP1 passthrough=yes
add action=mark-connection chain=input in-interface=ether2-ISP2 \
    new-connection-mark=ISP2 passthrough=yes
add action=mark-connection chain=prerouting comment="Mark Connection" \
    in-interface=ether1-ISP1 new-connection-mark=ISP1 passthrough=yes
add action=mark-connection chain=prerouting in-interface=ether2-ISP2 \
    new-connection-mark=ISP2 passthrough=yes
add action=mark-connection chain=prerouting comment=PCC dst-address-type=local \
    in-interface=LAN new-connection-mark=ISP1 passthrough=yes \
    per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=local \
    in-interface=LAN new-connection-mark=ISP2 passthrough=yes \
    per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=output comment=Output connection-mark=ISP1 \
    new-routing-mark=to-ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2 new-routing-mark=\
    to-ISP2 passthrough=yes
add action=mark-routing chain=prerouting comment="Mark Route" connection-mark=\
    ISP1 in-interface=LAN new-routing-mark=to-ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2 in-interface=LAN \
    new-routing-mark=to-ISP2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1-ISP1
add action=masquerade chain=srcnat out-interface=ether2-ISP2
[admin@MikroTik] /ip/firewall>



/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10

Any help is greatly appreciated.

  1. what is the routerboard model?
  2. did you try the same before with RouterOS 6.x (if available for that device)?

It’s an RBM33G (not using modem component, only ethernet). I have not tried with ROS6. Is that recommended?

Not really necessary to try with ROS 6, look here - the item “routing with 25 filter rules” is closest to your case, and unless you use only large packets for testing, it’s about 500 Mbit/s total throughput. And ROS 7 has no routing cache so I’m afraid the figures will be even worse.

According to the Test Results your speeds are totally fine…
Check the CPU when you reach 400-500 Mbps and you will see that the CPU will be 80-90% or more…

It’s in the >90% range under load and hitting 400-500. Any recommendation on a better router to use to get the speeds up. I guess I’m expecting a lot out of a ~$30 device. Any recommendation on another Mikrotik router that can come close to 800mb/sec with 25 bridged rules. Not looking for anything crazy expensive.

Well, hAP ac2 is quite a good value for price, but not that much better (1.8 times or so higher throughput, so about your 800 Mb/s, which doesn’t match your 1.4 Gb/s originally mentioned).

Then there is nothing between hAP ac2 (hAP ac3 is basically a facelift of the same) and a $200 RB4011 and a $220 RB5009.

Well take a look in the products page https://mikrotik.com/products/group/ethernet-routers
Personally i would suggest an RB4011 that on ROS 7 can do VLANs in hardware level as well…

Sindy and Zacharias,
Thanks for your feedback, will look into both options. I may downgrade the 1gb/sec to 400mb/sec then I can use the hAP mentioned above.

Last question, based on my rules, do you have input into how I can set a route or interface down if next hop to an external IP address is not responding (8.8.8.8 or 8.8.4.4) is down? This currently works if gateway is down only, but concerned about outside.

http://forum.mikrotik.com/t/advanced-routing-failover-without-scripting/136599/1

I tried the above linked guide for detecting external ISP failure but the scripts do no work on ROS7, specifically this command ‘routing-mark=ISP1’ under /ip route. I had to use ‘routing-table=to-ISP1’ which did not work to stop traffic when I simulated an ISP failure. Any other thoughts?

Yes, to post an export of what you have actually configured and then do some debugging. That concept works for many people, no reason why it shoud not work for you. And it is true that it’s now routing-table, not routing-mark, in ROS 7.

For ROS v7 @Chupaka answers here Advanced Routing Failover without Scripting
There is actually a change as to how you should use the target-scope attribute in comparison with ROS v6.

He does, but whilst he has updated the scope values in the original post, he has not changed routing-mark to routing-table in the routes, nor has he mentioned that in ROS 7, the routing table names have to be defined before referring to them (which @gutowscr471 seems to have managed on his own). I am not sure whether the routing-markrouting-table change has been there ever since ROS 7.0.x or whether it has been implemented later, so no judging on why @Chupaka didn’t mention that.

@sindy you 're right on that…
Indeed on ROS 7 you can’t refer to a Routing Table unless first it is created… ( found as Routing mark under /ip route and as Table under /ip route rule in ROS v6 )

Here is my IP route config which is not working for some reason. LB works perfectly fine across both WAN ports. Just want to get this last fix in to have stability if downstream ISP is down. Any help/input is greatly appreciated.

Checks: 74.6.143.25=yahoo.com; 108.177.122.100=google.com

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=108.177.122.100/32 gateway=55.188.40.1 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=74.6.143.25/32 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping distance=1 gateway=108.177.122.100 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=2 gateway=74.6.143.25 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=1 gateway=74.6.143.25 routing-table=to-ISP2 \
    target-scope=11
add check-gateway=ping distance=2 gateway=108.177.122.100 routing-table=to-ISP2 \
    target-scope=11
[admin@MikroTik] /ip/route>



/routing table
add disabled=no fib name=to-ISP1
add disabled=no fib name=to-ISP2

In order that the network transparency monitoring worked, you cannot keep the default routes via the direct gateways in routing table to-ISP1 - only the recursive ones can be there. So disable them and see what happens.

If it still doesn’t work, post the output of both ip/route/export and ip/route/print detail - the latter one first when the primary WAN is “working” and then when it is “broken” due to your simulation (a drop rule in output chain I guess).

@gutowscr471 you are using the default Gateways, the direct ones as @sindy said…
Check again here http://forum.mikrotik.com/t/advanced-routing-failover-without-scripting/136599/1
I would suggest you the Multiple host checking per Uplink, as you can see there a virtual host is used as well…

Can you notate which ones I should disable. I tried a few and it failed miserably to the point where load balancing didn’t even work and 1/2 of my connections went down. This is ok as it’s in a lab/home.

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=108.177.122.100/32 gateway=55.188.40.1 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=74.6.143.25/32 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping distance=1 gateway=108.177.122.100 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=2 gateway=74.6.143.25 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=1 gateway=74.6.143.25 routing-table=to-ISP2 \
    target-scope=11
add check-gateway=ping distance=2 gateway=108.177.122.100 routing-table=to-ISP2 \
    target-scope=11

These two:

distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=“” routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 pref-src=“” routing-table=to-ISP2 scope=30 suppress-hw-offload=no target-scope=10