New Load Balanced Setup - Poor Performance

gutowscr471 · Sun Jan 09, 2022 5:09 pm

Using ROS 7.1.1
WAN1: 400mb/sec, 55.x.x.x
WAN2: 1000mb/sec 192.x.x.x

Simple load balance setup using PCC 'both addresses' across both WAN connections. No matter what I do (even disabling each WAN interface one at a time) I can't get the speed above 450/mbs. If I test each link independently by removing them from the Mikrotik, I get the expected results. Any ideas?

/ip firewall mangle
add action=accept chain=prerouting comment=Accept dst-address=55.188.40.0/24
add action=accept chain=prerouting dst-address=192.168.1.0/24
add action=accept chain=prerouting dst-address=10.10.10.0/24
add action=mark-connection chain=input comment=Input in-interface=ether1-ISP1 \
    new-connection-mark=ISP1 passthrough=yes
add action=mark-connection chain=input in-interface=ether2-ISP2 \
    new-connection-mark=ISP2 passthrough=yes
add action=mark-connection chain=prerouting comment="Mark Connection" \
    in-interface=ether1-ISP1 new-connection-mark=ISP1 passthrough=yes
add action=mark-connection chain=prerouting in-interface=ether2-ISP2 \
    new-connection-mark=ISP2 passthrough=yes
add action=mark-connection chain=prerouting comment=PCC dst-address-type=local \
    in-interface=LAN new-connection-mark=ISP1 passthrough=yes \
    per-connection-classifier=both-addresses:2/0
add action=mark-connection chain=prerouting dst-address-type=local \
    in-interface=LAN new-connection-mark=ISP2 passthrough=yes \
    per-connection-classifier=both-addresses:2/1
add action=mark-routing chain=output comment=Output connection-mark=ISP1 \
    new-routing-mark=to-ISP1 passthrough=yes
add action=mark-routing chain=output connection-mark=ISP2 new-routing-mark=\
    to-ISP2 passthrough=yes
add action=mark-routing chain=prerouting comment="Mark Route" connection-mark=\
    ISP1 in-interface=LAN new-routing-mark=to-ISP1 passthrough=yes
add action=mark-routing chain=prerouting connection-mark=ISP2 in-interface=LAN \
    new-routing-mark=to-ISP2 passthrough=yes
/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1-ISP1
add action=masquerade chain=srcnat out-interface=ether2-ISP2
[admin@MikroTik] /ip/firewall>

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10

Any help is greatly appreciated.

sindy · Sun Jan 09, 2022 5:20 pm

1. what is the routerboard model?
2. did you try the same before with RouterOS 6.x (if available for that device)?

gutowscr471 · Sun Jan 09, 2022 6:14 pm

1. what is the routerboard model?
2. did you try the same before with RouterOS 6.x (if available for that device)?

It's an RBM33G (not using modem component, only ethernet). I have not tried with ROS6. Is that recommended?

sindy · Sun Jan 09, 2022 6:52 pm

It's an RBM33G (not using modem component, only ethernet). I have not tried with ROS6. Is that recommended?

Not really necessary to try with ROS 6, look here - the item "routing with 25 filter rules" is closest to your case, and unless you use only large packets for testing, it's about 500 Mbit/s total throughput. And ROS 7 has no routing cache so I'm afraid the figures will be even worse.

Zacharias · Sun Jan 09, 2022 6:57 pm

According to the Test Results your speeds are totally fine...
Check the CPU when you reach 400-500 Mbps and you will see that the CPU will be 80-90% or more...

gutowscr471 · Sun Jan 09, 2022 8:24 pm

According to the Test Results your speeds are totally fine...
Check the CPU when you reach 400-500 Mbps and you will see that the CPU will be 80-90% or more...

It's in the >90% range under load and hitting 400-500. Any recommendation on a better router to use to get the speeds up. I guess I'm expecting a lot out of a ~$30 device. Any recommendation on another Mikrotik router that can come close to 800mb/sec with 25 bridged rules. Not looking for anything crazy expensive.

sindy · Sun Jan 09, 2022 8:37 pm

Well, hAP ac2 is quite a good value for price, but not that much better (1.8 times or so higher throughput, so about your 800 Mb/s, which doesn't match your 1.4 Gb/s originally mentioned).

Then there is nothing between hAP ac2 (hAP ac3 is basically a facelift of the same) and a $200 RB4011 and a $220 RB5009.

Zacharias · Sun Jan 09, 2022 8:48 pm

According to the Test Results your speeds are totally fine...
Check the CPU when you reach 400-500 Mbps and you will see that the CPU will be 80-90% or more...
It's in the >90% range under load and hitting 400-500. Any recommendation on a better router to use to get the speeds up. I guess I'm expecting a lot out of a ~$30 device. Any recommendation on another Mikrotik router that can come close to 800mb/sec with 25 bridged rules. Not looking for anything crazy expensive.

Well take a look in the products page https://mikrotik.com/products/group/ethernet-routers
Personally i would suggest an RB4011 that on ROS 7 can do VLANs in hardware level as well...

gutowscr471 · Sun Jan 09, 2022 10:30 pm

It's in the >90% range under load and hitting 400-500. Any recommendation on a better router to use to get the speeds up. I guess I'm expecting a lot out of a ~$30 device. Any recommendation on another Mikrotik router that can come close to 800mb/sec with 25 bridged rules. Not looking for anything crazy expensive.
Well take a look in the products page https://mikrotik.com/products/group/ethernet-routers
Personally i would suggest an RB4011 that on ROS 7 can do VLANs in hardware level as well...

Sindy and Zacharias,
Thanks for your feedback, will look into both options. I may downgrade the 1gb/sec to 400mb/sec then I can use the hAP mentioned above.

Last question, based on my rules, do you have input into how I can set a route or interface down if next hop to an external IP address is not responding (8.8.8.8 or 8.8.4.4) is down? This currently works if gateway is down only, but concerned about outside.

sindy · Sun Jan 09, 2022 10:50 pm

viewtopic.php?t=157048

gutowscr471 · Wed Jan 12, 2022 2:57 am

viewtopic.php?t=157048

I tried the above linked guide for detecting external ISP failure but the scripts do no work on ROS7, specifically this command 'routing-mark=ISP1' under /ip route. I had to use 'routing-table=to-ISP1' which did not work to stop traffic when I simulated an ISP failure. Any other thoughts?

sindy · Wed Jan 12, 2022 9:35 am

Any other thoughts?

Yes, to post an export of what you have actually configured and then do some debugging. That concept works for many people, no reason why it shoud not work for you. And it is true that it's now routing-table, not routing-mark, in ROS 7.

Zacharias · Wed Jan 12, 2022 12:42 pm

viewtopic.php?t=157048
I tried the above linked guide for detecting external ISP failure but the scripts do no work on ROS7, specifically this command 'routing-mark=ISP1' under /ip route. I had to use 'routing-table=to-ISP1' which did not work to stop traffic when I simulated an ISP failure. Any other thoughts?

For ROS v7 @Chupaka answers here viewtopic.php?t=157048#p891976
There is actually a change as to how you should use the target-scope attribute in comparison with ROS v6.

sindy · Wed Jan 12, 2022 12:49 pm

For ROS v7 @Chupaka answers here viewtopic.php?t=157048#p891976

He does, but whilst he has updated the scope values in the original post, he has not changed routing-mark to routing-table in the routes, nor has he mentioned that in ROS 7, the routing table names have to be defined before referring to them (which @gutowscr471 seems to have managed on his own). I am not sure whether the routing-mark -> routing-table change has been there ever since ROS 7.0.x or whether it has been implemented later, so no judging on why @Chupaka didn't mention that.

Zacharias · Wed Jan 12, 2022 1:04 pm

@sindy you 're right on that...
Indeed on ROS 7 you can't refer to a Routing Table unless first it is created... ( found as Routing mark under /ip route and as Table under /ip route rule in ROS v6 )

gutowscr471 · Wed Jan 12, 2022 4:14 pm

Here is my IP route config which is not working for some reason. LB works perfectly fine across both WAN ports. Just want to get this last fix in to have stability if downstream ISP is down. Any help/input is greatly appreciated.

Checks: 74.6.143.25=yahoo.com; 108.177.122.100=google.com

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=108.177.122.100/32 gateway=55.188.40.1 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=74.6.143.25/32 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping distance=1 gateway=108.177.122.100 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=2 gateway=74.6.143.25 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=1 gateway=74.6.143.25 routing-table=to-ISP2 \
    target-scope=11
add check-gateway=ping distance=2 gateway=108.177.122.100 routing-table=to-ISP2 \
    target-scope=11
[admin@MikroTik] /ip/route>

/routing table
add disabled=no fib name=to-ISP1
add disabled=no fib name=to-ISP2

sindy · Wed Jan 12, 2022 4:30 pm

In order that the network transparency monitoring worked, you cannot keep the default routes via the direct gateways in routing table to-ISP1 - only the recursive ones can be there. So disable them and see what happens.

If it still doesn't work, post the output of both ip/route/export and ip/route/print detail - the latter one first when the primary WAN is "working" and then when it is "broken" due to your simulation (a drop rule in output chain I guess).

Zacharias · Wed Jan 12, 2022 5:01 pm

@gutowscr471 you are using the default Gateways, the direct ones as @sindy said...
Check again here viewtopic.php?t=157048#p773229
I would suggest you the Multiple host checking per Uplink, as you can see there a virtual host is used as well...

gutowscr471 · Wed Jan 12, 2022 5:50 pm

In order that the network transparency monitoring worked, you cannot keep the default routes via the direct gateways in routing table to-ISP1 - only the recursive ones can be there. So disable them and see what happens.

If it still doesn't work, post the output of both ip/route/export and ip/route/print detail - the latter one first when the primary WAN is "working" and then when it is "broken" due to your simulation (a drop rule in output chain I guess).

Can you notate which ones I should disable. I tried a few and it failed miserably to the point where load balancing didn't even work and 1/2 of my connections went down. This is ok as it's in a lab/home.

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=108.177.122.100/32 gateway=55.188.40.1 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=74.6.143.25/32 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping distance=1 gateway=108.177.122.100 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=2 gateway=74.6.143.25 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=1 gateway=74.6.143.25 routing-table=to-ISP2 \
    target-scope=11
add check-gateway=ping distance=2 gateway=108.177.122.100 routing-table=to-ISP2 \
    target-scope=11

sindy · Wed Jan 12, 2022 5:53 pm

These two:

distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src="" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no target-scope=10

Zacharias · Wed Jan 12, 2022 7:39 pm

That should work

/ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
	
add disabled=no distance=1 dst-address=108.177.122.100/32 gateway=55.188.40.1 \
    pref-src="" routing-table=main scope=12 suppress-hw-offload=no \
    target-scope=11
add disabled=no distance=1 dst-address=74.6.143.25/32 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=12 suppress-hw-offload=no \
    target-scope=11
	
add check-gateway=ping distance=1 gateway=108.177.122.100 routing-table=to-ISP1 \
    target-scope=12
add check-gateway=ping distance=2 gateway=74.6.143.25 routing-table=to-ISP1 \
    target-scope=12
add check-gateway=ping distance=1 gateway=74.6.143.25 routing-table=to-ISP2 \
    target-scope=12
add check-gateway=ping distance=2 gateway=108.177.122.100 routing-table=to-ISP2 \
    target-scope=12

gutowscr471 · Thu Jan 13, 2022 1:24 am

@sindy

After disabling those routes, still having issues with existing/new connections. I can make WAN2 fail in simulation, still working on WAN1 simulation. Here is some data:

Route Export:

ip route
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=main scope=30 suppress-hw-offload=no target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=55.188.40.1 pref-src=\
    "" routing-table=to-ISP1 scope=30 suppress-hw-offload=no target-scope=10
add disabled=yes distance=1 dst-address=0.0.0.0/0 gateway=192.168.1.254 \
    pref-src="" routing-table=to-ISP2 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=108.177.122.100/32 gateway=55.188.40.1 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=74.6.143.25/32 gateway=192.168.1.254 \
    pref-src="" routing-table=main scope=11 suppress-hw-offload=no \
    target-scope=10
add check-gateway=ping disabled=no distance=1 dst-address=0.0.0.0/0 gateway=\
    108.177.122.100 pref-src="" routing-table=to-ISP1 scope=30 \
    suppress-hw-offload=no target-scope=11
add check-gateway=ping distance=2 gateway=74.6.143.25 routing-table=to-ISP1 \
    target-scope=11
add check-gateway=ping distance=1 gateway=74.6.143.25 routing-table=to-ISP2 \
    target-scope=11
add check-gateway=ping distance=2 gateway=108.177.122.100 routing-table=to-ISP2 \
    target-scope=11

Here is route Print Detail BOTH WANs WORKING:

 /ip route print detail
Flags: D - dynamic; X - disabled, I - inactive, A - active; 
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - mode>
H - hw-offloaded; + - ecmp 
 0  Xs   dst-address=0.0.0.0/0 routing-table=to-ISP1 pref-src="" 
         gateway=55.188.40.1 distance=1 scope=30 target-scope=10 
H - hw-offloaded; + - ecmp 
 0  Xs   dst-address=0.0.0.0/0 routing-table=to-ISP1 pref-src="" 
         gateway=55.188.40.1 distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 1  Xs   dst-address=0.0.0.0/0 routing-table=to-ISP2 pref-src="" 
         gateway=192.168.1.254 distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 2  As + dst-address=0.0.0.0/0 routing-table=main pref-src="" 
         gateway=192.168.1.254 immediate-gw=192.168.1.254%ether2-ISP2 
         distance=1 scope=30 target-scope=10 suppress-hw-offload=no 

 3  As + dst-address=0.0.0.0/0 routing-table=main pref-src="" gateway=55.188.40.>
         immediate-gw=55.188.40.1%ether1-ISP1 distance=1 scope=30 
         target-scope=10 suppress-hw-offload=no 

   DAc   dst-address=10.10.10.0/24 routing-table=main gateway=LAN 
         immediate-gw=LAN distance=0 scope=10 suppress-hw-offload=no 
         local-address=10.10.10.1%LAN 

   DAc   dst-address=65.188.80.0/20 routing-table=main gateway=ether1-ISP1 
         immediate-gw=ether1-ISP1 distance=0 scope=10 suppress-hw-offload=no 
         local-address=55.188.40.126%ether1-ISP1 

 4  As   dst-address=74.6.143.25/32 routing-table=main pref-src="" 
         gateway=192.168.1.254 immediate-gw=192.168.1.254%ether2-ISP2 
         distance=1 scope=11 target-scope=10 suppress-hw-offload=no 

 5  As   dst-address=108.177.122.100/32 routing-table=main pref-src="" 
         gateway=55.188.40.1 immediate-gw=55.188.40.1%ether1-ISP1 distance=1 
         scope=11 target-scope=10 suppress-hw-offload=no 

   DAc   dst-address=192.168.1.0/24 routing-table=main gateway=ether2-ISP2 
         immediate-gw=ether2-ISP2 distance=0 scope=10 suppress-hw-offload=no 
         local-address=192.168.1.105%ether2-ISP2 

 6  As   dst-address=0.0.0.0/0 routing-table=to-ISP1 pref-src="" 
         gateway=108.177.122.100 immediate-gw=55.188.40.1%ether1-ISP1 
         check-gateway=ping distance=1 scope=30 target-scope=11 suppress-hw-offload=no 

 7   s   dst-address=0.0.0.0/0 routing-table=to-ISP1 pref-src="" 
         gateway=74.6.143.25 immediate-gw=192.168.1.254%ether2-ISP2 
         check-gateway=ping distance=2 scope=30 target-scope=11 
         suppress-hw-offload=no 

 8   s   dst-address=0.0.0.0/0 routing-table=to-ISP2 pref-src="" 
         gateway=108.177.122.100 immediate-gw=55.188.40.1%ether1-ISP1 
         check-gateway=ping distance=2 scope=30 target-scope=11 
         suppress-hw-offload=no 

 9  As   dst-address=0.0.0.0/0 routing-table=to-ISP2 pref-src="" 
         check-gateway=ping distance=1 scope=30 target-scope=11 
         suppress-hw-offload=no

WAN2 DOWN Detail:

WAN 2 DOWN
 /ip route print detail
Flags: D - dynamic; X - disabled, I - inactive, A - active; 
c - connect, s - static, r - rip, b - bgp, o - ospf, d - dhcp, v - vpn, m - mode>
H - hw-offloaded; + - ecmp 
 0  Xs   dst-address=0.0.0.0/0 routing-table=to-ISP1 pref-src="" 
         gateway=55.188.40.1 distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 1  Xs   dst-address=0.0.0.0/0 routing-table=to-ISP2 pref-src="" 
         gateway=192.168.1.254 distance=1 scope=30 target-scope=10 
         suppress-hw-offload=no 

 2  As + dst-address=0.0.0.0/0 routing-table=main pref-src="" 
         gateway=192.168.1.254 immediate-gw=192.168.1.254%ether2-ISP2 
         distance=1 scope=30 target-scope=10 suppress-hw-offload=no 

 3  As + dst-address=0.0.0.0/0 routing-table=main pref-src="" gateway=55.188.40.>
         immediate-gw=55.188.40.1%ether1-ISP1 distance=1 scope=30 
         target-scope=10 suppress-hw-offload=no 

   DAc   dst-address=10.10.10.0/24 routing-table=main gateway=LAN 
         immediate-gw=LAN distance=0 scope=10 suppress-hw-offload=no 
         local-address=10.10.10.1%LAN 

   DAc   dst-address=55.188.40.0/20 routing-table=main gateway=ether1-ISP1 
         immediate-gw=ether1-ISP1 distance=0 scope=10 suppress-hw-offload=no 
         local-address=55.188.40.126%ether1-ISP1 

 4  As   dst-address=74.6.143.25/32 routing-table=main pref-src="" 
         gateway=192.168.1.254 immediate-gw=192.168.1.254%ether2-ISP2 
         distance=1 scope=11 target-scope=10 suppress-hw-offload=no 

 5  As   dst-address=108.177.122.100/32 routing-table=main pref-src="" 
         gateway=55.188.40.1 immediate-gw=55.188.40.1%ether1-ISP1 distance=1 
         scope=11 target-scope=10 suppress-hw-offload=no 

   DAc   dst-address=192.168.1.0/24 routing-table=main gateway=ether2-ISP2 
         immediate-gw=ether2-ISP2 distance=0 scope=10 suppress-hw-offload=no 
         local-address=192.168.1.105%ether2-ISP2 

 6  As   dst-address=0.0.0.0/0 routing-table=to-ISP1 pref-src="" 
         gateway=108.177.122.100 immediate-gw=55.188.40.1%ether1-ISP1 
         check-gateway=ping distance=1 scope=30 target-scope=11 
         suppress-hw-offload=no 

 7  IsH  dst-address=0.0.0.0/0 routing-table=to-ISP1 pref-src="" 
         gateway=74.6.143.25 immediate-gw="" check-gateway=ping distance=2 
         scope=30 target-scope=11 suppress-hw-offload=no 

 8  As   dst-address=0.0.0.0/0 routing-table=to-ISP2 pref-src="" 
         gateway=108.177.122.100 immediate-gw=55.188.40.1%ether1-ISP1 
         check-gateway=ping distance=2 scope=30 target-scope=11 
         suppress-hw-offload=no 

 9  IsH  dst-address=0.0.0.0/0 routing-table=to-ISP2 pref-src="" 
         gateway=74.6.143.25 immediate-gw="" check-gateway=ping distance=1 
         scope=30 target-scope=11 suppress-hw-offload=no

sindy · Thu Jan 13, 2022 11:11 am

After disabling those routes, still having issues with existing/new connections.

What kind of issues in particular? Some connections do not get through although you assume they should, some connections do get through although you assume they should not, or some connections establish via a WAN that is simulated to be down?

I can make WAN2 fail in simulation, still working on WAN1 simulation. Here is some data:

Both print outputs show that everything works as expected:

when both WANs are simulated to be OK, routes 6 and 9 are active because their recursive gateways are reachable, and routes 7 and 8 are not used (not marked as Active) simply because they have higher distance values than 6 and 9 ones.
when WAN 2 is simulated to be down, routes 6 and 8 are active whereas 7 and 9 are marked as Inactive because their recursive gateway (74.6.143.25) is unreachable. So even though route 8 has higher distance than route 9, it becomes Active.

So what is the issue when WAN 2 is simulated to be down, i.e. what behaves different than you expect?

As for simulation of WAN 1 failure, it is enough to add the following firewall rule:

/ip/firewall/filter
add chain=output out-interface=ether1-ISP protocol=icmp dst-address=108.177.122.100 action=drop

This will prevent the check-gateway pings from reaching the recursive gateway, having the same effect as if it becomes unreachable due to some issue further away in the network.

gutowscr471 · Thu Jan 13, 2022 4:20 pm

So what is the issue when WAN 2 is simulated to be down, i.e. what behaves different than you expect?

Existing clients connected have no path to internet. They are unable to re-establish existing connections in most cases. Seems like the router is still trying to send traffic to the down WAN.

sindy · Thu Jan 13, 2022 5:25 pm

Existing clients connected have no path to internet. They are unable to re-establish existing connections in most cases. Seems like the router is still trying to send traffic to the down WAN.

Connections that are established at the moment WAN2 goes down cannot continue if there is NAT, because the connection tracking keeps sending the packets belonging to these connections out with the source address of the WAN2 interface even though they are routed via WAN1. So either the routers on the path to the destination drop them, or the server sends the response to this private address and thus they are routed to that private address within the context of the server's network, so they cannot reach your router neither via WAN 1 nor via WAN 2. The clients have to initiate new connections whose first packet will be sent via WAN 1 and NATed accordingly. E.g. in case of ping, you have to wait 10 seconds so that the tracked connection is forgotten, and then try again.

You can use /tool/sniffer and /ip/firewall/connection/print detail where src-address~"sour.ce.i.p:port" dst-address~"de.st.i.p:port" to visualise this.

gutowscr471 · Thu Jan 13, 2022 9:29 pm

Thanks @sindy. So from your perspective, my setup is working as expected based on the information I was able to provide. I'll I'm going to keep testing scenarios. Truly appreciate all the help this group has provided.

sindy · Thu Jan 13, 2022 9:44 pm

So from your perspective, my setup is working as expected based on the information I was able to provide.

If you can confirm that clients who use routing table to-ISP2 establish their connections via WAN 2 while the internet is reachable through WAN 2, and establish new connections via WAN 1 while it is not, then yes, it works as expected. Including the fact that existing connections via WAN 2 do not automatically continue via WAN 1 after internet becomes unreachable via WAN 2.

Zacharias · Thu Jan 13, 2022 11:47 pm

@sindy are the 10 seconds the TCP timeout you re referring to ?

sindy · Fri Jan 14, 2022 1:20 am

@sindy are the 10 seconds the TCP timeout you re referring to ?

10 seconds are ICMP default timeout. For TCP, there are different timeouts depending on the current state of the session (24 h if everything has been ACKed, just 5 min if I remember well if there are unACKed data, etc.). I was referring specifically to ICMP as it is both simple to test and the timeouts are short.

gutowscr471 · Fri Jan 14, 2022 4:34 pm

So the current solution is having problems with clients that have active sessions (which we know won't work) AND trying to establish new connections. If I use the drop script OR another method (disconnect WAN, but gateway remains IP remains up), traffic is still being sent to both WAN's, even though it looks down in the routing table.

I'm going to re-do this setup using ROSv6. Wonder if something is not 'baked' right in ROSv7 just yet. Will report my findings back soon.

Zacharias · Sun Jan 16, 2022 12:13 am

ok @sindy so you 're referring to the check-gateway ping timeout...
Yes indeed, for the unacked and retransmitted TCP packets the default timeout is 5 minutes...
But there are as well 10 second timeouts for some TCP wait timeouts etc. and i thought you were referring to them.

sindy · Sun Jan 16, 2022 1:03 am

so you 're referring to the check-gateway ping timeout...

Not even that, it's a coincidence (or maybe not?) that the lifetime of a pinhole (tracked connection) in firewall is 10 s by default for pings, and that the check-gateway pings are also being sent 10 s apart.

I really only suggested a connection type (ping) that is the fastest one to die off from the connection tracking module of the firewall, allowing you to initiate a new connection rather than reusing the existing one very soon after imitating the failure of the uplink - you need to wait at max 10 seconds for the failure to be detected, then stop the outgoing ping for at least 10 seconds so that the pinhole could be dropped, and then ping again to see whether it succeeds via the working uplink or not (or, as @gutowscr471 says to be the case in ROS 7.1, whether it continues to be sent via the uplink that should be down).

I still don't understand that behaviour, as when the route via WAN 2 in table to-ISP2 becomes inactive because the check-gateway ping fails, there is still the route via WAN 1 in that table, so there is no reason why packets bearing a routing-mark to-ISP2 should fall back to routing table main and use the route via WAN 2 from that table. So maybe the routing-mark is actually not assigned properly, but I'm tired asking people over and over again to post the complete configurations rather than just those parts they deem related to their issue.

New Load Balanced Setup - Poor Performance

New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Re: New Load Balanced Setup - Poor Performance

Who is online