Lagging on load-balance setup

microx · August 8, 2014, 2:17am

Hello everyone,

I had set up a RB2011 as a load balancer for 4 WAN connections as shown below. It worked fine for around 2 months, and then recently I started seeing really poor performance. Around the same time, I also added a 5th WAN connection in the same way, which I doubt is the problem, but I am not so sure. I tried disconnecting the 5th WAN, and I am still getting problems.

The behavior we’re getting is, when you request a web-page, 20% of the time it will work right away, but most of the time, you get nothing for several minutes or you never get it. It just says “Connecting” or similar in the browser. If you keep hitting reload for a few minutes, then maybe you get it, and then it loads quickly. Sometimes rebooting the router resolves the problem for a few minutes, or an hour or two, but not always.

I am thinking it could be one of three things:

The configuration is incorrect
The RB2011 “burnt out” or something after 2 months
We are being attacked, DDoS or similar

So I wanted to run this configuration by you guys to try and rule out that I did a bad job of setting it up, before I jump to the latter 2 more extreme conclusions.

If its not the config, how do I go about troubleshooting this?

For my setup, basically, I have 5 PPPoE connections, going through bridged ADSL modems, each providing 1Mbps of bandwidth (which is the best I can get from my ISP in this location, and the main reason why I am doing this setup to aggregate bandwidth).

I have placed the 5 WANs on ports called gate1..gate5, and I’ve put each on a separate 192.168.X.0/24 subnet.
Ports ether1 and ether2 are bridged and connect to the LAN, which is a 10.1.0.0/16 subnet.
I then use PCC to load balance connections from the LAN to one of the WAN connections.
I am also allowing access to SSH and HTTP/HTTPS to 2 local servers, using port forwarding.

Here is my full config, from an export command. I’ve replaced private specifics with Xs.
/interface bridge
add admin-mac=XX:XX:XX:XX:XX:XX arp=proxy-arp auto-mac=no l2mtu=1598 name=bridge-local
/interface ethernet
set [ find default-name=ether3 ] name=gate1
set [ find default-name=ether4 ] name=gate2
set [ find default-name=ether5 ] name=gate3
set [ find default-name=ether6 ] name=gate4
set [ find default-name=ether7 ] name=gate5
set [ find default-name=ether8 ] name=gate6
set [ find default-name=ether9 ] name=gate7
set [ find default-name=ether10 ] name=gate8
/ip neighbor discovery
set gate1 discover=no
set gate2 discover=no
set gate3 discover=no
set gate4 discover=no
set gate5 discover=no
set gate6 discover=no
set gate7 discover=no
set gate8 discover=no
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=3des
/ip pool
add name=default-dhcp ranges=10.1.200.1-10.1.209.254
/interface pppoe-client
add ac-name=“” add-default-route=no allow=pap,chap,mschap1,mschap2 dial-on-demand=no disabled=no interface=gate1 keepalive-timeout=5
max-mru=1480 max-mtu=1480 mrru=disabled name=pppoe1 password=XXX profile=default service-name=“” use-peer-dns=no user=XXX
add ac-name=“” add-default-route=no allow=pap,chap,mschap1,mschap2 dial-on-demand=no disabled=no interface=gate2 keepalive-timeout=5
max-mru=1480 max-mtu=1480 mrru=disabled name=pppoe2 password=XXX profile=default service-name=“” use-peer-dns=no user=XXX
add ac-name=“” add-default-route=no allow=pap,chap,mschap1,mschap2 dial-on-demand=no disabled=no interface=gate3 keepalive-timeout=5
max-mru=1480 max-mtu=1480 mrru=disabled name=pppoe3 password=XXX profile=default service-name=“” use-peer-dns=no user=XXX
add ac-name=“” add-default-route=no allow=pap,chap,mschap1,mschap2 dial-on-demand=no disabled=no interface=gate4 keepalive-timeout=5
max-mru=1480 max-mtu=1480 mrru=disabled name=pppoe4 password=XXX profile=default service-name=“” use-peer-dns=no user=XXX
add ac-name=“” add-default-route=no allow=pap,chap,mschap1,mschap2 dial-on-demand=no disabled=no interface=gate5 keepalive-timeout=5
max-mru=1480 max-mtu=1480 mrru=disabled name=pppoe5 password=XXX profile=default service-name=“” use-peer-dns=no user=XXX
/interface bridge port
add bridge=bridge-local interface=ether1
add bridge=bridge-local interface=ether2
/ip address
add address=10.1.1.1/16 interface=bridge-local network=10.1.0.0
add address=192.168.1.100/24 interface=gate1 network=192.168.1.0
add address=192.168.2.100/24 interface=gate2 network=192.168.2.0
add address=192.168.3.100/24 interface=gate3 network=192.168.3.0
add address=192.168.4.100/24 interface=gate4 network=192.168.4.0
add address=192.168.5.100/24 interface=gate5 network=192.168.5.0
/ip dns
set allow-remote-requests=yes servers=X.X.X.X,X.X.X.X
/ip dns static
add address=10.1.1.1 name=mikrotik
add address=192.168.1.1 name=dsl1
add address=192.168.2.1 name=dsl2
add address=192.168.3.1 name=dsl3
add address=192.168.4.1 name=dsl4
add address=192.168.5.1 name=dsl5
add address=10.1.100.1 name=localserver1
add address=10.1.100.2 name=localserver2
/ip firewall filter
add chain=input protocol=icmp
add chain=input connection-state=established
add chain=input connection-state=related
add chain=forward connection-state=established
add chain=forward connection-state=related
add action=drop chain=forward connection-state=invalid
/ip firewall mangle
add chain=prerouting dst-address=10.1.0.0/16 in-interface=bridge-local
add chain=prerouting dst-address=192.168.1.0/24 in-interface=bridge-local
add chain=prerouting dst-address=192.168.2.0/24 in-interface=bridge-local
add chain=prerouting dst-address=192.168.3.0/24 in-interface=bridge-local
add chain=prerouting dst-address=192.168.4.0/24 in-interface=bridge-local
add chain=prerouting dst-address=192.168.5.0/24 in-interface=bridge-local
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe1 new-connection-mark=wan1
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe2 new-connection-mark=wan2
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe3 new-connection-mark=wan3
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe4 new-connection-mark=wan4
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe5 new-connection-mark=wan5
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local
new-connection-mark=wan1 per-connection-classifier=both-addresses:5/0
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local
new-connection-mark=wan2 per-connection-classifier=both-addresses:5/1
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local
new-connection-mark=wan3 per-connection-classifier=both-addresses:5/2
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local
new-connection-mark=wan4 per-connection-classifier=both-addresses:5/3
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local in-interface=bridge-local
new-connection-mark=wan5 per-connection-classifier=both-addresses:5/4
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=bridge-local new-routing-mark=wan1
add action=mark-routing chain=prerouting connection-mark=wan2 in-interface=bridge-local new-routing-mark=wan2
add action=mark-routing chain=prerouting connection-mark=wan3 in-interface=bridge-local new-routing-mark=wan3
add action=mark-routing chain=prerouting connection-mark=wan4 in-interface=bridge-local new-routing-mark=wan4
add action=mark-routing chain=prerouting connection-mark=wan5 in-interface=bridge-local new-routing-mark=wan5
add action=mark-routing chain=output connection-mark=wan1 new-routing-mark=wan1
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=wan2
add action=mark-routing chain=output connection-mark=wan3 new-routing-mark=wan3
add action=mark-routing chain=output connection-mark=wan4 new-routing-mark=wan4
add action=mark-routing chain=output connection-mark=wan5 new-routing-mark=wan5
/ip firewall nat
add action=masquerade chain=srcnat out-interface=all-ppp to-addresses=0.0.0.0
add action=masquerade chain=srcnat out-interface=gate1 to-addresses=192.168.1.0/24
add action=masquerade chain=srcnat out-interface=gate2 to-addresses=192.168.2.0/24
add action=masquerade chain=srcnat out-interface=gate3 to-addresses=192.168.3.0/24
add action=masquerade chain=srcnat out-interface=gate4 to-addresses=192.168.4.0/24
add action=masquerade chain=srcnat out-interface=gate5 to-addresses=192.168.5.0/24
add action=dst-nat chain=dstnat dst-port=22 in-interface=all-ppp protocol=tcp to-addresses=10.1.100.1 to-ports=22
add action=dst-nat chain=dstnat dst-port=80 in-interface=all-ppp protocol=tcp to-addresses=10.1.100.2 to-ports=80
add action=dst-nat chain=dstnat dst-port=443 in-interface=all-ppp protocol=tcp to-addresses=10.1.100.2 to-ports=443
/ip route
add check-gateway=ping distance=1 gateway=X.X.X.X%pppoe1 routing-mark=wan1
add check-gateway=ping distance=1 gateway=X.X.X.X%pppoe2 routing-mark=wan2
add check-gateway=ping distance=1 gateway=X.X.X.X%pppoe3 routing-mark=wan3
add check-gateway=ping distance=1 gateway=X.X.X.X%pppoe4 routing-mark=wan4
add check-gateway=ping distance=1 gateway=X.X.X.X%pppoe5 routing-mark=wan5
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe2
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe1
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe3
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe4
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe5
/ip upnp
set allow-disable-external-interface=no
/system clock
set time-zone-name=Etc/GMT-3
/system identity
set name=mikrotik
/system ntp client
set enabled=yes primary-ntp=196.43.1.14 secondary-ntp=41.73.40.9
/tool graphing interface
add
/tool mac-server
set [ find default=yes ] disabled=yes
add interface=bridge-local
/tool mac-server mac-winbox
set [ find default=yes ] disabled=yes
add interface=bridge-local

Caci99 · August 8, 2014, 8:55am

I would make slight changes to your config as follows:

/ip firewall mangle
add action=mark-connection chain=input connection-mark=no-mark in-interface=pppoe1 new-connection-mark=wan1
add action=mark-connection chain=input connection-mark=no-mark in-interface=pppoe2 new-connection-mark=wan2
add action=mark-connection chain=input connection-mark=no-mark in-interface=pppoe3 new-connection-mark=wan3
add action=mark-connection chain=input connection-mark=no-mark in-interface=pppoe4 new-connection-mark=wan4
add action=mark-connection chain=input connection-mark=no-mark in-interface=pppoe5 new-connection-mark=wan5

In your config you have used chain=prerouting, while these rules are used for keeping the connections from outside to the router itself on the same gateway, so chain=input should be used. Then these rules should be followed by the routing mark, so they don’t go on to be processed by the other rules:

/ip firewall mangle
add action=mark-routing chain=output connection-mark=wan1 new-routing-mark=wan1 passthrough=no
add action=mark-routing chain=output connection-mark=wan2 new-routing-mark=wan2 passthrough=no
add action=mark-routing chain=output connection-mark=wan3 new-routing-mark=wan3 passthrough=no
add action=mark-routing chain=output connection-mark=wan4 new-routing-mark=wan4 passthrough=no
add action=mark-routing chain=output connection-mark=wan5 new-routing-mark=wan5 passthrough=no

Remember to add passthrough=no so they don’t get processed further down the other rules. This will ease a bit the CPU.
Also, on the rules you mark routing for the PCC, remember to put passthorugh=no, otherwise the packet might enter in some kind of loop in the marking rules:

/ip firewall mangle
add action=mark-routing chain=prerouting connection-mark=wan1 in-interface=bridge-local new-routing-mark=wan1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan2 in-interface=bridge-local new-routing-mark=wan2 passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan3 in-interface=bridge-local new-routing-mark=wan3 passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan4 in-interface=bridge-local new-routing-mark=wan4 passthrough=no
add action=mark-routing chain=prerouting connection-mark=wan5 in-interface=bridge-local new-routing-mark=wan5 passthrough=no

In the routes table you have five gateways with the same distance with no routing mark. I am talking about this ones:

add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe2
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe1
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe3
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe4
add check-gateway=ping distance=2 gateway=X.X.X.X%pppoe5

From these ones, I think only one is active because they are doing the same thing, so router will choose one of them. This usually serves for connections of the router itself, like dns resolving. The lag you are seeing might be related to the dns resolving. Now, if all your internet connections come from the same ISP you can use their dns server, but if they are from different ISP you should use some public dns server (like google dns server for example). I would also suggest to redirect all users to the router dns cache using this rule:

/ip firewall nat
add chain=dstnat action=redirect to-ports=53 protocol=udp dst-address-type=!local dst-port=53 
add chain=dstnat action=redirect to-ports=53 protocol=tcp dst-address-type=!local dst-port=53

And finally, protect the dns cache from outside attacks:

/ip firewall filter
add chain=input action=drop protocol=tcp in-interface=pppoe1 dst-port=53 
add chain=input action=drop protocol=udp in-interface=pppoe1 dst-port=53

repeat these rules for every pppoe interface

microx · August 11, 2014, 1:53pm

Thanks Toni, this is very helpful!

I actually figured out that a DNS attack was involved. I found many connections with port 53.

So I added the last 2 rules you mentioned right away, the ones protecting port 53, and everything went smoother finally!

I also added a “chain=input in-interface=!bridge-local action=drop” rule at the very end of the filter section to stop similar connections to the router, and this seems to have gotten rid of a few other suspicious connections.

I’ll try the rest of your suggestions also and see what effect they have to optimize.

Thanks!
–Hesham

Caci99 · August 12, 2014, 9:13am

Glad to hear it helped

Be carefull with that rule though, you might be blocking useful connections from your LAN to the router, might even left yourself out from the router.

arafat586 · December 27, 2014, 9:39am

Bandwidth Merge or Load Balance with Failover step by step
Bandwidth Merge or Load Balance with Failover is One of the most generally utilized applications of load balancing is to supply a single Internet service from double service, such as when we take from two ISP internet line 4Mbps+4Mbps=8Mbps than it will be equal 8Mbps. If we do two ISP line merge than we will get double Bandwidth. On the other hand, when one connection will be down then another connection automatically will run. It is generally mixed with Failover and backup services occasionally known as a server farm and Local area Network. Generally load-balanced systems include common web sites, heavy Internet Relay gossip networks, high-bandwidth File Transfer communications protocol sites.in a single word it is called redundant communications Links

arafat586 · December 30, 2014, 11:14am

Step 1: At first put IP/subnet mask=192.168.0.1/24, G.W:192.168.0.1 Ether=3>Apply>ok (ISP 1)
second put IP/Subnet mask=10.10.0.1/24, G.W: 10.10.0.1 Ether=4 >Apply>ok (ISP 2)

winbox>IP>Firewall>Mangle+General>Chain=Prerouting,>Dst.Address=192.168.0.0/24

Step2:IP>Firewall>Mangle+ Action=accept > Apply >OK

Step3:IP>Firewall>Mangle+General>Chain=Prerouting,Dst.Address=10.10.0.0/24, Action=accept >Apply>OK (ISP2)

Step 4: IP>Firewall> Mangle+General >Chain=input> In. Interface=ether3>Apply>OK

Step 5:IP>Firewall + Mangle + Action=Mark connection > New connection Mark=ISP1 >Apply>ok

Step 6: IP>Firewall> Mangle+General >Chain=input> In. Interface=ether4>Apply>OK

Step 7: IP>Firewall>Mangle+ Action=mark connection>New Connection Mark=ISP2>OK

Step 8: IP>Firewall + Mangle > General>Chain=output>connection Mark=ISP1>Apply>OK

Step 9: IP>Firewall + Mangle > Action=mark routing>New Routing Mark=ISP1>Apply>OK

Step 10:IP>Firewall + Mangle >General>Chain=output>connection Mark=ISP2>Apply>OK

Step11: IP>Firewall + Mangle > Action=mark routing>New Routing Mark=ISP2>Apply>OK

Step12.IP>Routes+General>Getway=192.168.0.1> Routing Mark=ISP1>Apply>OK

Step 12. IP>Routes +General> Getway=192.168.0.1 > Routing Mark=ISP2>Apply>OK

step13. when I will complete the all step then we will check load balance. I will create the LAN site. IP>address + address=172.16.0.1/24 >Interface=ether5

At last I will put the LAN site IP on PC or Laptop then we can check load balance and Failover