Load Balance/Dual WAN (PPPoE Dial out) + PPPoE Server HELP

ZSam · September 5, 2018, 1:52pm

ADDRESS NETWORK INTERFACE

0 192.168.88.2/24 192.168.88.0 bridge
1 10.0.0.2/32 10.0.0.0 ether1-gateway DSL 1
2 192.168.88.4/32 192.168.88.4 ether4-slave-local BASE HOUSE
3 D 192.168.88.2/24 192.168.88.0 ether1-gateway DSL 1
4 D 10.0.0.3/24 10.0.0.0 ether3-slave-local DSL 2
5 D 192.168.88.198/32 192.168.88.106 pppoe-attamohamed@spiderweb
6 D 192.168.88.197/32 192.168.88.105 pppoe-bardien@spiderweb
7 D 192.168.88.200/32 192.168.88.103 pppoe-domingo@spiderweb
8 D 1.1.1.1/32 3.3.3.3 ISP2
9 D 2.2.2.2/32 3.3.3.3 ISP1
10 D 192.168.88.199/32 192.168.88.110 pppoe-samodien@spiderweb

ZSam · September 5, 2018, 1:59pm

Diagram:

https://ibb.co/bQmjvK

sindy · September 5, 2018, 2:35pm

That’s nothing compared to my confusion by your network topology.

ether1 seems to, at the same time,

be a carrier interface for a WAN PPPoE client interface ISP1
have an IP address assigned dynamically by DHCP from the same subnet you use on your LAN bridge (192.168.88.2/24),
bear another IP address assigned statically, 10.0.0.2/32.

ether3 is a carrier interface for a WAN PPPoE client interface ISP2 and at the same time it has a dynamically assigned IP address 10.0.0.3/24.

So I don’t understand the overall topology and I just hope you do.

I wouldn’t mind much the IP addresses on carrier interfaces of PPPoE client interfaces (maybe they are used to manage the connected DSL modem), but I am afraid of the same or overlapping subnets being used on unrelated interfaces.

Other than that, I can see that you have replaced the public IP assigned by the ISP by 1.1.1.1 and the other one by 2.2.2.2, but the gateway IP of both is replaced by 3.3.3.3. Does it really mean that it is the same in both cases? If yes, are both these connections from the same ISP?

Also, please post /ip route print detail, I forgot that the routing-marks are not shown without the detail.

Only from the /ip address print and /ip route print I’ve understood that you probably have no client devices connected to the device’s local LAN and you test the load distribution from your PPPoE clients pppoe-userXXX@spiderweb, correct? If so, replace the in-interface=bridge in the two mangle rules with action=mark-routing by in-interface-list=all-ppp and try again.

ZSam · September 12, 2018, 10:39am

Forgive me hahaha. I built this network when I knew VERY little about networking. (Still do), so thanks for the patience.

They are supposed to be used for managing the DSL routers, yes.

Yes - they are the same (This refers to the network IP. Is this the gateway?)

THIS HAS MADE IT WORK!!! THANK YOU!!

ZSam · September 12, 2018, 10:41am

Would you still like to se ip route print detail?

sindy · September 12, 2018, 11:14am

That’s actually a tough question.

For an interface connected to a point-to-multipoint “environment” (ethernet, most wireless ones), you need the IP address of the gateway device so the networking stack could identify first the interface to use (as the gateway IP address and the local address on one of the multipoint interfaces belong to the same IP subnet) and then, using the ARP protocol, the MAC address of the gateway device accessible through that interface, and send the packets for other IP addresses to that device via that interface for forwarding them further.

For an interface connected to a point-to-point channel (serial lines, tunnels), the gateway device is by nature “the remote end of the line/tunnel”, so although that remote device may have an IP address, you rarely need to use that address on your end. You can specify such a remote IP address as a gateway and the networking stack will translate it into the interface name, but you can get the same result by specifying the interface name directly as route’s gateway.

In some situations (recursive next-hop search, which is among other things useful for implementation of scriptless failover monitoring the actual availability of a WAN link), you have to use only IP addresses as gateway identifiers, but in these situations two identical addresses break things because you need unique ones by principle - you cannot augment that identifier with anything else.

sindy · September 12, 2018, 11:15am

As it works, no need now.

ZSam · October 25, 2018, 9:36am

Hi Sindy

This method has been working really well over the last while - thanks again.

Now, as user number increases, I’ve noticed something -

When the network is busy, there are a few clients who end up getting no bandwidth , even though for example only WAN1 is maxed out, but bandwidth is available on WAN 2.

This results in those specific clients having momentary internet disconnection and cannot surf/browse until say another client stops browsing and space becomes available on WAN 1, or the connection somehow realises it can go to WAN2.

Is this an unfixable disadvantage of this setup?

It seems to be affecting QoS negatively.

Thanks!

sindy · October 25, 2018, 10:31am

The only thing which decides which connection will use which WAN are the two rules below:

add action=mark-routing chain=prerouting in-interface=bridge
new-routing-mark=to_ISP1 passthrough=no per-connection-classifier=
src-address-and-port:2/0
add action=mark-routing chain=prerouting in-interface=bridge
new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=
src-address-and-port:2/1

So first question would be whether both PPPoE connections have the same contracted speed (and thus the reason why there remains free bandwidth on WAN2 is that the two rules do not distribute the traffic evenly among the WANs) or whether you need to point a bigger share of the total traffic to WAN2 as it has more bandwidth available in absolute figures.

If the WANs have a different contract bandwidth, or if you can see that the traffic volume distribution is systematically directing more traffic to WAN1, you can modify the rules above to change the distribution ratio from 1:1 to n:m (n<m) by using n+1 rules instead of those two, the first n ones with per-connection-classifier=src-address-and-port/X/Y where X=n+m and Y is one of n randomly chosen numbers from 0 to (n+m-1) assigning the routing-mark for the thinner WAN and the last one having no per-connection-classifier confition and assigning the routing-mark for the fatter WAN.

The per-connection-classifier hash may not distribute the traffic evenly enough because some clients (addresses) may be more hungry than others and the way the hash is calculated may send requests from these clients to the same WAN. So reassigning client addresses might help if you are able to track this down.

Introducing full randomness in WAN choice would be quite complicated here, given that it requires to use connection-mark. The per-connection-classifier gives the same result (match or non-match) on all packets belonging to the same connection because it only uses the invariant attributes of the connection (addresses and port) for its verdict; if you use random, you have to “remember” the choice made when handling the initial packet of the connection, which is what connection-mark is used for. But you already use connection-mark for QoS, so you would have to use combined connection marks representing both the traffic class (to be converted into packet-mark) and the wan choice (to be converted into routing-mark). This would actually have also a positive side as you could then control QoS on each uplink separately, but it would require a serious reworking of the mangle rules.

ZSam · October 25, 2018, 10:50am

WAN 1 is about 10Mbit and WAN 2 about 7.5Mbit.

Will have to take some time to understand this.

This may be the case, so is it correct to say that hungry addresses are maxing WAN1, but then why are non-hungry addresses not jumping to WAN2? Would it be possible to split bandwidth from hungry addresses over both WAN to reduce pressure?

I would like to manage QoS per client. Been reading up and watching tutorials.

sindy · October 25, 2018, 11:08am

Putting the n,m maths simply using an example (I do know that the actual distribution of traffic is reverse, it is just an example):
WAN1 has 10 Mbit/s, WAN2 only 7.5 Mbit/s. So let’s think about it as 4 times 2.5 Mbit/s and 3 times 2.5Mbit/s. So you want 4/7 of the total traffic on WAN1 and 3/7 of the total traffic on WAN2.
And we are lazy so we want as few rules as possible.

So we use three rules to send the 3/7 to WAN2 and let the rest go to WAN1.

So three rules would have the per-connection-classifier divider and remainder values set to 7/0, 7/1 and 7/2. What none of these rules matches on would match 7/3 to 7/6, so we may assign the other routing-mark using a single rule which must be the very last one in the chain (but you can create a custom chain only for the purpose).

“Non-hungry addresses” cannot jump anywhere. per-connection-classifier has no knowledge of the current load of the WANs, it only uses addresses and ports to decide. Nor do the routiing tables chosen by routing-mark have an information about the WAN load, so adding a route via WAN2 with higher distance value to th routing table “via-WAN1” won’t help.

ZSam · October 25, 2018, 11:43am

I get this part.

A bit confused here.

ZSam · October 25, 2018, 11:48am

add action=mark-routing chain=prerouting in-interface=all-ppp \
    new-routing-mark=to_ISP1 passthrough=no per-connection-classifier=\
    src-address-and-port:7/0
add action=mark-routing chain=prerouting in-interface=all-ppp \
    new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=\
    src-address-and-port:7/1
add action=mark-routing chain=prerouting in-interface=all-ppp \
    new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=\
    src-address-and-port:7/2

Is this correct?

sindy · October 25, 2018, 12:19pm

The above sends 1/7 of all requests to WAN1 (first rule), 2/7 of all requests to WAN2 (second and thir rule), and remaining 4/7 to whatever is the WAN of the default route in the default routing table (with no routing-mark).

What I had in mind was

add action=mark-routing chain=prerouting in-interface-list=all-ppp
new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=
src-address-and-port:7/0
add action=mark-routing chain=prerouting in-interface-list=all-ppp
new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=
src-address-and-port:7/1
add action=mark-routing chain=prerouting in-interface-list=all-ppp
new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=
src-address-and-port:7/2
add action=mark-routing chain=prerouting in-interface-list=all-ppp
new-routing-mark=to_ISP1 passthrough=no per-connection-classifier=
src-address-and-port:7/2

Which BTW brings me to a thought - if there is some traffic from other sources than pppoe clients, that traffic doesn’t match in-interface-list=all-ppp and thus it uses the default route of the default routing table which probably sends it out via WAN1. Which would explain better than a biased per-connection-classifier hash why you can see more traffic on WAN1. Counter rules in chain=postrouting of /ip firewall mangle, which seems to be empty so you don’t need to worry about ruining anything by adding them, should show you that:

/ip firewall mangle
add chain=postrouting action=accept routing-mark=to_ISP1 out-interface=ISP1
add chain=postrouting action=accept routing-mark=to_ISP2 out-interface=ISP2
add chain=postrouting action=accept out-interface=ISP1
add chain=postrouting action=accept out-interface=ISP2

The rules with routing-mark match conditions count the routing-marked ougoing packets, the other two ones count those without a routing mark.

ZSam · October 25, 2018, 12:56pm

Okay, so I’ve added accordingly.

What I notice now is that one connection eg. my home pppoe cannot access both WANs and run full bandwidth i.e 17.5Mbit.

Is this correct?

ZSam · October 25, 2018, 1:03pm

Sorry - re-added and it seems to be working.

ZSam · October 25, 2018, 1:08pm

Will wait to monitor the network once everyone is home from work etc. and things get busy.

ZSam · October 25, 2018, 1:32pm

So to improve QoS and strain on bandwidth, I’d like to priorotize bandwidth in the following way:

1 - HTTP Browsing, Youtube, WhatsApp, Facebook Instagram and other social media
2 - WhatsApp/Skype/VoIP and Video calls (mobile)
3 - Video Streaming sites and Netflix etc.
4 - P2P limited to ZERO when the network is busy, but allowed the user’s full bandwidth allocation when not.

Questions:
a) Can this be implemented for the network as a whole or does it have to be done per client?
b) Do the queues HAVE to be allocated only a portion of the bandwidth (relative in amount to importance), or can they (all protocols) each be allowed to pull the full available bandwidth in the event that say (for understanding purposes), all clients are doing HTTP, or YouTube, or Netflix?
So essentially, if everyone is streaming, the full bandwidth is used (limited per user by their PPPoE profile limit), but when someone wants to browse, they can then use their full bandwidth to browse or YouTube, while others continue streaming at their full bandwidth?

HOPE THIS MAKES SENSE.

I have the following mangle rules:
*I am not sure if they are set up correctly in terms of interfaces they are pointed at etc. as when trying to create queue trees, I do not see traffic flow through the queues.

/ip firewall mangle
add action=mark-packet chain=prerouting comment=\
    "internal-traffic packet mark" dst-address-list=internal-nets \
    new-packet-mark=internal-traffic passthrough=yes src-address-list=\
    internal-nets
add action=mark-packet chain=prerouting comment=\
    "customer-servers-out packet mark" new-packet-mark=customer-servers-out \
    passthrough=yes src-address-list=customer-servers
add action=mark-packet chain=prerouting comment=\
    "customer-servers-in packet mark" dst-address-list=customer-servers \
    new-packet-mark=customer-servers-in passthrough=yes
add action=mark-packet chain=prerouting comment="admin-in packet mark DNS" \
    in-interface="ether1-gateway DSL 1" new-packet-mark=admin-in passthrough=\
    yes protocol=udp src-port=53
add action=mark-packet chain=prerouting comment="admin-in packet mark snmp" \
    dst-port=161 in-interface="ether1-gateway DSL 1" new-packet-mark=admin-in \
    passthrough=yes protocol=udp
add action=mark-connection chain=prerouting comment=\
    "Remote Protocols admin connection mark" new-connection-mark=admin \
    passthrough=yes port=20,21,22,23,3389,8291 protocol=tcp
add action=mark-connection chain=prerouting comment=\
    "icmp connection mark as admin" new-connection-mark=admin passthrough=yes \
    protocol=icmp src-address-list=internal-nets
add action=mark-packet chain=prerouting comment="admin-in packet mark" \
    connection-mark=admin in-interface="ether1-gateway DSL 1" \
    new-packet-mark=admin-in passthrough=yes
add action=mark-packet chain=prerouting comment="admin-out packet mark" \
    connection-mark=admin new-packet-mark=admin-out passthrough=yes
add action=mark-packet chain=prerouting comment=\
    "streaming video in packet mark" connection-mark=streaming-video \
    in-interface="ether1-gateway DSL 1" new-packet-mark=streaming-video-in \
    passthrough=yes
add action=mark-packet chain=prerouting comment=\
    "streaming video out packet mark" connection-mark=streaming-video \
    new-packet-mark=streaming-video-out passthrough=yes
add action=mark-connection chain=prerouting comment=\
    "http traffic connection mark" dst-port=80,443 new-connection-mark=http \
    passthrough=yes protocol=tcp src-address-list=internal-nets
add action=mark-connection chain=prerouting comment=\
    "http traffic connection mark" connection-bytes=5000000-4294967295 \
    dst-port=80,443 new-connection-mark=http-download passthrough=yes \
    protocol=tcp src-address-list=internal-nets
add action=mark-packet chain=prerouting comment="http in packet mark" \
    connection-mark=http in-interface=ISP1 new-packet-mark=http-in \
    passthrough=yes
add action=mark-packet chain=prerouting comment="http out packet mark" \
    connection-mark=http in-interface=ISP1 new-packet-mark=http-out \
    passthrough=yes
add action=mark-connection chain=prerouting comment=\
    "wow connetion mark as gaming" dst-port=\
    1119,3724,6112-6114,4000,6881-6999 new-connection-mark=games passthrough=\
    yes protocol=tcp src-address-list=internal-nets
add action=mark-connection chain=prerouting comment=\
    "eve online connetion mark as gaming" dst-address=87.237.38.200 \
    new-connection-mark=games passthrough=yes src-address-list=internal-nets
add action=mark-connection chain=prerouting comment=\
    "starcraft 2 connetion mark as gaming" dst-port=1119 new-connection-mark=\
    games passthrough=yes protocol=tcp src-address-list=internal-nets
add action=mark-connection chain=prerouting comment=\
    "heros of newerth connetion mark as gaming" dst-port=11031,11235-11335 \
    new-connection-mark=games passthrough=yes protocol=tcp src-address-list=\
    internal-nets
add action=mark-connection chain=prerouting comment=\
    "steam connetion mark as gaming" dst-port=27014-27050 \
    new-connection-mark=games passthrough=yes protocol=tcp src-address-list=\
    internal-nets
add action=mark-connection chain=prerouting comment=\
    "xbox live connetion mark as gaming" dst-port=3074 new-connection-mark=\
    games passthrough=yes protocol=tcp src-address-list=internal-nets
add action=mark-connection chain=prerouting comment=\
    "ps3 online connetion mark as gaming" dst-port=5223 new-connection-mark=\
    games passthrough=yes protocol=tcp src-address-list=internal-nets
add action=mark-connection chain=prerouting comment=\
    "wii online connetion mark as gaming" dst-port=28910,29900,29901,29920 \
    new-connection-mark=games passthrough=yes protocol=tcp src-address-list=\
    internal-nets
add action=mark-packet chain=prerouting comment=\
    "games packet mark forever-saken-game" dst-address-list=external-nets \
    new-packet-mark=games-in passthrough=yes src-address-list=\
    forever-saken-game
add action=mark-packet chain=prerouting comment=\
    "games packet mark starcraft2" dst-address-list=external-nets \
    new-packet-mark=games-in passthrough=yes protocol=udp src-port=1119,6113
add action=mark-packet chain=prerouting comment="games packet mark wow" \
    dst-address-list=external-nets new-packet-mark=games-in passthrough=yes \
    protocol=udp src-port=53,3724
add action=mark-packet chain=prerouting comment="games packet mark HoN" \
    dst-address-list=external-nets new-packet-mark=games-in passthrough=yes \
    protocol=udp src-port=11031,11235-11335
add action=mark-packet chain=prerouting comment="games packet mark steam in" \
    dst-address-list=external-nets new-packet-mark=games-in passthrough=no \
    port=4380,28960,27000-27030 protocol=udp
add action=mark-packet chain=prerouting comment="games packet mark steam out" \
    dst-port=53,1500,3005,3101,3478,4379-4380,4380,28960,27000-27030,28960 \
    new-packet-mark=games-out passthrough=yes protocol=udp src-address-list=\
    internal-nets
add action=mark-packet chain=prerouting comment="games packet mark xbox live" \
    dst-address-list=external-nets new-packet-mark=games-in passthrough=yes \
    protocol=udp src-port=88,3074,3544,4500
add action=mark-packet chain=prerouting comment=\
    "games packet mark ps3 online" dst-address-list=external-nets \
    new-packet-mark=games-in passthrough=yes protocol=udp src-port=\
    3478,3479,3658
add action=mark-packet chain=prerouting comment="games packet mark in" \
    connection-mark=games dst-address-list=external-nets new-packet-mark=\
    games-in passthrough=yes
add action=mark-packet chain=prerouting comment="games packet mark out" \
    connection-mark=games new-packet-mark=games-out passthrough=yes
add action=mark-packet chain=prerouting comment=\
    "voip-in packet mark teamspeak" dst-address-list=external-nets \
    new-packet-mark=voip-in passthrough=yes protocol=udp src-port=9987
add action=mark-packet chain=prerouting comment=\
    "voip-out packet mark teamspeak" dst-port=9987 new-packet-mark=voip-out \
    passthrough=yes protocol=udp src-address-list=internal-nets
add action=mark-packet chain=prerouting comment=\
    "voip-out packet mark teamspeak" dst-address-list=external-nets \
    new-packet-mark=voip-in passthrough=yes protocol=udp src-port=9987
add action=mark-packet chain=prerouting comment=\
    "voip-in packet mark ventrilo" dst-address-list=external-nets \
    new-packet-mark=voip-in passthrough=yes protocol=udp src-port=3784
add action=mark-packet chain=prerouting comment=\
    "voip-out packet mark ventrilo" dst-port=3784 new-packet-mark=voip-out \
    passthrough=yes protocol=udp src-address-list=internal-nets
add action=mark-packet chain=prerouting comment=\
    "voip-in packet mark ventrilo" dst-address-list=external-nets \
    new-packet-mark=voip-in passthrough=yes protocol=tcp src-port=3784
add action=mark-packet chain=prerouting comment=\
    "voip-out packet mark ventrilo" dst-port=3784 new-packet-mark=voip-out \
    passthrough=yes protocol=tcp src-address-list=internal-nets
add action=mark-packet chain=prerouting comment="voip-in packet mark SIP" \
    dst-address-list=internal-nets new-packet-mark=voip-in passthrough=yes \
    port=5060 protocol=tcp
add action=mark-packet chain=prerouting comment="voip-out packet mark SIP" \
    new-packet-mark=voip-out passthrough=yes port=5060 protocol=tcp \
    src-address-list=internal-nets
add action=mark-packet chain=prerouting comment="voip-in packet mark udp SIP" \
    dst-address-list=internal-nets new-packet-mark=voip-in passthrough=yes \
    port=5004,5060 protocol=udp
add action=mark-packet chain=prerouting comment=\
    "voip-out packet mark udp SIP" new-packet-mark=voip-out passthrough=yes \
    port=5004,5060 protocol=udp src-address-list=internal-nets
add action=mark-packet chain=prerouting comment="voip-in packet mark RTP" \
    dst-address-list=internal-nets new-packet-mark=voip-in packet-size=\
    100-400 passthrough=yes port=16348-32768 protocol=udp
add action=mark-packet chain=prerouting comment="voip-out packet mark RTP" \
    new-packet-mark=voip-in packet-size=100-400 passthrough=yes port=\
    16348-32768 protocol=udp src-address-list=internal-nets
add action=mark-packet chain=prerouting comment="vpn-in packet mark GRE" \
    in-interface="ether1-gateway DSL 1" new-packet-mark=vpn-in passthrough=\
    yes protocol=gre
add action=mark-packet chain=prerouting comment="vpn-out packet mark GRE" \
    new-packet-mark=vpn-out passthrough=yes protocol=gre
add action=mark-packet chain=prerouting comment="vpn-in packet mark ESP" \
    in-interface="ether1-gateway DSL 1" new-packet-mark=vpn-in passthrough=\
    yes protocol=ipsec-esp
add action=mark-packet chain=prerouting comment="vpn-out packet mark ESP" \
    new-packet-mark=vpn-out passthrough=yes protocol=ipsec-esp
add action=mark-packet chain=prerouting comment=\
    "vpn-in packet mark VPN UDP ports" in-interface="ether1-gateway DSL 1" \
    new-packet-mark=vpn-in passthrough=yes protocol=udp src-port=\
    500,1701,4500
add action=mark-packet chain=prerouting comment=\
    "vpn-out packet mark VPN UDP ports" new-packet-mark=vpn-out passthrough=\
    yes protocol=udp src-port=500,1701,4500
add action=mark-packet chain=prerouting comment="vpn-in packet mark PPTP" \
    in-interface="ether1-gateway DSL 1" new-packet-mark=vpn-in passthrough=\
    yes protocol=tcp src-port=1723
add action=mark-packet chain=prerouting comment="vpn-out packet mark PPTP" \
    new-packet-mark=vpn-out passthrough=yes protocol=tcp src-port=1723
add action=mark-routing chain=prerouting in-interface=all-ppp \
    new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=\
    src-address-and-port:7/0
add action=mark-routing chain=prerouting in-interface=all-ppp \
    new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=\
    src-address-and-port:7/1
add action=mark-routing chain=prerouting in-interface=all-ppp \
    new-routing-mark=to_ISP2 passthrough=no per-connection-classifier=\
    src-address-and-port:7/2
add action=mark-routing chain=prerouting in-interface=all-ppp \
    new-routing-mark=to_ISP1 passthrough=no