Using most available bandwidth wan

RouterOS General
1

Hello

I have two ISPs. Both have 50mbit download and 5mbit upload individually. I believe I have load balancing working correctly. When i do speed tests I can see around 90mbit download speeds. However during uploads, it never passes 5mbit. I even tried paralel but independent upload tests with iperf3. The result always seemed to be, one ISP using all upload bandwidth, and the other one doing nothing because there is nothing left for it. how can I setup mikrotik to know how much bandwith is available for each ISP and use that outgoing connection for higher speeds? Is this even possible? Thank you

Two wan with in the same subnet
2

You cannot learn what are the limitations of your upload bandwidth at ISP side or further in the network. You can use Mikrotik’s queues to actively limit the upload bandwidth of different classes of uplink traffic generated in your LAN (and by the router itself) according to your preferences as it leaves your router. There’s a nice guide on that by @pcunite.

But it is still weird that you’d be getting full 5 Mbit/s on one uplink and zero on the other one, it rather sounds like some mistake in the load balancing… do you use per-connection-classifier or some other strategy to choose an uplink for a connection? The per-connection-classifier may allocate the same uplink to all iperf3 instances due to inappropriate choice of the classifying hash.

If there is really no issue in your configuration, it may also be the ISP cheating.

3

I was hoping something like this.
I know my maximum upload bandwidth is 5mbit per ISP. If for ISP1, I’m using 70% of upload bandwidth, and for ISP2, I’m using 30% of upload bandwidth, then the new upload should use ISP2. Is this an unrealistic expectation?

As for my config. It is like this. I’m not sure what strategy it uses. I followed the instructions on https://www.youtube.com/watch?v=DDMD1GVg84M It just seemed to work.

When I do

 curl ifconfig.me

several times, I get returned different ips. It almost alternates between ISP1 and ISP2 Ips evenly.

My config is below

reducted
4

This setup uses ECMP load balancing for connections initiated from LAN or the router itself, and makes sure that incoming requests to the router itself (like VPN connections initiated by remote client, remote management connections via WAN) are responded via the same WAN through which they arrived. ECMP load balancing is iterating the list of gateways on the chosen route using round robin, hence in your case, every odd connection uses one gateway and every even one uses the other one; I’m not sure whether it is still the case if both source IP and destination IP of the connection are the same, i.e. whether the routing cache is somehow linked with connection tracking and thus also takes source and destination ports into account.


I’m afraid that watching the current traffic volume at each uplink and choosing the least loaded one (in % of its maximum capacity) for a new connection would be too much work for too little effect as the networking environment is very dynamic. So even if you measure the currently occupied bandwidth every second (which is about the fastest pace you can obtain using scripts), during that second tens to hundreds of existing connections can finish and tens to hundreds new ones can emerge. Plus each connection usually requires a different bandwidth, as the speed accomodates to the bandwidth available at the bottleneck of the path between the endpoints (client and server). So the best you can do is to distribute the connections evenly (taking the absolute bandwidth into account, i.e. if you have a 100 Mbit/s link and a 20 Mbit/s link, “evenly” means 5:1) and hope that statistically, it will work as needed. Where the upload/download ratios differ between the uplinks (like 50/50 and 20/4), there’s no “ideal” strategy.

5

I suspect that video does not shoe the “full truth”

ECMP is based on per connection, so if src and dst address is same, you will only use one of the uplinks, it is not a “per packet” solution

6

Hello

So is the recommended approach to abandon ECMP and instead use PCC? Will this allow me to distribute the connections evenly? Can i make the disrtibution ratio different for download and upload?
Thank yoh

7

As for me, yes. But it will cost you a bit of CPU power, as mangling is incompatible with fasttracking (which is only used for forwarded packets, not for connections to/from the router itself). So depending on your Mikrotik model, the consequences may vary from “unnoticed” to “unusable”.


Yes


No, because connections almost always use both directions (even where the payload as such goes in only one direction, some control packets like TCP ACK must be sent in the opposite direction). There are few exceptions to this (an audio/video stream in UDP may theoretically not need the reverse channel). And the choice of WAN has to be done when the very first packet of the connection is handled, and at that moment, it is not known yet whether the connection will be used for upload or for download; later, while the connection is already running, the router could compare the download and upload rates, but it cannot change the WAN used by an ongoing connection if there is src-nat anywhere on the way between the endpoints, because the remote server asociates the public IP of that WAN with that connection. So it would ignore packets coming from another address, and it keeps sending to the address from which the initial request came.

8

Ok. Let’s say I want to apply the example here. https://wiki.mikrotik.com/wiki/Manual:PCC How would it work to my current setup? I have many vlans and a bridge.

TTNET1 and TTNET2 are my ISPs. My modems only allows dhcp connection from it. Therefore, I have set DMZ on the modems, with a static ip for the mikrotik DHCP clients.

Since I have so many VLANs, what do I choose as the interface for lines such as in the example

reducted
9

One could not expect to receive more than 50Mbps download and thus this thread is suspect from the first post.
The MT router will not use both ISPs for a single session as far I understand, but then again my MTUNA certification doesnt cover magical MT thinking. :stuck_out_tongue_winking_eye:

10

I think speedtest.net uses multiple connections for download. That is where i saw more than 50mbit. But yeah for the rest i get it. I just need help to change my config to pcc now. Can you help me with that please. I am not sure how to adapt my vlans and bridge to the pcc example

11

Yes, speedtest.net does use multiple connections, but it is unclear if it uses multiple servers.

If the src / dst ports are also used in the route caching algorithm, then it will make sense why you probably see full bandwidth from a single device during speedtest. The Wiki article only mentions source / destination addresses, etc being used, nothing about src/dst port

EDIT: To add, you should use ECMP if no NAT is involved and PCC if NAT is involved

12

I dont use NAT right now. I just have. Mikrotik vpn server setup. People vpn first, grab a local ip, and co nect to internal servers.

So if this is the case i should use ecmp? Then is what i currenctly have what i can have as the ideal setup?

13

Where does VPN come into the picture now?

Your config shows LAN IPs in 10.x.x.x range, which is private IPs, so there must be NAT somewhere in the path on its way out to the internet?

14

I just have this for NAT. The modems have DMZ set, and direct all traffic to mikrotik dhcp clients with static ip addresses. When I said, I don’t use NAT, I meant no special port forwarding. This much NAT seems to work with ECMP.

/ip firewall nat
add action=masquerade chain=srcnat  \
    src-address=10.0.0.0/8
add action=masquerade chain=srcnat out-interface="TTNET1 VLAN"
add action=masquerade chain=srcnat out-interface="TTNET2 VLAN"

I have something like this for VPN:

/interface l2tp-server
add name="AHMET VPN" user=ahmet
/ppp profile
add bridge="SWITCH BRIDGE" local-address="104 TRUSTED POOL" name="VPN PROFILE" \
    remote-address="104 TRUSTED POOL"
/interface l2tp-server server
set default-profile="VPN PROFILE" enabled=yes use-ipsec=\
    yes
/ppp secret
add name=ahmet profile="VPN PROFILE"
15

Is that VPN only for connection to company LAN or the VPN users can connect to something in the internet via the VPN? If only for LAN, there is no need to distribute the load using ECMP or PCC, as the WAN uplink is chosen already by the user which connects to one or the other public IP.

16

Hello

I think the conversation led off tangent. I dont have a problem with vpn or try to load balance my connections via vpn to the internet. I want to do it to the physical servers on the LAN.

Right now, i use ECMP, I underatanf PCC can distrubute the load better. I have many vlans and a bridge. I am not sure how to apply the PCC example on mikrotik wiki. Thabk you all for helping btw.

17

The per-connection-classifier just provides always the same result (match or mismatch) for all the packets belonging to the same connection, because it calculates a hash from source address, source port, destination address, and destination port (a combination of which is the same for each connection). There are cases where it is better not to include some of these identifiers into the hash, but we’ll get to these later.

What action the rule takes if the per-connection-classifier matches, and what other matchers you add to the rule, is up to you. Without connection marking, the typical use is that the rules matching on per-connection-classifier directly assign a routing-mark, which is then used to choose the necessary set of routes (routing table). But the caveat is that if a packet has a routing-mark assigned, routes without any routing mark are only used to route it if none of the routes with that routing-mark matches. As you only have default routes with a routing-mark, they match any packet with such a routing-mark, even if its destination is one of your local subnets.

So either the firewall mangle rules assigning the routing-mark must assign it only to packets which you know will need to be sent via WAN, or you have to override the routing-mark assignment later on using /ip route rule rows. So one possible solution would be:

/interface list
add name=WAN

/interface list member
add list=WAN interface=TTNET1
add list=WAN interface=TTNET2

/ip firewall address-list
add list=local-subnets address=local.sub.net.A/mask
…
add list=local-subnets address=local.sub.net.Z/mask

/ip firewall mangle
add chain=prerouting in-interface-list=!WAN dst-address-list=!local-subnetsper-connection-classifier=src-address:2/0
action=mark-routing new-routing-mark=“TTNET1 ROUTING”
add chain=prerouting in-interface-list=!WAN dst-address-list=!local-subnetsper-connection-classifier=src-address:2/1
action=mark-routing new-routing-mark=“TTNET2 ROUTING”

If nothing at private addresses needs to be reached via the WANs, you can use an address-list of all private IPs instead of al your local subnets, which consists of just three prefixes for this purpose - 10.0.0.0/8, 172.16.0.0/12, and 192.168.0.0/16.

Regarding calculating the hash only from the local address (the source one in these rules in particular) - some services have problems if different TCP sessions related to the same operation arrive from different addresses. So for a small number of clients who have a short enough path to complain, it may give a smoother traffic distribution to use both-addresses-and-ports; if the clients who eventually experience the issues with these picky servers cannot deliver the information to you efficiently enough, it is much safer to use the same WAN for all connections of a given local user, otherwise the information about these easy to solve problems reaches you in the form of “your service is a (beep), it doesn’t work”. But for just a few local users with different traffic volumes, this approach causes uneven traffic distribution.

18

Thank you. So just to be clear. All I have to do is to have this on my mangle, and remove existing records. Nothing else? The routing rules stays the same? I am afraid it did work. When I had these none of the servers were able to access the internet.

/ip firewall mangle
add action=mark-routing chain=prerouting disabled=yes dst-address=!10.0.0.0/8 \
    in-interface-list="!WAN" new-routing-mark="TTNET1 ROUTING" \
    passthrough=no per-connection-classifier=both-addresses-and-ports:2/0
add action=mark-routing chain=prerouting disabled=yes dst-address=!10.0.0.0/8 \
    in-interface-list="!WAN" new-routing-mark="TTNET2 ROUTING" \
    passthrough=no per-connection-classifier=both-addresses-and-ports:2/1
19

Nope, if you should have removed the existing mangle rules, I would have said that. You cannot remove the ones added based on the video (mangle chains input and output) because this would prevent your VPN server from working.


Routes stay the same; I don’t know whether you use any routing rules as you haven’t posted them. And yes, nothing else to be added, nothing to be removed.


That’s strange. Post your complete configuration export following the hint in my automatic signature just below. Obfuscate also the usernames of the VPN users before posting.

20

My routes seem to be simple. It makes sense, I get it, but it doesn’t work for some weird reason.

reducted