Community discussions

MikroTik App
 
angboontiong
Forum Guru
Forum Guru
Topic Author
Posts: 1136
Joined: Fri Jan 16, 2009 9:59 am

NTH load balancing

Sat Oct 10, 2020 5:14 pm

Hi,
as search over the google, most NTH is years ago..
where can i get the latest and correct nth settings ?

and, what's the thing nth may not perform well or broken the connection ?
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: NTH load balancing

Sat Oct 10, 2020 8:37 pm

A while ago I created a write-up about NTH;
viewtopic.php?f=2&t=159174&p=781975
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Sat Oct 10, 2020 10:33 pm

Using Nth does give a more "even" bandwidth distribution as per this research paper: https://www.researchgate.net/publicatio ... t_Networks

But you'll need to take of the "sticky connections" for secure protocol sites/services like PayPal that does not tolerate source IP changing every second.

What I do is use PCC for TCP and UDP (QUIC protocol) on port 80/443 with "both addresses" as the classifier. It will not solve any app/site that uses a secure protocol other than the stated ones however the majority of them do not use anything else and I have never seen a broken app/website. But it's still possible.

Then simply use Nth for the rest of the traffic types. You will be able to get aggregated and particularly if it's something like Steam or Battle.net whereby it seems Multi-Path TCP is in place.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: NTH load balancing

Tue Oct 13, 2020 9:05 pm

Hi Darknate,
Can you post a generic config with the useful bits to show that split personality config on bandwidth load balancing........
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Wed Oct 14, 2020 3:49 am

Hi Darknate,
Can you post a generic config with the useful bits to show that split personality config on bandwidth load balancing........
I have been using this config for more than a month now. PayPal works, GPay, App Store, Play Store and full aggregated bandwidth when downloading/uploading (50/50 chance it would be single ISP however, which is obvious as this is not bonding)

Credits goes to that research paper though. I created this model based on the paper.

For HTTPS
#ForHTTPS###
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet dst-port=80,443 in-interface=LAN new-connection-mark=ISP1_conn passthrough=yes per-connection-classifier=both-addresses:2/0 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet dst-port=80,443 in-interface=LAN new-connection-mark=ISP2_conn passthrough=yes per-connection-classifier=both-addresses:2/1 protocol=tcp
For Nth for all traffic types
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP1_conn passthrough=yes nth=2,1
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP2_conn passthrough=yes nth=2,2
 
WeWiNet
Long time Member
Long time Member
Posts: 591
Joined: Thu Sep 27, 2018 4:11 pm

Re: NTH load balancing

Wed Oct 14, 2020 10:53 am

Hi Darknate,
Can you post a generic config with the useful bits to show that split personality config on bandwidth load balancing........
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP1_conn passthrough=yes nth=2,1
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP2_conn passthrough=yes nth=2,2
What is your address list not_in_internet? Why would you need this AND the dst-address-type=Not Local?
Have you any comparison between doing all via PCC and this PCC/Nth split? In the end you split by a factor of 2 in both cases, once based on IP address and the other via odd and even connection. If you have one client surly Nth is better, but once you have say 4-6 clients/local hosts, both should be similar performance.

PS: I use both address + port in PCC and have not seen any issue with any web page/connection (banking etc..).
But I can not comment on games, where this might be a problem (this is a don't care for me :-)
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Wed Oct 14, 2020 5:03 pm

Hi Darknate,
Can you post a generic config with the useful bits to show that split personality config on bandwidth load balancing........
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP1_conn passthrough=yes nth=2,1
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP2_conn passthrough=yes nth=2,2
What is your address list not_in_internet? Why would you need this AND the dst-address-type=Not Local?
Have you any comparison between doing all via PCC and this PCC/Nth split? In the end you split by a factor of 2 in both cases, once based on IP address and the other via odd and even connection. If you have one client surly Nth is better, but once you have say 4-6 clients/local hosts, both should be similar performance.

PS: I use both address + port in PCC and have not seen any issue with any web page/connection (banking etc..).
But I can not comment on games, where this might be a problem (this is a don't care for me :-)
Here's a full discussion on why that address list and !local is required together: viewtopic.php?f=13&t=167069

Dude, did you even bother to read the research paper I linked above? It's already been proven that Nth does a better job of bandwidth distribution than PCC due to the obvious reason that Nth is Per Packet whereas PCC is Per Connection.
https://help.mikrotik.com/docs/display/ ... +Balancing

If you think Nth 2,1 is the same as PCC 2,0, then you do not understand either or both them properly.
 
WeWiNet
Long time Member
Long time Member
Posts: 591
Joined: Thu Sep 27, 2018 4:11 pm

Re: NTH load balancing

Wed Oct 14, 2020 6:19 pm

Dude, did you even bother to read the research paper I linked above? It's already been proven that Nth does a better job of bandwidth distribution than PCC due to the obvious reason that Nth is Per Packet whereas PCC is Per Connection.
https://help.mikrotik.com/docs/display/ ... +Balancing

If you think Nth 2,1 is the same as PCC 2,0, then you do not understand either or both them properly.
Yes did read what you call a research paper....

In my sense, If you mark a connection than the connection is marked, basta! ( At least this is my understanding and the way it works in my setup).
Your rule of "mark unmarked connection" won't catch that connection anymore... regardless if PCC or Nth.
So once a connection goes onto ISP1 it stays there and only new connection has the chance to go somewhere else...
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: NTH load balancing

Wed Oct 14, 2020 6:34 pm

Dude, in the real world connection tracking ( or connection NTH ) is the best way for browsing the internet. NTH is predictable and a listener knows which connection is used next to sent the packet.

I am using this for my web browser and a new connection, even to same site, uses a 'unpredictable' path. When you have more than one user browsing the you could choose "both addresses and ports" and simpler is "dst address and port".

NTH is absolute better to use when you have multiple connections to an destination and don't use NTH or PCC as obfuscation.

General remark: why use 2,1 and 2,2 when you have 2,1 first. Think what is left after I caught half of the packet...yes the other half. So no need to also put 2,2 in there..it could then also 1,1 or 100000,100000 it makes no different if you take all packets left by the first line.
This way you can make your distribution a-symmetric. Example you have three connections 100Mbit/s 50Mbit/s 50Mbit/s:

First line: 2,1 (50%) then second line 2,1 (50%) and third line nothing (50%) or put better, all that is left and that is 25%.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Wed Oct 14, 2020 6:35 pm

Dude, did you even bother to read the research paper I linked above? It's already been proven that Nth does a better job of bandwidth distribution than PCC due to the obvious reason that Nth is Per Packet whereas PCC is Per Connection.
https://help.mikrotik.com/docs/display/ ... +Balancing

If you think Nth 2,1 is the same as PCC 2,0, then you do not understand either or both them properly.
Yes did read what you call a research paper....

In my sense, If you mark a connection than the connection is marked, basta! ( At least this is my understanding and the way it works in my setup).
Your rule of "mark unmarked connection" won't catch that connection anymore... regardless if PCC or Nth.
So once a connection goes onto ISP1 it stays there and only new connection has the chance to go somewhere else...
Which part of "per packet" don't you understand? We don't mark connections, we mark packets when using Nth.
 
WeWiNet
Long time Member
Long time Member
Posts: 591
Joined: Thu Sep 27, 2018 4:11 pm

Re: NTH load balancing

Wed Oct 14, 2020 6:47 pm

Why are you agressive? This is a forum for help and interaction.

I only see this code from you:
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP1_conn passthrough=yes nth=2,1
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP2_conn passthrough=yes nth=2,2
With my limited knowledge on ROS I believe this marks the connection "action=mark connection". is this wrong?
If you would have action "mark packet" I would agree, but you don't do that here...
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Wed Oct 14, 2020 8:07 pm

Why are you agressive? This is a forum for help and interaction.

I only see this code from you:
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP1_conn passthrough=yes nth=2,1
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-type=!local dst-address-list=!not_in_internet in-interface=bridge new-connection-mark=ISP2_conn passthrough=yes nth=2,2
With my limited knowledge on ROS I believe this marks the connection "action=mark connection". is this wrong?
If you would have action "mark packet" I would agree, but you don't do that here...
You're right. But Nth is NOT PCC. When we mark-connection using Nth, it marks the connection based on the Nth classier which is more random (more deeper) as it's per packet (of that particular unmarked connection), hence increasing the chances that the connection to passthrough to the next mangle rule.

But if we did use mark-packets instead, it would definitely increases the chances of the packets being passthrough to the next mangle rule more so than mark connection.

Thanks to you, I've now set up using mark-packet, I'll test for a few days, let's see how it goes.
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: NTH load balancing

Wed Oct 14, 2020 9:40 pm

When we mark-connection using Nth, it marks the connection based on the Nth classier which is more random (more deeper) as it's per packet (of that particular unmarked connection), hence increasing the chances that the connection to passthrough to the next mangle rule.
A connection is a connection and a packet is a packet. You can't mix those so Nth is not better than PCC when using the divider to control connection distribution.

I have multiple connections and tested it eith Nth and both-adresses-ports for a extensive period (weeks). Nth distributes perfect, but is predictable and boyh-addresses-port is very close and has obfuscation properties.

This forum for example does not allow obfuscation when being logged in. So it on the fixed addres-list and I wil not use a different source address while I am here.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Wed Oct 14, 2020 10:36 pm

@DarkNate, I'm afraid there may be just some confusion of terms.

First, know your audience - many people here deal only with the typical "home router with two ISPs" case, where the ultimate public IP on each WAN is different, so a real per-packet (means per-mid-connection-packet) load distribution can only break things, as packets routed via the "wrong" path will not be delivered at all or will be ignored at recipient side.

Even at ISP level where NAT is not an issue, sending different packets of the same TCP session via different paths is not preferred, as rumour has it that some TCP stacks don't deal well with packets arriving in shuffled order, which occurs frequently where multiple paths exist between the endpoints.

So as an ISP, whilst you'll get a smoother distribution of the traffic across the WANs if routing marks are assigned per-packet, the overall throughput may suffer if the TCP senders retransmit because they get indication about missing parts of the TCP stream from the recipients before the missing parts arrive through the longer path. Maybe this is a fear of the past, but protocols at various layers do include means specifically intended to address this aspect.

If you mean per-packet choice of WAN in terms that you use nth matching to assign a connection-mark to the initial packet of each new connection, and thus to choose the WAN for the connection as a whole, as your mangle rules suggest, what you effectively get is effectively a per-connection distribution, and the only difference to per-connection-classifier is that the nth matcher itself doesn't care about any property of the packet, whereas for per-connection-classifier you can specify which fields of the packet header will be used as a base to the choice. So if you stick with just source-address, the same LAN host will always use the same WAN; if you allow also source-port to be used for the decision, you'll get about the same distribution like with nth, because the client-side ports chosen for TCP connections are more or less random.

The cited paper doesn't actually compare nth to any other load distribution method in particular - in fact I was unable to find in there to what previous configuration the benefits of nth were compared. As there are several WANs in use and there was some traffic on all of them, I deduct there must have been something that was distributing the traffic among all of them, and as the "before" results were so much worse, I assume the previous configuration must have been terribly broken. Plus there are gems like "it can be seen that Nth load balancing has successfully spread the same packet on all three interfaces (sic!)" which I hope is just a typing mistake, not the way how the research team actually understands the area.

So I'm all for giving a conservative treatment (per-connection-classifier with source-address alone) to connections which can potentially suffer application layer issues if the same client establishes connections to the same server from different public IPs (as seen by the server), but I don't believe there is any difference whether you use nth or per-connection-classifier with a more random component (src-port) enabled for the rest.


Also, can you please re-word the
increasing the chances that the connection to passthrough to the next mangle rule.
statement?

I don't understand why increasing a chance to pass to the next mangle rule should be anything positive. The goal is to choose the WAN for the first packet of each connection (any choice made later just breaks things), and to let each packet pass as few rules as inevitable for the intended purpose.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Thu Oct 15, 2020 12:45 am

@DarkNate, I'm afraid there may be just some confusion of terms.

First, know your audience - many people here deal only with the typical "home router with two ISPs" case, where the ultimate public IP on each WAN is different, so a real per-packet (means per-mid-connection-packet) load distribution can only break things, as packets routed via the "wrong" path will not be delivered at all or will be ignored at recipient side.

Even at ISP level where NAT is not an issue, sending different packets of the same TCP session via different paths is not preferred, as rumour has it that some TCP stacks don't deal well with packets arriving in shuffled order, which occurs frequently where multiple paths exist between the endpoints.

So as an ISP, whilst you'll get a smoother distribution of the traffic across the WANs if routing marks are assigned per-packet, the overall throughput may suffer if the TCP senders retransmit because they get indication about missing parts of the TCP stream from the recipients before the missing parts arrive through the longer path. Maybe this is a fear of the past, but protocols at various layers do include means specifically intended to address this aspect.

If you mean per-packet choice of WAN in terms that you use nth matching to assign a connection-mark to the initial packet of each new connection, and thus to choose the WAN for the connection as a whole, as your mangle rules suggest, what you effectively get is effectively a per-connection distribution, and the only difference to per-connection-classifier is that the nth matcher itself doesn't care about any property of the packet, whereas for per-connection-classifier you can specify which fields of the packet header will be used as a base to the choice. So if you stick with just source-address, the same LAN host will always use the same WAN; if you allow also source-port to be used for the decision, you'll get about the same distribution like with nth, because the client-side ports chosen for TCP connections are more or less random.

The cited paper doesn't actually compare nth to any other load distribution method in particular - in fact I was unable to find in there to what previous configuration the benefits of nth were compared. As there are several WANs in use and there was some traffic on all of them, I deduct there must have been something that was distributing the traffic among all of them, and as the "before" results were so much worse, I assume the previous configuration must have been terribly broken. Plus there are gems like "it can be seen that Nth load balancing has successfully spread the same packet on all three interfaces (sic!)" which I hope is just a typing mistake, not the way how the research team actually understands the area.

So I'm all for giving a conservative treatment (per-connection-classifier with source-address alone) to connections which can potentially suffer application layer issues if the same client establishes connections to the same server from different public IPs (as seen by the server), but I don't believe there is any difference whether you use nth or per-connection-classifier with a more random component (src-port) enabled for the rest.


Also, can you please re-word the
increasing the chances that the connection to passthrough to the next mangle rule.
statement?

I don't understand why increasing a chance to pass to the next mangle rule should be anything positive. The goal is to choose the WAN for the first packet of each connection (any choice made later just breaks things), and to let each packet pass as few rules as inevitable for the intended purpose.
I linked the wrong paper, here's the right one: https://www.ijcnis.org/index.php/ijcnis ... /view/4340

I'm not sure what "Nth" does in "Mark connection" if not similar to what PCC does.

However, the reason why we pass it to the next mangle rule is to load balance? We have two ISPs for example and would like a 50/50 split? Using "Mark Packet" + Nth instead of Mark connection+Nth randomises things even further.

Regarding TCP. Multipath TCP is more widespread than you'd think. Cisco has guidelines on how to allow Multipath TCP through: https://www.cisco.com/c/en/us/support/d ... cp-00.html

Now based on my observations, Battle.net, Steam, Epic Games launcher all establishes multiple TCP connections during game downloads, all of which are able to take advantage of my load balancing setup to get fully aggregated bandwidth, the closest thing to actual bonding.

I'm using the following right now for testing purposes. If Banking works, the rest should likely work too, nothing is more security oriented than banking apps and sites, so far I tested PayPal, GPay, Local banks even. Nothing breaks yet... Gaming works fine too, fully open NAT on CoD Warzone.

I have two ISPs whereby it's basically 100% of traffic divided by 3 resulting in 33.33 and I sent 66/67% to ISP2, 33% to ISP1. Why? Because ISP2 has no data caps.
add comment="Packet marking for Nth" action=mark-packet chain=prerouting packet-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,1 passthrough=yes
add action=mark-packet chain=prerouting packet-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,2 passthrough=yes
add action=mark-packet chain=prerouting packet-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP1_conn nth=3,3 passthrough=yes

add comment="Routing mark for Nth" action=mark-routing chain=prerouting packet-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=no
add comment="Routing mark for Nth" action=mark-routing chain=prerouting packet-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Thu Oct 15, 2020 10:49 am

Well, as this topic was a bit emotional throughout its history, let me be a bit emotional too.

Yes, this other paper is much better, but still there are two points:
  • the apparently best dynamic load distribution method (nth) gives astounding 101.5% of the download throughput of the apparently worst one (ECMP) - this is a statistical error to me, given that they've tested each method for 40 (consecutive?) days, and there is no information on what was the test pattern, i.e. how comparable these 40 day intervals actually were
  • the results of the failover tests show that the nth was used for per-connection distribution, not per-packet one - otherwise the connection recovery after squelching the primary uplink would have taken much less time than with the other methods. So it doesn't prove any advantage of per-packet distribution (and there is none if there is NAT).
So for me, neither this paper is a sufficient argument to start redoing everything everywhere to nth.

Regarding multipath TCP, I happily admit it may be more widespread than I think, but RFC 8684 doesn't suggest that it would be able to deal with packets belonging to a particular TCP subflow to come from any remote socket - there is no common session ID which would be part of the TCP header in every single mid-connection packet, the relationship of a packet to a member flow of the MPTCP session is determined based on the (remote IP and port, local IP and port) tuple as it is with plain TCP. The latter also means that an endpoint cannot detect the availability of multiple addresses on the remote one autonomously, just by seeing packets from it coming from different remote IP and port, as it has no means to identify them as ones belonging to the same MPTCP session. It must always be the endpoint itself that announces the availability of additional interfaces (with distinct IP addresses) on its side. So for a mid-path router, a MPTCP session is just several independent TCP sessions not related to each other. There would have to be a complex store-and-forward "helper" on the router to allow the connection to use MPTCP between router's WAN and the server whilst the connection from the client to the router would be a plain TCP one.

Regarding use of packet-mark rather than connection-mark as a basis for routing-mark assignment, yes, this way you really create per-mid-connection-packet distribution, so if there is NAT between the LAN clients and the remote destinations, those packets which do not take the same path like the initial one of their connection get lost. So the traffic may be well distributed among the WANs, but about 1/2 of the traffic distributed this way are retransmissions.

And instead of assigning a packet-mark first and later translating it to routing-mark, you may assign the routing-mark straight away, the outcome will be the same but it will cost less CPU.

Last, dst-address-type=local only matches to own IP addresses of the router, not to all IP addresses from the connected subnets. So this match condition doesn't serve the intended purpose in your rules - routing of packets for the router itself is not controlled by routing-mark anyway.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Thu Oct 15, 2020 3:37 pm

Well, as this topic was a bit emotional throughout its history, let me be a bit emotional too.

Yes, this other paper is much better, but still there are two points:
  • the apparently best dynamic load distribution method (nth) gives astounding 101.5% of the download throughput of the apparently worst one (ECMP) - this is a statistical error to me, given that they've tested each method for 40 (consecutive?) days, and there is no information on what was the test pattern, i.e. how comparable these 40 day intervals actually were
  • the results of the failover tests show that the nth was used for per-connection distribution, not per-packet one - otherwise the connection recovery after squelching the primary uplink would have taken much less time than with the other methods. So it doesn't prove any advantage of per-packet distribution (and there is none if there is NAT).
So for me, neither this paper is a sufficient argument to start redoing everything everywhere to nth.

Regarding multipath TCP, I happily admit it may be more widespread than I think, but RFC 8684 doesn't suggest that it would be able to deal with packets belonging to a particular TCP subflow to come from any remote socket - there is no common session ID which would be part of the TCP header in every single mid-connection packet, the relationship of a packet to a member flow of the MPTCP session is determined based on the (remote IP and port, local IP and port) tuple as it is with plain TCP. The latter also means that an endpoint cannot detect the availability of multiple addresses on the remote one autonomously, just by seeing packets from it coming from different remote IP and port, as it has no means to identify them as ones belonging to the same MPTCP session. It must always be the endpoint itself that announces the availability of additional interfaces (with distinct IP addresses) on its side. So for a mid-path router, a MPTCP session is just several independent TCP sessions not related to each other. There would have to be a complex store-and-forward "helper" on the router to allow the connection to use MPTCP between router's WAN and the server whilst the connection from the client to the router would be a plain TCP one.

Regarding use of packet-mark rather than connection-mark as a basis for routing-mark assignment, yes, this way you really create per-mid-connection-packet distribution, so if there is NAT between the LAN clients and the remote destinations, those packets which do not take the same path like the initial one of their connection get lost. So the traffic may be well distributed among the WANs, but about 1/2 of the traffic distributed this way are retransmissions.

And instead of assigning a packet-mark first and later translating it to routing-mark, you may assign the routing-mark straight away, the outcome will be the same but it will cost less CPU.

Last, dst-address-type=local only matches to own IP addresses of the router, not to all IP addresses from the connected subnets. So this match condition doesn't serve the intended purpose in your rules - routing of packets for the router itself is not controlled by routing-mark anyway.
Regarding the Nth everywhere thing, well I'm not suggesting the whole world to use it, now am I? Did I? Ever? No I did not.

Regarding Multipath TCP, you're right, ROS does not have any traversal mechanism to help manage Multipath TCP unlike Cisco. There are threads on the forum requesting the feature. Which is why I'm leaving MikroTik in the gutter as soon as I can afford to build an OP pfSense/VyOS box or buy a Cisco edge router.

I'm testing the mark-packet for Nth right now, I am not seeing any performance degradation or packet loss, I do see performance improvements. If half of the traffic is a wastage of "retransmission" wouldn't I be seeing packet loss and degradation?

Regarding !local, here's the full discussion: viewtopic.php?f=13&t=167069

The advantage of !local is it also implies any IPs inside IP>Address that are assigned to any of the interfaces. It helps avoid sending traffic destined to specific ISP gateway through mangle rules, and instead uses the main routing table correctly. You can easily replicate this. Set up LB without !local, try to ping both gateways of the ISPs, if you're lucky that LB sent it to the right ISP, it'll work else it won't. With !local traceroutes, ping tests to local gateway works 100%. I have two IPs assigned to each WAN port. One is public IP, one is local IP from the ONT on my GPON and EPON connections. So !local is very critical in my setup. I can access local gateway of both ISPs and the ONT's on each ethernet port without issues thanks to !local.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Thu Oct 15, 2020 6:09 pm

Regarding the Nth everywhere thing, well I'm not suggesting the whole world to use it, now am I? Did I? Ever? No I did not.
I've announced in advance I'm going to be a bit emotional ;)

So the neutral form of the same statement would have been that the results haven't proven any significant advantage of nth as compared to the other load distribution methods.

Regarding Multipath TCP, you're right, ROS does not have any traversal mechanism to help manage Multipath TCP unlike Cisco.
But that's a misunderstanding. The whole material from Cisco just says that:
  • Cisco devices not involved in security enforcement are completely MPTCP agnostic - they don't interfere with it and it doesn't complicate their lives
  • Cisco devices involved in security enforcement and doing DPI (payload analysis) may either prevent MPTCP from being negotiated between the endpoints by wiping out the negotiation information from the packets, or they will be unable to do the DPI because they will only see part of the traffic as the other part may bypass them completely.
There isn't a single word suggesting that any Cisco router could act as a TCP proxy adding MPTCP capability to a LAN device which itself has a single interface.

Which is why I'm leaving MikroTik in the gutter as soon as I can afford to build an OP pfSense/VyOS box or buy a Cisco edge router.
I won't convince anyone to stick with Mikrotik, but currently no Cisco edge router will act as a MPTCP proxy, see above. No idea regarding pfSense and/or VyOS. Other than that, it doesn't sound too complex to implement; the amount of CPU required is another question, and the real demand for this feature is yet another one (how many % of SOHO users have multiple connections, and how many servers out there actually support it. Bear in mind that QUIC has nothing to do with MPTCP although it may possibly be using multi-path as well).

I'm testing the mark-packet for Nth right now, I am not seeing any performance degradation or packet loss, I do see performance improvements. If half of the traffic is a wastage of "retransmission" wouldn't I be seeing packet loss and degradation?
Sure you would. So either you don't have NAT between the LAN host and the remote servers, or something else in your configuration prevents the per-packet distribution of the traffic from actually working, or maybe the NAT is common for both WANs and the original source address makes it all the way to that NAT, no idea. I'd have to see the complete configuration to say whether the reason is inside or outside the Mikrotik.

If the remote endpoint can see the packets to come from the same source address and port regardless which WAN they've been sent through, the per-packet distribution works fine. If the source address or port or both is different for some mid-connection packets than those of the initial packet of the connection, those differing packets are dropped at the remote endpoint. There's no way around this point.

Regarding !local, here's the full discussion: viewtopic.php?f=13&t=167069
Yes, and in the first response marked as Solved, @Sob says the same what I say (and then repeats it multiple times).

The advantage of !local is it also implies any IPs inside IP>Address that are assigned to any of the interfaces.
Correct, again the same what I say, except the "also" part. It's "only".

It helps avoid sending traffic destined to specific ISP gateway through mangle rules, and instead uses the main routing table correctly. You can easily replicate this. Set up LB without !local, try to ping both gateways of the ISPs, if you're lucky that LB sent it to the right ISP, it'll work else it won't. With !local traceroutes, ping tests to local gateway works 100%. I have two IPs assigned to each WAN port. One is public IP, one is local IP from the ONT on my GPON and EPON connections. So !local is very critical in my setup. I can access local gateway of both ISPs and the ONT's on each ethernet port without issues thanks to !local.
OK, now I maybe start to understand. When you ping the WAN gateway from a LAN host, and the echo request reaches the WAN gateway "somehow" after getting src-NATed by the router to one of its WAN IPs, the response from the WAN gateway arrives with that WAN IP as destination. So while handled by chain prerouting of mangle, which handles the packets before they get "un-src-nated", it does match dst-address-type=local, and if the prerouting rule assigns (directly or indirectly via connection-mark) a routing-mark to it, this response is sent via the corresponding WAN gateway instead of using the routing table main to reach the LAN host.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Thu Oct 15, 2020 7:31 pm

Regarding the Nth everywhere thing, well I'm not suggesting the whole world to use it, now am I? Did I? Ever? No I did not.
I've announced in advance I'm going to be a bit emotional ;)

So the neutral form of the same statement would have been that the results haven't proven any significant advantage of nth as compared to the other load distribution methods.

Regarding Multipath TCP, you're right, ROS does not have any traversal mechanism to help manage Multipath TCP unlike Cisco.
But that's a misunderstanding. The whole material from Cisco just says that:
  • Cisco devices not involved in security enforcement are completely MPTCP agnostic - they don't interfere with it and it doesn't complicate their lives
  • Cisco devices involved in security enforcement and doing DPI (payload analysis) may either prevent MPTCP from being negotiated between the endpoints by wiping out the negotiation information from the packets, or they will be unable to do the DPI because they will only see part of the traffic as the other part may bypass them completely.
There isn't a single word suggesting that any Cisco router could act as a TCP proxy adding MPTCP capability to a LAN device which itself has a single interface.

Which is why I'm leaving MikroTik in the gutter as soon as I can afford to build an OP pfSense/VyOS box or buy a Cisco edge router.
I won't convince anyone to stick with Mikrotik, but currently no Cisco edge router will act as a MPTCP proxy, see above. No idea regarding pfSense and/or VyOS. Other than that, it doesn't sound too complex to implement; the amount of CPU required is another question, and the real demand for this feature is yet another one (how many % of SOHO users have multiple connections, and how many servers out there actually support it. Bear in mind that QUIC has nothing to do with MPTCP although it may possibly be using multi-path as well).

I'm testing the mark-packet for Nth right now, I am not seeing any performance degradation or packet loss, I do see performance improvements. If half of the traffic is a wastage of "retransmission" wouldn't I be seeing packet loss and degradation?
Sure you would. So either you don't have NAT between the LAN host and the remote servers, or something else in your configuration prevents the per-packet distribution of the traffic from actually working, or maybe the NAT is common for both WANs and the original source address makes it all the way to that NAT, no idea. I'd have to see the complete configuration to say whether the reason is inside or outside the Mikrotik.

If the remote endpoint can see the packets to come from the same source address and port regardless which WAN they've been sent through, the per-packet distribution works fine. If the source address or port or both is different for some mid-connection packets than those of the initial packet of the connection, those differing packets are dropped at the remote endpoint. There's no way around this point.

Regarding !local, here's the full discussion: viewtopic.php?f=13&t=167069
Yes, and in the first response marked as Solved, @Sob says the same what I say (and then repeats it multiple times).

The advantage of !local is it also implies any IPs inside IP>Address that are assigned to any of the interfaces.
Correct, again the same what I say, except the "also" part. It's "only".

It helps avoid sending traffic destined to specific ISP gateway through mangle rules, and instead uses the main routing table correctly. You can easily replicate this. Set up LB without !local, try to ping both gateways of the ISPs, if you're lucky that LB sent it to the right ISP, it'll work else it won't. With !local traceroutes, ping tests to local gateway works 100%. I have two IPs assigned to each WAN port. One is public IP, one is local IP from the ONT on my GPON and EPON connections. So !local is very critical in my setup. I can access local gateway of both ISPs and the ONT's on each ethernet port without issues thanks to !local.
OK, now I maybe start to understand. When you ping the WAN gateway from a LAN host, and the echo request reaches the WAN gateway "somehow" after getting src-NATed by the router to one of its WAN IPs, the response from the WAN gateway arrives with that WAN IP as destination. So while handled by chain prerouting of mangle, which handles the packets before they get "un-src-nated", it does match dst-address-type=local, and if the prerouting rule assigns (directly or indirectly via connection-mark) a routing-mark to it, this response is sent via the corresponding WAN gateway instead of using the routing table main to reach the LAN host.
I can't bother to quote everything cleanly, so I'll reply in paragraph wise.

Yeah, I need to read more about Multipath TCP, my understanding in it is basic at best.

Yeah, it's simple. !local means any traffic destined to anything inside IP>Address is not marked for the mangle rules. Hence uses main routing table.

Regarding my network config. It's dead simple. Two ISPs, both PPPoE clients. Both public IPs. One is dynamic hence I use masquerade, one is static hence I use srcnat.
The remote endpoints sees both IPs depending on the Nth's chances of what goes were, one refresh it's ISP1, next refresh it's ISP2.
So I'm still confused as to how there's no packet-loss, broken config. Even traceroutes from LAN to remote sites works fine and shows their corresponding routes based on whichever ISP was assigned. I use MTR and see 0 packet loss. And gaming, works!? What?
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no
add action=mark-connection chain=prerouting comment="66/33 Split" connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=\
    ISP2_conn passthrough=yes per-connection-classifier=both-addresses:3/0 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/1 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/2 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/0 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/1 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/2 port=80,443 protocol=udp
    
add action=mark-packet chain=prerouting comment="Packet marking for Nth" dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,1 \
    packet-mark=no-mark passthrough=yes
add action=mark-packet chain=prerouting dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,2 packet-mark=no-mark passthrough=yes
add action=mark-packet chain=prerouting dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP1_conn nth=3,3 packet-mark=no-mark passthrough=yes

add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting comment="Routing mark for Nth" in-interface-list=LAN new-routing-mark=to_ISP1 packet-mark=ISP1_conn passthrough=no
add action=mark-routing chain=prerouting comment="Routing mark for Nth" in-interface-list=LAN new-routing-mark=to_ISP2 packet-mark=ISP2_conn passthrough=no
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 out-interface=pppoe-out1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 out-interface=pppoe-out2 passthrough=no
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Thu Oct 15, 2020 7:38 pm

I can't bother to quote everything cleanly, so I'll reply in paragraph wise.
You don't have to quote at all, I suggest you try it sometimes.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Thu Oct 15, 2020 8:15 pm

I can't bother to quote everything cleanly, so I'll reply in paragraph wise.
You don't have to quote at all, I suggest you try it sometimes.
No forum rule is violated by quoting anything dude
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Thu Oct 15, 2020 8:16 pm

So I'm still confused as to how there's no packet-loss, broken config. Even traceroutes from LAN to remote sites works fine and shows their corresponding routes based on whichever ISP was assigned. I use MTR and see 0 packet loss. And gaming, works!? What?
So what happens is the following (tracking the establishment of a connection which is eligible for handling by the nth rules, not the per-connection-classifier ones):
  • the initial packet from the LAN host to the remote server
    • is caught by one of the three action=mark-packet ... nth=3,x ... rules and gets a packet-mark, let's say ISP2_conn
    • as the rule above has passthrough=yes, the packet is next caught by the action=mark-routing ... in-interface-list=LAN packet-mark=ISP2_conn rule and gets a routing-mark to_ISP2
    • as the rule above has passthrough=no, this is where the mangling of this packet in prerouting ends, so no connection-mark is assigned to the connection during processing of this packet
    • thanks to the routing-mark being assigned to the packet, the packet gets routed through ISP2's gateway to the destination and src-nated to ISP2's public IP
  • the first response packet from the remote server
    • is caught by the rule action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn, so a connection-mark ISP2_conn is assigned to the connection
    • as the rule above has passthrough=no, this is where the mangling of this packet in prerouting ends
    • the packet then gets un-src-nated and routed to the LAN host
  • the second (and every subsequent) packet from the LAN host to the remote server
    • is caught by the rule action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 and gets a routing-mark to_ISP2
    • as the rule above has passthrough=no, this is where the mangling of this packet in prerouting ends, so it ends up with that routing-mark

So no packet except the initial one of each connection ever suffers any nth handling.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Thu Oct 15, 2020 9:02 pm

So I'm still confused as to how there's no packet-loss, broken config. Even traceroutes from LAN to remote sites works fine and shows their corresponding routes based on whichever ISP was assigned. I use MTR and see 0 packet loss. And gaming, works!? What?
So what happens is the following (tracking the establishment of a connection which is eligible for handling by the nth rules, not the per-connection-classifier ones):
  • the initial packet from the LAN host to the remote server
    • is caught by one of the three action=mark-packet ... nth=3,x ... rules and gets a packet-mark, let's say ISP2_conn
    • as the rule above has passthrough=yes, the packet is next caught by the action=mark-routing ... in-interface-list=LAN packet-mark=ISP2_conn rule and gets a routing-mark to_ISP2
    • as the rule above has passthrough=no, this is where the mangling of this packet in prerouting ends, so no connection-mark is assigned to the connection during processing of this packet
    • thanks to the routing-mark being assigned to the packet, the packet gets routed through ISP2's gateway to the destination and src-nated to ISP2's public IP
  • the first response packet from the remote server
    • is caught by the rule action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn, so a connection-mark ISP2_conn is assigned to the connection
    • as the rule above has passthrough=no, this is where the mangling of this packet in prerouting ends
    • the packet then gets un-src-nated and routed to the LAN host
  • the second (and every subsequent) packet from the LAN host to the remote server
    • is caught by the rule action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 and gets a routing-mark to_ISP2
    • as the rule above has passthrough=no, this is where the mangling of this packet in prerouting ends, so it ends up with that routing-mark

So no packet except the initial one of each connection ever suffers any nth handling.
So how does the above give aggregated bandwidth in downloads/uploads? I see doubled bandwidth across the board.

So as per your own explanation, my Nth setup does NOT cause degradation/packet loss and as per the aggregated bandwidth outcome, it is in fact better than PCC in regards to bandwidth.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Thu Oct 15, 2020 9:20 pm

OT: (Correct, you can quote anything you want, it was just suggestion, because I don't think it has any added value whatsoever. But if you feel otherwise...)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Thu Oct 15, 2020 9:23 pm

So how does the above give aggregated bandwidth in downloads/uploads? I see doubled bandwidth across the board.
Like any other dynamic load distribution method. If you make a throughput test using a single session (=connection), it will show a single uplink bandwidth. speedtest.net, as well as other bandwidth tests available on the web, use multi-session tests by default, so they show you the summary bandwidth.

So as per your own explanation, my Nth setup does NOT cause degradation/packet loss and as per the aggregated bandwidth outcome, it is in fact better than PCC in regards to bandwidth.
I've never said that per-connection traffic distribution controlled using nth causes degradation/packet loss. But I did say multiple times, and still maintain, that your setup does not execute a per-packet traffic distribution as you believe and use as an argument why nth is superior to other methods of traffic distribution.

If you modify your mangle rules in such a way that the nth really starts doing per-packet traffic distribution, you'll get the degradation caused by packet loss I am talking about all the time.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: NTH load balancing

Thu Oct 15, 2020 9:52 pm

Awesome, now I can put Nth where the sun don't shine and not worry my one brain cell with it anymore. ( I like brevity - NTH sucks, PCC good ;-) )
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Thu Oct 15, 2020 10:25 pm

It's not so easy. First, there's not just one PCC, you have different options.

If you use both-addresses, it's good for compatibility with servers that don't like clients changing their addresses all the time, but distribution will be worse, because each combination of client-server will always use only one same link.

If you use both-addresses-and-ports, then distribution will be better, because source port is different for each connection, so in long average it will split evenly. But some servers won't like it, because you'll be connecting to them from different addresses.

Other PCC options don't differ much, their behaviour is always close to one of described oned.

Nth will be pretty much like PCC's both-addresses-and-ports. It's always one connection to one link and next one to the other (for two links), which will guarantee 50:50 split, but each connection uses different amount of traffic, so that won't be exactly equal.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Thu Oct 15, 2020 10:31 pm

So how does the above give aggregated bandwidth in downloads/uploads? I see doubled bandwidth across the board.
Like any other dynamic load distribution method. If you make a throughput test using a single session (=connection), it will show a single uplink bandwidth. speedtest.net, as well as other bandwidth tests available on the web, use multi-session tests by default, so they show you the summary bandwidth.

So as per your own explanation, my Nth setup does NOT cause degradation/packet loss and as per the aggregated bandwidth outcome, it is in fact better than PCC in regards to bandwidth.
I've never said that per-connection traffic distribution controlled using nth causes degradation/packet loss. But I did say multiple times, and still maintain, that your setup does not execute a per-packet traffic distribution as you believe and use as an argument why nth is superior to other methods of traffic distribution.

If you modify your mangle rules in such a way that the nth really starts doing per-packet traffic distribution, you'll get the degradation caused by packet loss I am talking about all the time.
So how do we make it per-packet when I'm already using mark-parket+Nth?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Thu Oct 15, 2020 10:46 pm

So how do we make it per-packet when I'm already using mark-parket+Nth?
Since it is only needed for a test, just add dst-port=80,443 or packet-mark=no-mark to the two rules which assign a routing-mark based on connection-mark - i.e. the action=mark-routing chain=prerouting connection-mark=ISPx_conn in-interface-list=LAN new-routing-mark=to_ISPx passthrough=no ones. This will make these rules ignore the packets eligible for the nth treatment, so these packets will reach the next two rules translating the packet-mark into a routing-mark. Or simply move the latter pair of rules before (above) the former pair, it will have the same effect.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 12:44 am

So how do we make it per-packet when I'm already using mark-parket+Nth?
Since it is only needed for a test, just add dst-port=80,443 or packet-mark=no-mark to the two rules which assign a routing-mark based on connection-mark - i.e. the action=mark-routing chain=prerouting connection-mark=ISPx_conn in-interface-list=LAN new-routing-mark=to_ISPx passthrough=no ones. This will make these rules ignore the packets eligible for the nth treatment, so these packets will reach the next two rules translating the packet-mark into a routing-mark. Or simply move the latter pair of rules before (above) the former pair, it will have the same effect.
Why would I want 80 and 443 in Nth? I put them in PCC to avoid breaking HTTPS/Secure sites to begin with. Normal downloads/uploads do not use either ports. PCC uses connection marks and routing marks based the connection marks.

Either you don't download/upload much or you're ignorant right now. Like I keep telling you, Battle.net, Steam, Epic Games, IDM etc all gets aggregated bandwidth via Nth.

Your logic is flawed. My setup, does have dedicated routing marks just for the packet marks based on Nth. Unless you're assuming that the counters are zero? Which isn't? The Nth is clearly working with the following, which for God knows why, you keep denying that it does not.t
add comment="Packet marking for Nth" action=mark-packet chain=prerouting [b]packet-mark[/b]=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,1 passthrough=yes
add action=mark-packet chain=prerouting [b]packet-mark[/b]=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,2 passthrough=yes
add action=mark-packet chain=prerouting [b]packet-mark[/b]=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP1_conn nth=3,3 passthrough=yes

add comment="Routing mark for Nth" action=mark-routing chain=prerouting [b]packet-mark[/b]=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=no
add comment="Routing mark for Nth" action=mark-routing chain=prerouting [b]packet-mark[/b]=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Fri Oct 16, 2020 2:43 am

If you want to make sure that you have everything "right" for testing, then temporarily disable these two rules:
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no
If you've shown all you have, it should be enough. Or in other words, keep enabled only those mangle rules with in-interface-list=LAN. Then you'll have true per-packet Nth balancing for all ports except 80 and 443.

Btw, @sindy's way is even better, but you misunderstood it.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 9:45 am

I wonder which part of the firewall handling you are missing or misunderstand that prevents you from seeing the logic:
  • do you realize that once you assign a connection-mark to a connection while handling any of that connection's packets, regardless the direction, this connection-mark is then automatically assigned to every subsequent packet of the connection? In particular, that even the connection-mark assigned to a WAN->LAN packet of a connection then matches when handling subsequent LAN->WAN packets of the same connection?
  • do you realize that the rules are evaluated top to bottom until the first rule providing a final verdict matches, and action of every matching rule is executed? Some actions (like accept) always provide a final verdict, some never provide it (like log), and for some, this is controlled by the passthrough option.

So what you need is that LAN->WAN packets towards TCP or UDP ports 80 or 443 (for which a per-connection-classifier handling is intended) get their routing-mark based only on the connection-mark, whereas all the other LAN->WAN packets (for which the per-packet nth handling is intended) need to get their routing-mark only based on the packet-mark.

In your setup, there is first a group of mangle/prerouting rules which assigns either connection-marks or packet-marks depending on the packet "category", with passthrough=yes.
Next, there are two groups of rules, which translate either connection-mark or packet-mark to routing-mark. However, you don't take into account that while handling the WAN->LAN packets, you assign a connection-mark to every connection (including those intended to be handled only by the set of rules for the nth distribution), and that the LAN->WAN packet reaches the rules assigning routing-mark based on connection-mark first. As these rules currently match neither on port nor on absence of packet-mark, they handle also the packets you want them to ignore. As they have passthrough=no, this routing-mark assignment is the final verdict for those packets, and they never reach the rules assigning a routing-mark based on packet-mark which would rewrite the routing-mark with the intended one.

So as I wrote yesterday - you can make the rules assigning a routing-mark based on connection-mark more selective, by letting them check for the port or for the absence of packet-mark. Or you can put the rules assigning the routing-mark based on packet-mark before the ones matching on connection-mark - they only match on packets with a packet-mark assigned so they will ignore the packets without packet-mark. Or you can set passthrough=yes on the rules assigning the routing-mark based on connection-mark so that the subsequent rules could change the routing-mark. Or you can simply change the rules which currently assign a packet-mark to assign directly a routing-mark and give them passthrough=no.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 3:09 pm

If you want to make sure that you have everything "right" for testing, then temporarily disable these two rules:
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no
If you've shown all you have, it should be enough. Or in other words, keep enabled only those mangle rules with in-interface-list=LAN. Then you'll have true per-packet Nth balancing for all ports except 80 and 443.

Btw, @sindy's way is even better, but you misunderstood it.
Disabled temporarily. Downloads/Uploads are still aggregated just fine.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 3:11 pm

I wonder which part of the firewall handling you are missing or misunderstand that prevents you from seeing the logic:
  • do you realize that once you assign a connection-mark to a connection while handling any of that connection's packets, regardless the direction, this connection-mark is then automatically assigned to every subsequent packet of the connection? In particular, that even the connection-mark assigned to a WAN->LAN packet of a connection then matches when handling subsequent LAN->WAN packets of the same connection?
  • do you realize that the rules are evaluated top to bottom until the first rule providing a final verdict matches, and action of every matching rule is executed? Some actions (like accept) always provide a final verdict, some never provide it (like log), and for some, this is controlled by the passthrough option.

So what you need is that LAN->WAN packets towards TCP or UDP ports 80 or 443 (for which a per-connection-classifier handling is intended) get their routing-mark based only on the connection-mark, whereas all the other LAN->WAN packets (for which the per-packet nth handling is intended) need to get their routing-mark only based on the packet-mark.

In your setup, there is first a group of mangle/prerouting rules which assigns either connection-marks or packet-marks depending on the packet "category", with passthrough=yes.
Next, there are two groups of rules, which translate either connection-mark or packet-mark to routing-mark. However, you don't take into account that while handling the WAN->LAN packets, you assign a connection-mark to every connection (including those intended to be handled only by the set of rules for the nth distribution), and that the LAN->WAN packet reaches the rules assigning routing-mark based on connection-mark first. As these rules currently match neither on port nor on absence of packet-mark, they handle also the packets you want them to ignore. As they have passthrough=no, this routing-mark assignment is the final verdict for those packets, and they never reach the rules assigning a routing-mark based on packet-mark which would rewrite the routing-mark with the intended one.

So as I wrote yesterday - you can make the rules assigning a routing-mark based on connection-mark more selective, by letting them check for the port or for the absence of packet-mark. Or you can put the rules assigning the routing-mark based on packet-mark before the ones matching on connection-mark - they only match on packets with a packet-mark assigned so they will ignore the packets without packet-mark. Or you can set passthrough=yes on the rules assigning the routing-mark based on connection-mark so that the subsequent rules could change the routing-mark. Or you can simply change the rules which currently assign a packet-mark to assign directly a routing-mark and give them passthrough=no.
I did this: viewtopic.php?f=14&t=167416#p822889
Still works.

And for the last time, connection-marks are used only on 80/443. Everything else is Nth. Do you not know how Steam, Battle.net, Epic Games, IDM, Downloads/Uploads work or something? NONE of them uses port 80/443, which is where my Nth mark-packet rules comes in for those, all ports except 80/443 goes through the mark-packet NOT mark connection.

As for WAN to LAN, obviously that's per-connection. Why would I use Nth which is meant for load balancing traffic leaving LAN to do what? Load balance traffic coming from a specific LAN interface?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 3:26 pm

What is your native language? I feel as if you do not attempt to understand what I write. You believe you only use connection-mark for 80 and 443, but the reality is that you use it for all connections, most likely because you still can't understand how the connection-mark actually works.

With the rules you've posted last time, only the first packet of each connection is treated differently for 80/443 and other ports; all the subsequent LAN->WAN packets are treated based on connection-mark, and the fact that some of them get a packet-mark doesn't affect it. I've given you the reasons and several possibilities how to change that in my previous posts.

You can use /tool sniffer to see what really happens - you won't see the routing-marks there as they are not part of packet contents, but you will see that all packets of any given connection which doesn't go to 80 or 443 always use the same WAN all the time, i.e. they are not distributed per packet but per connection.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Fri Oct 16, 2020 3:41 pm

There's definitely some misunderstanding. Because if it's regular dual WAN config, i.e. two independent ISPs, then per-packet load balancing will work great with single-packet exchanges like DNS queries, but everything else will be absolutely terrible, if it will work at all.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 3:59 pm

There's definitely some misunderstanding. Because if it's regular dual WAN config, i.e. two independent ISPs, then per-packet load balancing will work great with single-packet exchanges like DNS queries, but everything else will be absolutely terrible, if it will work at all.
Sob my dude, I never once disagreed with your technical opinion on Nth. Regarding DNS, I use static policy routing. Everything else simply does NOT break yet.

You've seen my PCC/Nth combo config, it's dead simple, it works without breaking anything. The reason why I like it? Is 80/443 is PCC both addresses hence minimise chances of breaking sites/apps whatever, Nth gives me what I wanted, that is aggregated bandwidth. I've run thorough download/upload tests and 8/10 times (since we have split the traffic 3 way) I get aggregated bandwidth.

Take a look at what Sindy is trying to say, dude, he/she/it keeps thinking my clearly defined PCC rules for 80/443 somehow magically captures ALL traffic types. Which of course is easily disproven simply by logging my Nth rules and the Nth routing marks OR like you suggested to temporarily disable the PCC routing marks and see if Nth works.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 4:01 pm

What is your native language? I feel as if you do not attempt to understand what I write. You believe you only use connection-mark for 80 and 443, but the reality is that you use it for all connections, most likely because you still can't understand how the connection-mark actually works.

With the rules you've posted last time, only the first packet of each connection is treated differently for 80/443 and other ports; all the subsequent LAN->WAN packets are treated based on connection-mark, and the fact that some of them get a packet-mark doesn't affect it. I've given you the reasons and several possibilities how to change that in my previous posts.

You can use /tool sniffer to see what really happens - you won't see the routing-marks there as they are not part of packet contents, but you will see that all packets of any given connection which doesn't go to 80 or 443 always use the same WAN all the time, i.e. they are not distributed per packet but per connection.
It's simple as this. If my Nth does zero work, I shouldn't be getting aggregated bandwidth on ports excluding 80/443. But I do. And I checked with logging too on the routing marks/mark packets, they are working as intended.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 4:22 pm

Yeah, the packet counters are a good idea by the way.

Reset the counters on both the nth rules which assign packet-marks and the rules which translate packet-mark into routing-mark, then run some traffic eligible for nth handling, and compare the counters after that. They should be equal (every packet that gets a packet-mark by one rule should also get a routing-mark by the other one), but they will not be, because the packet-marked packets do not reach the packet-mark -> routing-mark translation rules.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 4:34 pm

Yeah, the packet counters are a good idea by the way.

Reset the counters on both the nth rules which assign packet-marks and the rules which translate packet-mark into routing-mark, then run some traffic eligible for nth handling, and compare the counters after that. They should be equal (every packet that gets a packet-mark by one rule should also get a routing-mark by the other one), but they will not be, because the packet-marked packets do not reach the packet-mark -> routing-mark translation rules.
Bruh... I'm laughing at this point. I'm telling you, anything not 80/443 is using the Nth's mark-packets+packet-marks' routing mark.

No they are NOT equal, exactly the SAME thing with the PCC's connection-marks+connection-marks' routing mark, they are NOT equal either.

But they are NOT zero. Why don't you try my config yourself instead of assuming things in your brain? Try it with the logging and see for yourself.
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out1 new-connection-mark=ISP1_conn passthrough=no
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no

add action=mark-connection chain=prerouting comment="66/33 Split" connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=\
    ISP2_conn passthrough=yes per-connection-classifier=both-addresses:3/0 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/1 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/2 port=80,443 protocol=tcp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/0 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP2_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/1 port=80,443 protocol=udp
add action=mark-connection chain=prerouting connection-mark=no-mark dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-connection-mark=ISP1_conn passthrough=\
    yes per-connection-classifier=both-addresses:3/2 port=80,443 protocol=udp

add action=mark-packet chain=prerouting comment="Packet marking for Nth" dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,1 \
    packet-mark=no-mark passthrough=yes
add action=mark-packet chain=prerouting dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP2_conn nth=3,2 packet-mark=no-mark passthrough=yes
add action=mark-packet chain=prerouting dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN new-packet-mark=ISP1_conn nth=3,3 packet-mark=no-mark passthrough=yes

add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 passthrough=no
add action=mark-routing chain=prerouting comment="Routing mark for Nth" in-interface-list=LAN new-routing-mark=to_ISP1 packet-mark=ISP1_conn passthrough=no
add action=mark-routing chain=prerouting comment="Routing mark for Nth" in-interface-list=LAN new-routing-mark=to_ISP2 packet-mark=ISP2_conn passthrough=no
add action=mark-routing chain=output connection-mark=ISP1_conn new-routing-mark=to_ISP1 out-interface=pppoe-out1 passthrough=no
add action=mark-routing chain=output connection-mark=ISP2_conn new-routing-mark=to_ISP2 out-interface=pppoe-out2 passthrough=no
Last edited by DarkNate on Fri Oct 16, 2020 4:38 pm, edited 1 time in total.
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Fri Oct 16, 2020 4:35 pm

You're missing few details. For start, your original config that you thought does per-packet Nth didn't really do that, and when @sindy explained in detail what happens, you didn't get it at all. You also misundertood his other suggestion. And his interest in your native language is not to make fun of your English, but because people tend to make similar mistakes (or use "specific style of English") based on their native language. And when he knows their native language, he can better understand what they mean. That's how good he is, listen to the man, it's worth it. :)
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 4:39 pm

You're missing few details. For start, your original config that you thought does per-packet Nth didn't really do that, and when @sindy explained in detail what happens, you didn't get it at all. You also misundertood his other suggestion. And his interest in your native language is not to make fun of your English, but because people tend to make similar mistakes (or use "specific style of English") based on their native language. And when he knows their native language, he can better understand what they mean. That's how good he is, listen to the man, it's worth it. :)
I reposted my config here: viewtopic.php?f=14&t=167416&p=823004#p823004

Now do you see any flaws whereby traffic escapes the packet-marking rules + routing marking rules (based on packet marks)?

Also this explanation: viewtopic.php?p=822800#p822800
Assumes a single TCP stream/thread. We're talking multi-threaded downloads/uploads which is what I make use of to get aggregated bandwidth. Which is what most popular applications also make use of.

Obviously if it's single threaded/stream, it will always only use a single WAN. I know that. Did any of you thought about multi-threaded?

Edit: When I do download files manually outside applications, I make use of a miniumm of 16 threads and a maximum of 32 threads. All of which goes through Nth not the 80/443 PCC/connection-mark routing marks. What person in their right mind would download in single thread? Even Chrome natively uses a minimum of 2-4.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 5:02 pm

OK, so let me present another idea how to explain it.

When I say "per connection distribution of traffic", it does not mean the same as "traffic distribution controlled by per-connection-classifier". It simply means that all packets of the same connection use the same WAN, no matter what rule or matcher within a rule was used to stick that connection with that WAN.

When I say "per packet distribution of traffic", it means to me that each packet of the same connection may use another WAN.

Do you agree with this classification of what is per-connection distribution of traffic and what is per-packet distribution of traffic?
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 5:27 pm

OK, so let me present another idea how to explain it.

When I say "per connection distribution of traffic", it does not mean the same as "traffic distribution controlled by per-connection-classifier". It simply means that all packets of the same connection use the same WAN, no matter what rule or matcher within a rule was used to stick that connection with that WAN.

When I say "per packet distribution of traffic", it means to me that each packet of the same connection may use another WAN.

Do you agree with this classification of what is per-connection distribution of traffic and what is per-packet distribution of traffic?
I agree.

So how am I able to achieve aggregated bandwidth with the above config?
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Fri Oct 16, 2020 5:41 pm

The last config, it's still the same one, isn't it? It was already explained by @sindy, but once more, in the form of simple example:

- device in LAN wants to connect to 1.2.3.4:666 in internet
- PCC rules won't touch initial packet to port 666, because they are only for 80 and 443
- Nth marks packet with ISP2_conn packet mark (could have been ISP1_conn as well, but for this one it happens to be ISP2_conn)
- packet goes out via pppoe-out2, reaches target server and it responds
- response packet arrives to your router on pppoe-out2
- there will be regular processing like for every other packet, in this case it's important that it will go through mangle rules in prerouting chain

What do you think will happen when it reaches this one?
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=no-mark in-interface=pppoe-out2 new-connection-mark=ISP2_conn passthrough=no
- chain=prerouting - check
- connection-mark=no-mark - check
- in-interface=pppoe-out2 - check

=> rule matches => connection between LAN device and 1.2.3.4:666 now has connection mark ISP2_conn.

Do I have to explain what happens when device continues with communication and sends another packet to 1.2.3.4:666, which is part of same connection?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 6:49 pm

With multi-thread (or multi-session, whatever) download, you have to be extremely unlucky not to get the aggregated bandwidth, because each thread = TCP session = tracked connection is treated separately by per-connection traffic distribution. So the fact that you get aggregate bandwidth with multi-threaded download bears no information about how the distribution is actually done.

I don't dispute the fact that most applications (and bandwidth test utilities) use multi-session mode, that's clear (and it is not the same thing as MPTCP, as these applications do the traffic multiplexing at application level and open multiple TCP sockets, whilst MPTCP is providing a single socket to the application and splits the traffic into several sessions on its own, unnoticeably for the application).

As for
I reposted my config here: viewtopic.php?f=14&t=167416&p=823004#p823004

Now do you see any flaws whereby traffic escapes the packet-marking rules + routing marking rules (based on packet marks)?

Also this explanation: viewtopic.php?p=822800#p822800 assumes a single TCP stream/thread.
From the end - the explanation assumes a single TCP stream/session/thread/connection because it only makes sense to discuss the difference between per-connection distribution and per-packet distribution for a single stream. If you have a multi-thread application or test, you don't need per-packet distribution even where it could make some sense for a single session if no NAT was affecting the packets' addresses.

Next, in your newest posted config, which is the same one you've posted before, the traffic actually does not escape the packet marking rules (the nth ones). What it does escape are the rules translating the (just assigned) packet-mark into a corresponding routing-mark.

The reason why it escapes them is that the rules which translate a connection-mark into a corresponding routing-mark match that traffic as well, and as they have passthrough=no, they are the last ones to handle that traffic in mangle/prerouting.

So to be absolutely precise, the traffic doesn't escape the packet-mark => routing-mark rules, it never reaches them.

And the reason why the traffic matches those connection-mark => routing-mark rules is that even those connections whose first packet (LAN => WAN) didn't cause assignment of any connection-mark to the connection do get a connection-mark assigned when the first response packet from the remote server (WAN => LAN) is processed in mangle/prerouting.

The fact that this traffic's LAN => WAN packets have got also a packet-mark assigned doesn't matter because the connection-mark => routing-mark rules do not care about presence or absence of the packet-mark.

As a result, the nth rules only effectively affect the choice of WAN for the initial packet of each connection (the SYN one in case of TCP). All the subsequent packets of that connection inherit that choice by means of the connection-mark assigned when handling the WAN => LAN response (SYN+ACK in case of TCP).

So to conclude:
  • with a single-threaded transfer and per-connection distribution, you cannot get aggregate bandwidth
  • the fact that you do get an aggregate bandwidth for a multi-threaded transfer provides no information regarding the type of traffic distribution
  • with a single-threaded transfer and per-packet traffic distribution, you could get aggregate bandwidth in a transparent network, but presence of NAT on the WANs prevents it from working. But you cannot see that until you prevent the traffic intended to be distributed per-packet from being hijacked by the connection-mark => routing-mark rules and thus being distributed per-connection.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 8:23 pm

So to conclude:
  • with a single-threaded transfer and per-connection distribution, you cannot get aggregate bandwidth
  • the fact that you do get an aggregate bandwidth for a multi-threaded transfer provides no information regarding the type of traffic distribution
  • with a single-threaded transfer and per-packet traffic distribution, you could get aggregate bandwidth in a transparent network, but presence of NAT on the WANs prevents it from working. But you cannot see that until you prevent the traffic intended to be distributed per-packet from being hijacked by the connection-mark => routing-mark rules and thus being distributed per-connection.
For testing purposes, how do I replicate a pure per-packet distribution instead of per-connection distribution? (Excluding 80/443 for obvious reasons)
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 8:25 pm

The last paragraph of post #32 suggests multiple ways how to do that. Any of them is sufficient.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 8:36 pm

The last paragraph of post #32 suggests multiple ways how to do that. Any of them is sufficient.
So I did the following, so far nothing breaks (80/443 obviously will never go beyond the following), I'll test it for a few hours with gaming, VoIP, Banking. My thumb of rule is, if banking works everything else should work.

I hope this means a true per-packet config as per your explanations.
add action=mark-routing chain=prerouting connection-mark=ISP1_conn in-interface-list=LAN new-routing-mark=to_ISP1 packet-mark=no-mark passthrough=no
add action=mark-routing chain=prerouting connection-mark=ISP2_conn in-interface-list=LAN new-routing-mark=to_ISP2 packet-mark=no-mark passthrough=n
Edit: So I found a problem already, 80/443 gets overwritten in mark-packets with packet-marks hence broken secure sites.

I can't come up with an optimal to excluded 80/443 completely. If I use connection-mark=no-mark in Nth, this basically excludes everything without a connection even if they are not 80/443.

What do you suggest as an optimal solution to prevent 80/443 completely from Nth?
Last edited by DarkNate on Fri Oct 16, 2020 9:33 pm, edited 1 time in total.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 9:33 pm

EDIT: to prevent 80,443 from being packet-marked, add dst-port=!80,443 to the two rules above (translating connection-mark to routing-mark) or to the mark-packet ones. I've missed that the mark-packet ones aren't selective either.

It's always better to check the result than to just hope. If everything is set correctly, the rules translating the packet-mark into routing-mark must count exactly the same amount of packets as the rules which assign the packet-mark. Reset counters at all five of them using a single command so that they would start counting from zero at the same time.

But that's only a test that per-packet distribution is actually being done.

Another thing is how to find out the difference in throughput between per-packet and per-connection distribution. For TCP download, the difference may be almost unnoticeable, as the ACK packets are redundant in a way, and not every data packet needs to be ACKed. And only the ACK packets for the downloaded data will be actually lost when taking the wrong path.

A TCP upload test is another thing, as there the packets carrying the payload will get lost and thus will have to be retransmitted, so the throughput degradation should be clearly visible.

VoIP is a question - some exchanges may tolerate packets coming from different addresses, some may play just those which came from the same IP address like the first one, some may play nothing at all. But the problem will always be audible at the remote side, not at your one.
Last edited by sindy on Fri Oct 16, 2020 9:38 pm, edited 1 time in total.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 9:35 pm

It should, but it's always better to check than to just hope. If everything is set correctly, the rules translating the packet-mark into routing-mark must count exactly the same amount of packets as the rules which assign the packet-mark. Reset counters at all five of them using a single command so that they would start counting from zero at the same time.

But that's only a test that per-packet distribution is actually being done.

Another thing is how to find out the difference in throughput between per-packet and per-connection distribution. For TCP download, the difference may be almost unnoticeable, as the ACK packets are redundant in a way, and not every data packet needs to be ACKed. And only the ACK packets for the downloaded data will be actually lost when taking the wrong path.

A TCP upload test is another thing, as there the packets carrying the payload will get lost and thus will have to be retransmitted, so the throughput degradation should be clearly visible.

VoIP is a question - some exchanges may tolerate packets coming from different addresses, some may play just those which came from the same IP address like the first one, some may play nothing at all. But the problem will always be audible at the remote side, not at your one.
I edited my post earlier, some apparent issues.

Also in general, what do you think is the best possible load balancing strategy to maximise bandwidth distrubtion/aggregation without breaking anything? PCC or Nth (Mark connection or mark packet)?
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 9:47 pm

I've edited my post reacting to your edited one accordingly.

I've expressed my opinion earlier - in an environment with NAT, the impact of per-packet traffic distribution is always negative (except connections which consist of a single request packet and a single response one as @Sob has pointed out).

In an environment without NAT, per-packet distribution may save some resources on the router (let's say ECMP with no caching and no connection tracking), but can affect TCP throughput on endpoints that don't handle missequenced packets well.

And whether you use per-connection-classifier, random, or nth for the choice of WAN for a given connection in case of per-connection distribution doesn't make any significant difference if you include source-port to the per-connection-classifier's hash.

So per-connection-classifier with only source-address for ports 80,443 and with source-address-and-port for the rest is as good as using nth for the rest if you use nth to assign connection-mark as it was before. To be clear - I've never said nth is a bad method for choosing the WAN with per-connection distribution, I just say it is no better than the other methods.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 9:53 pm

I've edited my post reacting to your edited one accordingly.

I've expressed my opinion earlier - in an environment with NAT, the impact of per-packet traffic distribution is always negative (except connections which consist of a single request packet and a single response one as @Sob has pointed out).

In an environment without NAT, per-packet distribution may save some resources on the router (let's say ECMP with no caching and no connection tracking), but can affect TCP throughput on endpoints that don't handle missequenced packets well.

And whether you use per-connection-classifier, random, or nth for the choice of WAN for a given connection in case of per-connection distribution doesn't make any significant difference if you include source-port to the per-connection-classifier's hash.

So per-connection-classifier with only source-address for ports 80,443 and with source-address-and-port for the rest is as good as using nth for the rest if you use nth to assign connection-mark as it was before. To be clear - I've never said nth is a bad method for choosing the WAN with per-connection distribution, I just say it is no better than the other methods.
All right, I saw your edit. It raises complexity as I need to include protocol first before being about to add ports or !ports. At this point, I might as well just drop Nth completely.

If there's no difference between PCC and Nth in per-connection distribution, then what exactly is this paper talking about?

https://www.ijcnis.org/index.php/ijcnis ... /view/4340
 
Sob
Forum Guru
Forum Guru
Posts: 9119
Joined: Mon Apr 20, 2009 9:11 pm

Re: NTH load balancing

Fri Oct 16, 2020 10:16 pm

I'm not re-reading the whole thread again to find out if it was already mentioned in some way, but do you have any statistics how much are ports other than 80 and 443 actually used? In other words, if all your effort is really worth it. Because if you are developer and your application needs to access internet, the safest way is to just stick to https on default port 443. In most cases, even when everything else is blocked, https works.

You can try these rules as simple counters (move the first one to be the first rule in chain):
/ip firewall mangle
add action=jump chain=prerouting jump-target=stats connection-state=new \
    dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN src-address=192.168.x.x
add action=passthrough chain=stats dst-port=80 protocol=tcp
add action=passthrough chain=stats dst-port=443 protocol=tcp
add action=passthrough chain=stats dst-port=!80,443 protocol=tcp
add action=passthrough chain=stats dst-port=80 protocol=udp
add action=passthrough chain=stats dst-port=443 protocol=udp
add action=passthrough chain=stats dst-port=!80,443 protocol=udp
They don't influence anything, they just count new connections. The src-address=192.168.x.x is to limit it only to selected device, to easily test different things (web browsing, downloading games, whatever you do, ideally one thing at the time) and don't have it influenced by other devices. If you're interested in more details, change action=passthrough to action=log.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 10:22 pm

I'm not re-reading the whole thread again to find out if it was already mentioned in some way, but do you have any statistics how much are ports other than 80 and 443 actually used? In other words, if all your effort is really worth it. Because if you are developer and your application needs to access internet, the safest way is to just stick to https on default port 443. In most cases, even when everything else is blocked, https works.

You can try these rules as simple counters (move the first one to be the first rule in chain):
/ip firewall mangle
add action=jump chain=prerouting jump-target=stats connection-state=new \
    dst-address-list=!not_in_internet dst-address-type=!local in-interface-list=LAN src-address=192.168.x.x
add action=passthrough chain=stats dst-port=80 protocol=tcp
add action=passthrough chain=stats dst-port=443 protocol=tcp
add action=passthrough chain=stats dst-port=!80,443 protocol=tcp
add action=passthrough chain=stats dst-port=80 protocol=udp
add action=passthrough chain=stats dst-port=443 protocol=udp
add action=passthrough chain=stats dst-port=!80,443 protocol=udp
They don't influence anything, they just count new connections. The src-address=192.168.x.x is to limit it only to selected device, to easily test different things (web browsing, downloading games, whatever you do, ideally one thing at the time) and don't have it influenced by other devices. If you're interested in more details, change action=passthrough to action=log.
Oh yeah, this an easy one. Ephemeral Port Range is quite active in my local network because of reasons like the following:
Gaming (P2P aka true Open NAT)
VoIP (FaceTime, VoWiFi etc)
P2P apps (torrenting)
Mobile Apps etc etc
And because I enable UPnP (which is NOT a security risk as malwares makes use of STUN or NAT hole punching regardless of UPnP status and because I'm confident in my clean systems and devices)

I can check easily in connection tracking, there's thousands of different ports every day in the Ephemeral range. Not even required to use to a count rule for this one.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 10:26 pm

If there's no difference between PCC and Nth in per-connection distribution, then what exactly is this paper talking about?

https://www.ijcnis.org/index.php/ijcnis ... /view/4340
Let me quote myself from post #16:

"the apparently best dynamic load distribution method (nth) gives astounding 101.5% of the download throughput of the apparently worst one (ECMP) - this is a statistical error to me, given that they've tested each method for 40 (consecutive?) days, and there is no information on what was the test pattern, i.e. how comparable these 40 day intervals actually were".

Other information in that paper indicates that even nth was used to control per-connection distribution.

For upload, the ratio is more convincing, nth-controlled distribution gave 113% of the throughput of ECMP-controlled distribution.

So nth may be really be a bit better than ECMP and per-connection-classifier, or the external conditions may have been more favourable during the test with nth. With such a small difference and such a limited information about the test conditions it is hard to decide.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 10:31 pm

If there's no difference between PCC and Nth in per-connection distribution, then what exactly is this paper talking about?

https://www.ijcnis.org/index.php/ijcnis ... /view/4340
Let me quote myself from post #16:

"the apparently best dynamic load distribution method (nth) gives astounding 101.5% of the download throughput of the apparently worst one (ECMP) - this is a statistical error to me, given that they've tested each method for 40 (consecutive?) days, and there is no information on what was the test pattern, i.e. how comparable these 40 day intervals actually were".

Other information in that paper indicates that even nth was used to control per-connection distribution.

For upload, the ratio is more convincing, nth-controlled distribution gave 113% of the throughput of ECMP-controlled distribution.

So nth may be really be a bit better than ECMP and per-connection-classifier, or the external conditions may have been more favourable during the test with nth. With such a small difference and such a limited information about the test conditions it is hard to decide.
What do you think is the proper way to use Nth for per-connection distribution (implying mark connection)? The way I did originally?

Earlier you mentioned directly assigning routing marks instead of marking then routing them to reduce CPU usage. Would that logic also apply to normal PCC setups? Should we all use mark-routing directly with PCC and get rid mark connections completely?

I tried the following and it does not work at all
add action=mark-routing routing-mark=!to_ISP1 chain=prerouting in-interface=pppoe-out1 new-routing-mark=to_ISP1 passthrough=no comment="Handles WAN1"
add action=mark-routing routing-mark=!to_ISP2 chain=prerouting in-interface=pppoe-out2 new-routing-mark=to_ISP2 passthrough=no comment="Handles WAN2"

add action=mark-routing routing-mark=!to_ISP2 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP2 dst-address-type=!local dst-address-list=!not_in_internet protocol=tcp port=80,443 per-connection-classifier=both-addresses:3/0 passthrough=no comment="Handles HTTPS"
add action=mark-routing routing-mark=!to_ISP2 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP2 dst-address-type=!local dst-address-list=!not_in_internet protocol=tcp port=80,443 per-connection-classifier=both-addresses:3/1 passthrough=no comment="Handles HTTPS"
add action=mark-routing routing-mark=!to_ISP1 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP1 dst-address-type=!local dst-address-list=!not_in_internet protocol=tcp port=80,443 per-connection-classifier=both-addresses:3/2 passthrough=no comment="Handles HTTPS"

add action=mark-routing routing-mark=!to_ISP2 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP2 dst-address-type=!local dst-address-list=!not_in_internet protocol=udp port=80,443 per-connection-classifier=both-addresses:3/0 passthrough=no comment="Handles HTTPS"
add action=mark-routing routing-mark=!to_ISP2 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP2 dst-address-type=!local dst-address-list=!not_in_internet protocol=udp port=80,443 per-connection-classifier=both-addresses:3/1 passthrough=no comment="Handles HTTPS"
add action=mark-routing routing-mark=!to_ISP1 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP1 dst-address-type=!local dst-address-list=!not_in_internet protocol=udp port=80,443 per-connection-classifier=both-addresses:3/2 passthrough=no comment="Handles HTTPS"


add action=mark-routing routing-mark=!to_ISP2 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP2 dst-address-type=!local dst-address-list=!not_in_internet nth=3,1 passthrough=no
add action=mark-routing routing-mark=!to_ISP2 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP2 dst-address-type=!local dst-address-list=!not_in_internet nth=3,2 passthrough=no
add action=mark-routing routing-mark=!to_ISP1 chain=prerouting in-interface-list=LAN new-routing-mark=to_ISP1 dst-address-type=!local dst-address-list=!not_in_internet nth=3,3 passthrough=no
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: NTH load balancing

Fri Oct 16, 2020 10:34 pm

Then, you don't know up-front how much traffic will go over a marked connection.

I could look in NAT which connection, had not much traffic yet and then prefer that link. In real time, that is only possible if Mikrotik implement a distribution by clean switching of the source port. Maybe that is already possible but I don't have knowledge over that.

I use a adress-list that don't allow switching connection is the destination does not support like me coming from different source addresses.

This will have the PCC/Nth switch instantly to a/the next link. It will cause in the connection tracking a higher load and that is not very desirable.

@DarkNate try Nth 3-1 2-1 - which is the same as 3-1 3-2 3-3 and I think, less processor intensive.
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Fri Oct 16, 2020 10:47 pm

Then, you don't know up-front how much traffic will go over a marked connection.

I could look in NAT which connection, had not much traffic yet and then prefer that link. In real time, that is only possible if Mikrotik implement a distribution by clean switching of the source port. Maybe that is already possible but I don't have knowledge over that.

I use a adress-list that don't allow switching connection is the destination does not support like me coming from different source addresses.

This will have the PCC/Nth switch instantly to a/the next link. It will cause in the connection tracking a higher load and that is not very desirable.

@DarkNate try Nth 3-1 2-1 - which is the same as 3-1 3-2 3-3 and I think, less processor intensive.
I am not concerned with data usage per marked connection. My first ISP has 750GB data cap, second ISP has no data cap. So my 3-way split works well overall.

Nth 3,1 - 2,1 is likely not the same as Nth 3,1 - 3,2 - 3,3 and if I remember correctly from some MikroTik presentation files, it has to be in that order for either PCC/Nth where 2 means two WAN, 3 means 3 WAN in general, you can't mix the two without likely breaking something.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Fri Oct 16, 2020 11:13 pm

What do you think is the proper way to use Nth for per-connection distribution (implying mark connection)? The way I did originally?
Yes, that way was fine for per-connection distribution. All that confusion came just from the fact that you declared that it does per-packet distribution.

Earlier you mentioned directly assigning routing marks instead of marking then routing them to reduce CPU usage. Would that logic also apply to normal PCC setups? Should we all use mark-routing directly with PCC and get rid mark connections completely?
The same packet fields (source and destination addresses and ports) are used by both the connection tracking module to identify which packet belongs to which connection and by per-connection-classifier to calculate the hash. As these fields never change for packets belonging to the same connection, you don't need to use connection-mark if you use per-connection-classifier matching to assign routing-mark directly. BUT: whereas connection-marks are inherited also by "related" connections, such as ICMP notifications about MTU issues in a TCP connection, I'm afraid these "related" packets are not treated by per-connection-classifier the same way like the original connection's packets, so the ICMP notification might be sent through the wrong WAN. I'm not sure whether it really causes issues in real life, but it definitely can. MTU issues are among the most frequently encountered ones.

Another reason to keep using connection-marks is if you want to be able to connect to either of your WAN IPs from the internet, as then you need the responses to be sent out through the same WAN through which the requests came in. You seem to need this, as otherwise the connection-marking based on in-interface and routing-marking in mangle/output would be useless.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 18959
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: NTH load balancing

Fri Oct 16, 2020 11:50 pm

Does acquiescence equal an apology?? ;-)
 
DarkNate
Forum Veteran
Forum Veteran
Posts: 997
Joined: Fri Jun 26, 2020 4:37 pm

Re: NTH load balancing

Sat Oct 17, 2020 12:10 am

What do you think is the proper way to use Nth for per-connection distribution (implying mark connection)? The way I did originally?
Yes, that way was fine for per-connection distribution. All that confusion came just from the fact that you declared that it does per-packet distribution.

Earlier you mentioned directly assigning routing marks instead of marking then routing them to reduce CPU usage. Would that logic also apply to normal PCC setups? Should we all use mark-routing directly with PCC and get rid mark connections completely?
The same packet fields (source and destination addresses and ports) are used by both the connection tracking module to identify which packet belongs to which connection and by per-connection-classifier to calculate the hash. As these fields never change for packets belonging to the same connection, you don't need to use connection-mark if you use per-connection-classifier matching to assign routing-mark directly. BUT: whereas connection-marks are inherited also by "related" connections, such as ICMP notifications about MTU issues in a TCP connection, I'm afraid these "related" packets are not treated by per-connection-classifier the same way like the original connection's packets, so the ICMP notification might be sent through the wrong WAN. I'm not sure whether it really causes issues in real life, but it definitely can. MTU issues are among the most frequently encountered ones.

Another reason to keep using connection-marks is if you want to be able to connect to either of your WAN IPs from the internet, as then you need the responses to be sent out through the same WAN through which the requests came in. You seem to need this, as otherwise the connection-marking based on in-interface and routing-marking in mangle/output would be useless.
All right, connection marks it is. I do need to handle incoming WAN connections, yes.

So my mistake was my implementation of Nth originally was per-connection which I thought was per-packet. And if it was per-packet HTTPS breaks and excluding HTTPS was too much of a pain.

Is there anything else we could do short of a VPS+bonding solution? That would give us even better bandwidth distribution?
 
msatter
Forum Guru
Forum Guru
Posts: 2897
Joined: Tue Feb 18, 2014 12:56 am
Location: Netherlands / Nīderlande

Re: NTH load balancing

Sat Oct 17, 2020 1:10 am


@DarkNate try Nth 3-1 2-1 - which is the same as 3-1 3-2 3-3 and I think, less processor intensive.
Nth 3,1 - 2,1 is likely not the same as Nth 3,1 - 3,2 - 3,3 and if I remember correctly from some MikroTik presentation files, it has to be in that order for either PCC/Nth where 2 means two WAN, 3 means 3 WAN in general, you can't mix the two without likely breaking something.
I found the Wiki not clear on this and later I learned this interpretation and they need to be in sequence as lines.

Nth divides the traffic and a example: three conections split equal. 3-1 takes one of thee conections leaving two. 2-1 takes one of two and leaves one. So the last connection gets the last traffic 1-1 or you just omit the Nth for that connection. 1-1 or 3-3 or omit acts, also as a catch all incase any Nth lines gone missing.

When you work that way, it is much easier to create a asymetric Nth divider.

If you have a data cap then you can change to asymetric if you reach let's say 75% and each ever so many percent increase the traffic to the un-capped line.

Example 90% cap reached one capped line with three lines: capped 5-1 uncapped 2-1 uncaped 2-1
You would need two more lines to create that: capped 5-1 uncapped 5-2 + 5-3 uncapped 5-4 + 5-5
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: NTH load balancing

Sat Oct 17, 2020 11:45 am

try Nth 3-1 2-1 - which is the same as 3-1 3-2 3-3 and I think, less processor intensive.
Nth 3,1 - 2,1 is likely not the same as Nth 3,1 - 3,2 - 3,3 and if I remember correctly from some MikroTik presentation files, it has to be in that order for either PCC/Nth where 2 means two WAN, 3 means 3 WAN in general, you can't mix the two without likely breaking something.
Both approaches are correct, but each in a different context.

The nth matcher is specific - it is stateful, i.e. each instance of nth creates its own independent counter, and the counter is only incremented and compared to the trigger value if all the other match conditions of that rule match (or in other words, if the rule would match if the nth wasn't there). And whether the rule as a whole matches or not depends on the result of the comparison of the counter value with the trigger value. So let's call these possibilities "basic match" (all other match conditions are met and the counter is incremented and compared to the trigger value) and "full match" (on top of the basic match, the counter value matches the trigger value).

So if the action taken thanks to full match of an nth rule on a packet prevents a basic match of the subsequent nth rules on the same packet, or even prevents the packet from even reaching those rules (passthrough=no), the proper rule sequence for a 1:1:1 distribution is 3,1 -> 2,1 -> (1,1).
If the full match of an nth rule on a packet does not prevent a basic match of the subsequent nth rules on the same packet, the proper rule sequence for a 1:1:1 distribution is 3,1 -> 3,2 -> 3,3.

so:
nth=3,1                     action=mark-packet new-packet-mark=mark-1 passthrough=yes
nth=2,1 packet-mark=no-mark action=mark-packet new-packet-mark=mark-2 passthrough=yes
        packet-mark=no-mark action=mark-packet new-packet-mark=mark-3 passthrough=yes

give the same result as
nth=3,1 action=mark-packet new-packet-mark=mark-1 passthrough=yes
nth=3,2 action=mark-packet new-packet-mark=mark-2 passthrough=yes
nth=3,3 action=mark-packet new-packet-mark=mark-3 passthrough=yes

Who is online

Users browsing this forum: No registered users and 16 guests