I’d like to know what is the difference between src-add / scr-add and port / both-add / both-add and port in this specific context :
We have a gig connexion (routerA) and a gig connexion (routerB) connected to a rb5009 using wireless linnk. The 5009 has 500 customers connected to it. The only goal here is to distribute connexion between 2 wan connexion without breaking customer experience. Router A has 600mbps available (wireless limit) and router B has 400mbps. Thjat why I set it to 4/10 and 6/10 ratio
In my different tests I realise that src-add was the best choice but customers complains about lag and freeze an echo on Microsoft team. I tried both-add and from my experience, i was pretty sure Whatsapp was laggy and not responsive. And now im trying both-add and port to make connection more sticky. Not sure about the result.
Can someone tell me If these kind of mangle rules are good or not I read the wiki, Mikrotik propose output rules and a diffrent way to achieve that.
/ip firewall mangle
# Route all traffic to 10.1.50.2 via main routing table (VoIP or specific service)
add action=mark-routing chain=prerouting dst-address=10.1.50.2 new-routing-mark=main passthrough=no
# Force a specific client to always use main routing table
add action=mark-routing chain=prerouting src-address=10.6.6.10 new-routing-mark=main passthrough=no
# PCC: First 4 buckets to WAN1 based on both addresses and ports
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan1 per-connection-classifier=both-addresses-and-ports:10/0 comment="PCC WAN1 bucket 0"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan1 per-connection-classifier=both-addresses-and-ports:10/1 comment="PCC WAN1 bucket 1"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan1 per-connection-classifier=both-addresses-and-ports:10/2 comment="PCC WAN1 bucket 2"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan1 per-connection-classifier=both-addresses-and-ports:10/3 comment="PCC WAN1 bucket 3"
# PCC: Last 6 buckets to WAN2 based on both addresses and ports
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan2 per-connection-classifier=both-addresses-and-ports:10/4 comment="PCC WAN2 bucket 4"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan2 per-connection-classifier=both-addresses-and-ports:10/5 comment="PCC WAN2 bucket 5"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan2 per-connection-classifier=both-addresses-and-ports:10/6 comment="PCC WAN2 bucket 6"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan2 per-connection-classifier=both-addresses-and-ports:10/7 comment="PCC WAN2 bucket 7"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan2 per-connection-classifier=both-addresses-and-ports:10/8 comment="PCC WAN2 bucket 8"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan2 per-connection-classifier=both-addresses-and-ports:10/9 comment="PCC WAN2 bucket 9"
# Mark traffic based on connection marks for WAN2
add action=mark-routing chain=prerouting connection-mark=conn-wan2 in-interface=bridge-local new-routing-mark=to-link-A comment="Route WAN2 traffic to to-link-A"
# Mark traffic based on connection marks for WAN1
add action=mark-routing chain=prerouting connection-mark=conn-wan1 in-interface=bridge-local new-routing-mark=to-link-B comment="Route WAN1 traffic to to-link-B"
obviously I have routing table for each connexion and nat rules.
My quick take is you may be over-focused on the PCC part… While that may be involved, I’m not sure that’s the entire story. VoIP/realtime AV (i.e. “Microsoft teams”) is way more sensitive to latency/packet loss/etc than “normal” web traffic.
Are you using any queuing mechanism? To me, that where I’d start if you’re seeing lag/freezing in video calls even a simple fq_codel on WAN outbound like cleanup latency and ensure fairness between customers.
On PCC specifically… While both-address-and-port might get “most” balanced load, it comes at the cost of some application/services just don’t like connections coming from different source IPs. NOW… given you have 500 customers, that likely a lot of connections… So there plenty of opportunity to spread connections across your 2 WANs without using both-address-and-ports. Dunno you’re exact topology… but I just force each of the customer over only one WAN using just customer’s src-address - with high number of connections it likely still be pretty evenly divided.
But the complaint of “customers complains about lag and freeze an echo on Microsoft team” does suggest something related to either difference in upstream WANs, congestion, etc… I’d start by looking your wireless link quality and/or queuing strategy (and have PCC as just “src-address” if possible or both-addresses [no ports]*)… But my bet be the PCC config itself is NOT likely causing lag/drops/etc on video calls.
(* with PCC both-addresses is same as ECMP routes, which are easier/cleaner than PCC firewall rules IMO… but let’s not confuse this further since PCC is roughly working for you today)
I agree with Ammo.
ECMP is easier to go with for a relatively same throughput.
The issues sound in my mind to be possibly MTU issues.
Suggest at least posting the config of the Rb5009 to see what is going on
/export file=anynameyouwish ( minus router serial number, any public WANIP information, keys )
I have to agree with the others. Your mangling looks sane. I would also suggest using only the src address as the qualifier. The thing about a client changing IP mid-transaction is built into a lot of intrusion detection systems. (It is a recommended but not mandatory recommendation in PCI-DSS so it got implemented.)
The ultimate source of your problem is elsewhere. It would be a good idea to test actual MTU on both paths.
I’m just wondering why there are some folks here loved to use this PCC which consume lots of CPU because of mangle with little tangible results VS proper policy based route which is very predictable and easy to manipulate if you want to re-route traffic of a certain subnet or customer to specific upstream PCC has lots of quirk and doesn’t scale well beyond 800+ish customers at least from the experience or I just simply don’t know how to configure it properly? I’m talking CCR1036/2116 here the performance really varies our millage may vary of course
You dont need to define 10 separate PCC lines: 4:6 == 2:3 and that is a total of five PCC. The lines beneath will do the same as your 10 PCC lines.
/ip firewall mangle
# Route all traffic to 10.1.50.2 via main routing table (VoIP or specific service)
add action=mark-routing chain=prerouting dst-address=10.1.50.2 new-routing-mark=main passthrough=no
# Force a specific client to always use main routing table
add action=mark-routing chain=prerouting src-address=10.6.6.10 new-routing-mark=main passthrough=no
# PCC: First to WAN1 based on both addresses
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan1 per-connection-classifier=both-addresses:5/0 comment="PCC WAN1 bucket 0"
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan1 per-connection-classifier=both-addresses:5/4 comment="PCC WAN1 bucket 4"
# PCC: Not yet marked to WAN2
add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new in-interface=bridge-local new-connection-mark=conn-wan2 comment="PCC WAN2 bucket 1 2 3"
# Mark traffic based on connection marks for WAN2
add action=mark-routing chain=prerouting connection-mark=conn-wan2 in-interface=bridge-local new-routing-mark=to-link-A comment="Route WAN2 traffic to to-link-A"
# Mark traffic based on connection marks for WAN1
add action=mark-routing chain=prerouting connection-mark=conn-wan1 in-interface=bridge-local new-routing-mark=to-link-B comment="Route WAN1 traffic to to-link-B"
FWIW, in RouterOS v7, the routing engine for ECMP does not store duplicate routes like V6… so the asymmetrical splits are not possible as the routing engine will select only two routes in this case: 10.1.50.2 and 10.6.6.10
Just noticed that by trying it. Can the same not be archived by creating interfaces (VRF?)and IP/addresses and so creating dynamic ECMP routes? Addresses with the same dst in routes get a + symbol, standing for ECMP. This stuff is completely new to me and I have a working system so hesitant to change stuff.
both link are top quality and there are no issues as far as I know. Speed and latency are perfect.
both internet upstream are also top quality, no problem here.
I said 500 customers. Yes 494 to be exact. But they are not online for the moment, they will be connected in 3 months. Still building the network infrastructure! So I know that the network is not in use and its free of traffic, no bottle neck. Only 2-3 actives clients right now.
Mtu could be a problem but i don’t think, plese correct me if i’m wrong. To be more clear, both upstream have gre tunnel (1476 mtu) from their respective location to a datacenter far away. Then we have a wireless links going to a netpower16p, then a vlan carry the data and push it to a sfp+10g right through a crs310-1g-5s-4+. Then it goes to the rb5009 on port sfp+, also 10g.
The rb5009 has a bunch of crs326 connected to it and then it end up with TPLink AX routers. Wich have pppoe connection (1480) mtu. Those tplink are customers routers, 494 devices.
So, 1476 is the maximum MTU without fragmentation allowed.
I have a 10 pcc rules, I did that because I want it to be more precise/granular. Since our links have 400mbps and 600mbps, and a total of 1000mbps, I though it was a good idea to set the ratio on 1/10. Correct me if I’m wrong.
Msatter before Version 7.1X I forget which, PCC was clearly the way to go as I do not think ECMP was setup on straight policy routing. One had to invoke extra rules and not sure it worked all that well.
Once it became available, as part and parcel of normal policy routing, it was a no brainer for relatively equal throughput WANs be it 2,3,4 etc
PCC is still very effective for different throughtputs to try and even out the distribution.
Even better is the more intricate LOAD Balancing vice PCC where its based on percentage of traffic but who likes double sticky mangle rules just for the fun of it.
Also can you explain the logic of 5/0 & 5/4 in the two rules.
AKA will help explain what happened to 5/1, 5/2 and 5/3??
ECMP work on all routes within a table, so you could ECMP on main, or ECMP on a special table as well.
In addition, within PCC there are times when you create more PCC mangles and routes than normal, but that is to ensure a failover distribution as required.
Basically break up a mangling into smaller pieces so that when one route fails, the traffic is split amongst the remaining routes, vice all traffic from fail route going to next available route.
Let me ask it another way, how would you setup PCC for three WANS
WAN1 - 2.5 Gpbs ( 2.5x greater than WAN2, 10X greater than WAN1 )
WAN2 - 1 Gbps (4X greater than WAN3 )
WAN3 - 250Mbps
Basically, right or wrong I would start by thinking in this way. Approx 10 connections of WAN1 for every 4 connections of wan2 and 1 connection of wan1
It’s like using a DROP rule at the end of fw filter to capture “everything else” – except in reverse.
Each rule in firewall has a small CPU cost, so if you can reduce PCC rules, you improve latency.
So theory be you have the SMALL sized PCC buckets with specific rules, leaving the one that has most bandwidth covered by the catch-all rule (as this reduces the total number of rules).
And, important to keep in mind, PCC deals with connections, not packets. So while a PCC distribution may perfectly match available bandwidth… some connections use more bandwidth than others… so at some point, you’re just adding more rules to process with little effect.
Hi Ammo, I think I concluded the same so my answer would be appear to be correct.
Understood, however the amount (contents) of any connection is not predictable in the PCC method so one goes at it the best one can.
The double sticky approach of actual bandwidth increases the number of rules but allows one to base usage on actual throughput which would be more accurate.
What is bugging me now, is HOW in smatters example does the router know that WAN2 is actually involved at all in PCC.
There are no ratios applied in the rule!! NO PCC is attributed to WAN2 at all, so how does the router know to magically assign the other slots to wan2 in the example provided ???
Yup no PCC is applied to last one… so rule just assigns a connmark (and routing-mark) since it “escaped” PCC. PCC at end of day is just a “matching rule”, but it does not take action itself (i.e. you still need some action=).
So you are saying, in effect, that the router STILL HAS TO COUNT CONNECTION ASSIGNMENTS… in this case 14 …
SO it auto counts and assigns the missing ones to the non PCC available WAN in the same table.
Now this approach sucks since the OP has WANS 4 (future wisp) and 5,6 LTE backups in the mix…
In other words, would auto assign them to ??? if WAN1 was lower distance ( than 4,5,6) I guess would still work
I use only PCC because third party websites on the other site don’t like traffic to appear from different source addresses. By splitting up the traffic in connections and not packets most websites can still keep your session tracked.
When simplifying a simple PCC you just take the total number of rules, and lay that number in front on the with for example table matches. Then put the number of matches to each 'WAN stack" as you like to divide the traffic with the PCC rules. Then split each stack of matches in half till you can can’t split no further without breaking a match in two.
OP: 10 matches, four for wan1 and six for wan2…
It takes one iteration to two matches for wan1 and three matches for wan2 and two loops. You can’t divide three matches by two without breaking a match.
And so you only need two rules to get a split of 10=4:6 with PCC 3:0 and (3:1 or 3:2). Normally you would write 10:0 10:1 10:3…10:9 making ten PCC rules.
A PCC divider does not have to be equal parts when splitting connections. Look then at it as percentages to make it more simple.
first PCC rule 2:0 splits half of the connections (50%) and marks them for wan1
second PCC rule 5:0 splits a fifth of the remaining connections in to one (10%) and marks them for wan2. You are now doing 50%+10% = 60%
This leaves then the catch-all to wan3 having receiving all the not yet marked traffic (40%). Then you mark that traffic for wan3. You are now doing 50%+10%+40%=100% in a stream six connections.
So in total three rules (including catch-al) replaces six normal PCC lines: 2:0 5:0 5:1 5:2 5:3 5:4 saving 50% in the number of needed rules. Now you can also break matches when using percentage when going asymmetric. 50%:50%=100% escaping 50%=100% for the next PCC rules. Then 3:0 3:1 3:2 is possible now for that remaining 50%.
The catch-all will catch any packets that are not yet marked…making it foolproof when you make an error in the rules.
You can make it as complex as you need. It is more complex because PCC is using remainder of a hash and if you write 2:0 then all even numbers are taken out of the stream. Then the next PCC rules can only address uneven numbers and split those.
example: wan1 → 2:0 | wan2->6:1 & 6:3 | wan3 ->6:5 making it a 50:33:17 (rounded) and so still able to break matches and able to make complex distributions.
wan1->3:0 | wan2->2:1 | wan3->3:2 making it a 33:26:26 (rounded) catch-all wan4->15. This is a guess, because I can’t compute this in my brain.
We would be roughly happy with 67ish% 27ish% 6ish% percentages
So roughly speaking we do something like
15/3 for WAN2 (approx 6.75)
15/6 for WAN2 (approx 6.75)
15/9 for WAN2 (approx 6.75)
15/14 for WAN2 (approx 6.75) for a total of approx 27% for WAN2
15/13 for WAN3 ( approx 6.7%) for a total of approx 7% for WAN3
catch all ( the rest ) for WAN1 approx 66% ( so three percentages, in a stream of 15 connection with 5PCC rules and one non-catchall pcc rule)
???
OR msatter,is this better?
I suppose I could just do: ( 2 RULES, so more efficient ?? )
4/0 for WAN2 which equates to 25%
15/0 for WAN three which does 6.6 percent
The rest would go to WAN1 ( aka the other 3 slots equal 75 percent ) but first we subtract
6.6% leaving 68% for WAN1
I read the wiki, Mikrotik propose output rules and a diffrent way to achieve that.
