Mangle rules

mechcom · January 6, 2017, 7:09pm

Ok, long story short, I paid someone to fix my cloud core, TWICE. It still acting weird and it has to be something simple. The original problem had to do with load balancing. Below is a copy of my mangle rules. you can tell right off that there is inconsistency in the rules. I don’t understand what he was trying to do but it seems he change something on certain WAN connections but not all of them. Basically, I need to know which format is right.

To give background, what I’m noticing is that the load is not being shared even remotely even. One modem almost has to be completely full before the second begins the past traffic. If someone is familiar with this, please look at these rules and tell me if you see something wrong and what that is. If you wish, copy them, correct them, and re-paste them on here. if I can find a solution I would like to go back on the guy and explain what he did wrong.

/ip firewall mangle
add chain=prerouting src-address=10.10.0.1
add action=mark-routing chain=prerouting connection-mark=c1 new-routing-mark=to_P01-WAN1 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c2 new-routing-mark=to_P02-WAN2 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c3 new-routing-mark=to_P03-WAN3 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c4 new-routing-mark=to_P07-WAN7 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c5 new-routing-mark=to_P08-WAN8 passthrough=no src-address=10.10.0.0/22
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=c1 per-connection-classifier=both-addresses:5/0 src-address=
10.10.0.0/22
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=c2 per-connection-classifier=both-addresses:5/1 src-address=
10.10.0.0/22
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=c3 per-connection-classifier=both-addresses:5/2 src-address=
10.10.0.0/22
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=c4 per-connection-classifier=both-addresses:5/3 src-address=
10.10.0.0/22
add action=mark-connection chain=prerouting connection-state=new new-connection-mark=c5 per-connection-classifier=both-addresses:5/4 src-address=
10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c1 new-routing-mark=to_P01-WAN1 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c2 new-routing-mark=to_P02-WAN2 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c3 new-routing-mark=to_P03-WAN3 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c4 new-routing-mark=to_P07-WAN7 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=prerouting connection-mark=c5 new-routing-mark=to_P08-WAN8 passthrough=no src-address=10.10.0.0/22
add action=mark-routing chain=output connection-mark=c1 new-routing-mark=to_P01-WAN1 passthrough=no
add action=mark-routing chain=output connection-mark=c2 new-routing-mark=to_P02-WAN2 passthrough=no
add action=mark-routing chain=output connection-mark=c3 new-routing-mark=to_P03-WAN3 passthrough=no
add action=mark-routing chain=output connection-mark=c4 new-routing-mark=to_P07-WAN7 passthrough=no
add action=mark-routing chain=output connection-mark=c5 new-routing-mark=to_P08-WAN8 passthrough=no
add action=mark-connection chain=input connection-state=new in-interface=P01-WAN1 new-connection-mark=c1
add action=mark-connection chain=input connection-state=new in-interface=P02-WAN2 new-connection-mark=P02-WAN2_conn
add action=mark-connection chain=input connection-state=new in-interface=P03-WAN3 new-connection-mark=P03-WAN3_conn
add action=mark-connection chain=input connection-state=new in-interface=P07-WAN7 new-connection-mark=c4
add action=mark-connection chain=input connection-state=new in-interface=P08-WAN8 new-connection-mark=c5

razavim · January 6, 2017, 8:16pm

Actually, this export is meaning less untill you give more detail on your networking scenario or export other area of router such ass nat and rout as well.
and please take a look at mikrotik wiki on load balancing it has a very good examples that you could do and implement it all by your self.

Sent from my SM-N910C using Tapatalk

razavim · January 6, 2017, 8:17pm

sorry for my typo “as”

Sent from my SM-N910C using Tapatalk

MLubbe · January 6, 2017, 9:26pm

It looks like he is trying to achieve PCC Load-ballancing, although his way of working is very mixed up. I prefer a more logical approach and hopefully it will help you make sense of it all. I generally adapt and use this script to configure the mangle, nat & route rules, and have added notes to help you understand, which is key…

I have successfully configured and used it on 2-4 WAN connections already. (4 x DSL Routers as gateway’s, as well as 4 x PPPoE connections & combinations thereof)

You will need to adapt it slightly to match your configuration, and to increase it to 5 connections. Also find and replace “ether5” with your LAN interface… etc.

Accepts LAN to LAN Local Traffic.

add chain=prerouting dst-address=“10.10.0.0/22” in-interface=“ether5” action=accept comment=“Accept LAN Traffic”

Accepts LAN to Router Traffic (Otherwise you are unable to login to routers, as traffic is routed out balanced connections instead)

add chain=prerouting dst-address=“192.168.1.0/24” in-interface=“ether5” action=accept comment=“Accept Traffic to Router Network”
add chain=prerouting dst-address=“192.168.2.0/24” in-interface=“ether5” action=accept
add chain=prerouting dst-address=“192.168.3.0/24” in-interface=“ether5” action=accept
add chain=prerouting dst-address=“192.168.4.0/24” in-interface=“ether5” action=accept

To ensure connections don’t break we need to ensure WAN connections are replied to by correct Public IP / WAN Interface

Firstly we mark new connections inbound on WAN interfaces: [WAN->Router]

add chain=input in-interface=“ether1” connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn passthrough=yes comment=“Marks new WAN->Router connections inbound”
add chain=input in-interface=“ether2” connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn passthrough=yes
add chain=input in-interface=“ether3” connection-mark=no-mark action=mark-connection new-connection-mark=WAN3_conn passthrough=yes
add chain=input in-interface=“ether4” connection-mark=no-mark action=mark-connection new-connection-mark=WAN4_conn passthrough=yes

Marks traffic to go back out correct WAN interface, using appropriate routing table. [Router->WAN]

add chain=output connection-mark=WAN1_conn action=mark-routing new-routing-mark=WAN1 passthrough=no comment=“Marks Router->WAN responses outbound”
add chain=output connection-mark=WAN2_conn action=mark-routing new-routing-mark=WAN2 passthrough=no
add chain=output connection-mark=WAN3_conn action=mark-routing new-routing-mark=WAN3 passthrough=no
add chain=output connection-mark=WAN4_conn action=mark-routing new-routing-mark=WAN4 passthrough=no

Same principle applies to forwarded connections from LAN to WAN interface. [WAN->LAN]

add chain=forward in-interface=“ether1” connection-mark=no-mark action=mark-connection new-connection-mark=WAN1_conn passthrough=yes comment=“Marks new WAN->LAN connections inbound”
add chain=forward in-interface=“ether2” connection-mark=no-mark action=mark-connection new-connection-mark=WAN2_conn passthrough=yes
add chain=forward in-interface=“ether3” connection-mark=no-mark action=mark-connection new-connection-mark=WAN3_conn passthrough=yes
add chain=forward in-interface=“ether4” connection-mark=no-mark action=mark-connection new-connection-mark=WAN4_conn passthrough=yes

Marks traffic to go back out correct WAN interface, using appropriate routing table. [LAN->WAN]

add chain=prerouting connection-mark=WAN1_conn action=mark-routing new-routing-mark=WAN1 passthrough=no comment=“Mark LAN->WAN responses outbound”
add chain=prerouting connection-mark=WAN2_conn action=mark-routing new-routing-mark=WAN2 passthrough=no
add chain=prerouting connection-mark=WAN3_conn action=mark-routing new-routing-mark=WAN3 passthrough=no
add chain=prerouting connection-mark=WAN4_conn action=mark-routing new-routing-mark=WAN4 passthrough=no

Now we need to mark any LAN traffic outbound, so let’s create a jump rule.

add chain=prerouting in-interface=“ether5” connection-mark=no-mark action=jump jump-target=policy_routing comment=“Jump to custom policy routing chain”

PCC rules mark every unique combination of dst and src address with different connection mark (and will in turn, go out of a different WAN connection)

We now add rules to our “policy_routing” chain to ensure automatic per-address-pair load balancing.

add chain=policy_routing dst-address-type=!local per-connection-classifier=both-addresses:4/0 action=mark-connection new-connection-mark=WAN1_conn comment=“PCC Load-Balancing”
add chain=policy_routing dst-address-type=!local per-connection-classifier=both-addresses:4/1 action=mark-connection new-connection-mark=WAN2_conn
add chain=policy_routing dst-address-type=!local per-connection-classifier=both-addresses:4/2 action=mark-connection new-connection-mark=WAN3_conn
add chain=policy_routing dst-address-type=!local per-connection-classifier=both-addresses:4/3 action=mark-connection new-connection-mark=WAN4_conn

If you would like to ensure fail-over – enable “check-gateway” option in all default routes.

Finally we mark our seperated connections to go out the appropriate WAN interface.

add chain=prerouting connection-mark=WAN1_conn action=mark-routing new-routing-mark=WAN1 passthrough=no comment=“Mark routing for load-balanced connections”
add chain=prerouting connection-mark=WAN2_conn action=mark-routing new-routing-mark=WAN2 passthrough=no
add chain=prerouting connection-mark=WAN3_conn action=mark-routing new-routing-mark=WAN3 passthrough=no
add chain=prerouting connection-mark=WAN4_conn action=mark-routing new-routing-mark=WAN4 passthrough=no

Add our default masquerade rules

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ether1
add action=masquerade chain=srcnat out-interface=ether2
add action=masquerade chain=srcnat out-interface=ether3
add action=masquerade chain=srcnat out-interface=ether4

Add Routes for each WAN Connection

/ip route
add dst-address=0.0.0.0/0 gateway=“192.168.1.1” routing-mark=WAN1 distance=1
add dst-address=0.0.0.0/0 gateway=“192.168.2.1” routing-mark=WAN2 distance=1
add dst-address=0.0.0.0/0 gateway=“192.168.3.1” routing-mark=WAN3 distance=1
add dst-address=0.0.0.0/0 gateway=“192.168.4.1” routing-mark=WAN4 distance=1

If anyone has any improvements, please let me know.

Sob · January 7, 2017, 12:56am

The only clearly wrong thing I see are those two rules near the end that set “P02-WAN2_conn” and “P02-WAN2_conn” marks, because those are not used anywhere else. It should be “c2” and “c3”, to be consistent with the rest. But it can only break incoming connections from internet to router on those two WANs. It can’t influence LAN<->internet traffic.

Then you have rules #2-6 and #12-16 and both sets are exactly the same. I guess the idea was to save some cpu cycles for packets belonging to established connections. The question is, if it’s worth it (is your cpu anywhere near struggling?). It’s safe to remove rules #2-6, they don’t influence what happens, only how fast it happens. Personally, rather than having such duplicate rules, I’d move pcc to dedicated chain and jump to it for new connections.

How well is load balancing going to work, depends on how many internal hosts access how many externals servers. The more of both, the better it’s going to work. You have per-connection-classifier=both-addresses, which means that host A from LAN will access internet host X always using same WAN. And it’s generally good thing, because many servers don’t like each connection coming from completely different source address. It also means that if you had only A and B in LAN, accessing both only X and Y on internet, they will at most use only four WANs. That’s if you’re lucky, it might as well be only one. It depends on exact addresses (all A, B, X and Y). Another thing, each connection will transfer different amount of data (that’s obvious), so it may take just one big download to saturate one WAN.

mechcom · January 7, 2017, 8:46pm

sorry for the delay, I’ve tried no less than 500 combinations from different sources and documentations in how to do this. Needless to say, I remember why I paid someone else to try to fix this. I tried the new example of what you gave me and even knocked it down to two WAN connections just to make it easier to see what it was doing. I had no luck. My original configuration that I listed has a more serious problem but I thought. Not only is the load balancing nowhere close to being even, but it also causes a latency in our network. Which is to say, it’s almost like satellite whereas there’s a pause before it does anything. Within the next day or so I’m going to post a configuration or two that I’ve tried. I will simplify it as much as possible and perhaps someone will see something simple.

That said, my original post only showed my mangle rules and with the example that you posted I can see that my masquerade rules may have been different as well. The routes seem pretty straightforward however I’ve run into a few quirks with that as well. The thing I don’t understand is I can do the same thing 10 times from a brand new cloud core configuration and occasionally get different results. Simply put, these things seem very flaky to configure but pretty rock solid once configured correctly.

one of the things that was asked is what my WAN connections were and what my client base was. I have several of these in different locations in each network is a little different. Each cell has between two and five modems (most of which are 50 Mb down 5 mb up ). each one of those networks can have between 100 and 300 customers. Each customer having between three and 10 devices requesting access from web browser, e-mail, Netflix, etc. according to the cloud core, I average between 3000 and 5000 connection states at any one time. I’ll post more later. I really would like to find a solution to this. If I do I may even make a video or at least better documentation than what seems to be out there.