Mangle - how to do right?

upower3 · Thu May 07, 2015 12:07 pm

I'm implementing dual-home setup (which is quite easy and described a lot of times around, but mostly on 5.x), and what I miss is if I should mangle connections and then packets or just the connections, keeping in mind CPU efficiency.

I need to assign routing mark to some packets. Generally, the approach is simple (I numbered these rules to refer to them later):

(1) /ip firewall mangle add chain=forward protocol=tcp dst-address-list=acl_first connection-state=new action=mark-connection new-connection-mark=conn_first
(2) /ip firewall mangle add chain=forward connection-mark=conn_first action=mark-packet new-packet-mark=packet_first passthrough=no

This code based on wiki example ( http://wiki.mikrotik.com/wiki/Manual:IP ... ng_packets ). It should be an efficient as it we mark packets based on connection mark so the ACL-based decision is quite rare.

After that, I need to add routing-mark based on packet mark...

(3) /ip firewall mangle add action=mark-routing chain=prerouting new-routing-mark=route_first packet-mark=packet_first passthrough=no

...
right?

But here comes my two questions:

1. Do I really need to to do packet mark (rule #2) at all or I can do rule #3 based on connection mark? This way, I'll save a little of CPU by exclude rule #2.

2. How these rules be processed if rule #2 have passthrough=no on it? I should set passthrough=yes to be sure that rule #3 will ever work, right? In fact, #1 and #2 are in forward chain and #3 in prerouting, am I right with it? What if I put #1 and #2 to prerouting?

Thank you for your help in advance!

DLNoah · Mon May 11, 2015 3:17 pm

Your conn-mark and packet-mark rules should be in prerouting as well, and your packet-mark rule should allow passthrough.

If you only want to routing mark, and aren't trying to do queuing or something else that requires the packet-mark, then you should be able to disable the packet-mark rule. It's fairly easy to test by just disabling (rather than deleting) the rule, and if something breaks, you can turn it back on.

Mon May 11, 2015 5:57 pm

Don't confuse packet mark and connection mark. They're NOT the same.
Once a packet is connection marked, then all subsequent packets in that connection will be automatically connection marked by the connection tracking engine - no further checks in the mangle table will be required.

Packet marks must be accomplished by a rule chain traversal for each and every packet.

This is also true for route marking. (a third, separate mark)

If you have complicated traffic classification, it's best to move these rules into a separate custom chain (e.g. chain=classify) and in prerouting / output chains, use one check "connection-mark=no-mark" --> jump into "classify" chain.
The classification can be complicated and have lots of criteria, but once it's done, a connection is now classified and will be a very fast action for all subsequent packets. So no need to examine your complicated rules for each packet.

The idea is that packet marks and route marks must be done to every single packet. If the logic to create these is complicated, or sticky based on ingress/egress interface, etc - then the automatic connection marks are the way to quickly mark each packet individually.

(once you identify the train, it's easy to identify each car in the train)

upower3 · Wed May 13, 2015 11:41 am

Thank you for your answers, I really appreciate it!

But what I miss is: if I do connection-mark and then do route-mark based on connection mark, I'll be able to balance WAN load across two WANs. But in fact that will be for TCP only, isn't it?

So the question is:

It appears to me I still need to do
1. Connection-mark for "no-mark" new connections (so I'll save CPU for not marking every single packet)
2. Do packet-mark for UDP packets (for each of it).
3. Do route-mark based on marks set in 1 and 2 above.

Am I right?

Thank you again for you advices!

DLNoah · Wed May 13, 2015 2:20 pm

Your connection mark rule does not have to match only TCP protocol. You can remove the "protocol=tcp" match from that rule and it will still work as expected (emulating TCP connection states for non-TCP traffic).

upower3 · Wed May 13, 2015 3:21 pm

Your connection mark rule does not have to match only TCP protocol. You can remove the "protocol=tcp" match from that rule and it will still work as expected (emulating TCP connection states for non-TCP traffic).

So I do only connection marking, and then route marking based on these connections marks? For TCP connections it will work "natively", for non-connection-oriented protocols the "connections" will be emulated (I'm a bit surprised of; it that really true?) so everything will work fine?

DLNoah · Wed May 13, 2015 5:52 pm

That has been my experience with connection marking, yes. I'm usually connection marking in order to packet mark, but it should work the same with a routing mark.

Wed May 13, 2015 6:20 pm

So I do only connection marking, and then route marking based on these connections marks?

Yep. That's right.

For TCP connections it will work "natively", for non-connection-oriented protocols the "connections" will be emulated (I'm a bit surprised of; it that really true?) so everything will work fine?

Not quite - the TCP isn't working "natively" either.

Don't confuse a firewall's IP connection state tracking with TCP's stateful behavior. They're similar, but they're completely independent of each other. State tracking means that the router/firewall is keeping tabs on who's talking to whom, and whether it's been answered or not. It can track this for any protocol - udp, gre, ospf, ipsec, etc - and it does so with its own data structure, so it doesn't matter that TCP thinks it's in such-and-such of a state while udp says "state? what's that?".

e.g. - if a firewall sees that a TCP socket has been inactive for too long, it decides the connection must be dead. It will then remove the connection from its state tracking table. Thus the connection no longer exists as far as the firewall is concerned. Suppose the connection was really still there - suppose it was an ssh session, and the user went home for the day but didn't log out of the system. TCP on the client and server still consider the socket as established, so when the user gets there in the morning, the command prompt is still going to be on the screen. Whenever she presses a key, TCP will start sending that keystroke, and the firewall will see:
ssh packet from X to Y....
It's a new connection
It's not SYN request
Therefore this is evil, bad, naughty traffic - discard!

So the ssh client will see that TCP can't get any traffic through anymore, and will prompt the user that the connection is broken.

So - having said all of that, yes, connection tracking can make UDP have a "State" like tcp. This state is only in the "mind" of the router itself - it's not doing anything to UDP. The host and client won't notice anything different. It just lets your router track it more efficiently.

You know how you make a NAT rule that says to masquerade packets going to the Internet? Ever notice that you don't have to make an "un-masquerade" rule for the returning packets? That's because NAT makes a connections table and whenever packets arrive, they're compared to the NAT table before the nat rules are checked. If there's a match, there's no need to go through the rules again - the decision was already made.

Mangle's connection marking does the same thing. You might have some complicated logic to classify your connections - packets from such-and-such a host to some other specific port on some other specific host, but only if the rate exceeds limit 1 after 5pm, or rate limit 2 after 9pm, - blah blah - lots of rules... but once you've made the decision, you mark the CONNECTION so each and every packet is automatically recognized without having to check all of the rules again. (in both the original direction AND the replies as well)

Now each packet can be quickly stamped with packet marks or route marks based on which connection it's a part of.

If a train is going to Monte Carlo, then every car in the train is also going to Monte Carlo.... you don't have to make the decision for every car in the train.

I know this has been a long post with a lot of nit-picking details, but if you understand the concepts completely, then it lets you use the power of Mikrotik in any way you can imagine, and not just in a way that someone has posted a prefabricated how-to on the wiki.

upower3 · Wed May 13, 2015 7:04 pm

I know this has been a long post with a lot of nit-picking details, but if you understand the concepts completely, then it lets you use the power of Mikrotik in any way you can imagine, and not just in a way that someone has posted a prefabricated how-to on the wiki.

Thank you, really thank you for the explanation. The idea was familiar for me but its never late to repeat and add some details, isn't it?

Thank you!

What I still miss is if I mark connection which is 'new', should I also mark the same way all 'related' or 'established' connections from/to the same IP?

Thu May 14, 2015 12:52 am

What I still miss is if I mark connection which is 'new', should I also mark the same way all 'related' or 'established' connections from/to the same IP?

New just means it's the very first packet of a connection that the router has seen.
If you're matching "new" and in-interface=xxxx, then only new connections from that interface would get matched.
So if the connection originated from some other interface, going OUT interface xxxx, the first reply packet on that connection would not be a "new" connection so that, too, would be missed. In fact, the entire connection would go un-marked.

Of course, this behavior might be what you want, so it's not specifically right or wrong - it's just that if you want ALL connections to be marked with SOME mark or other, then using connection-tracking state = new is not necessarily going to get 100% of sockets.

If you also match established or related, then you're really not being very choosy at all. Pretty much anything that's not invalid.

You may have reasons for waiting until the connection is established before applying a connection mark... so again, it's not WRONG to do so - it's just less common.

upower3 · Thu May 14, 2015 1:33 pm

If you're matching "new" and in-interface=xxxx, then only new connections from that interface would get matched.
So if the connection originated from some other interface, going OUT interface xxxx, the first reply packet on that connection would not be a "new" connection so that, too, would be missed. In fact, the entire connection would go un-marked.

Clear explanation, thank you! So if I want to mark connections I should use form like

add action=mark-connection chain=prerouting connection-mark=no-mark new-connection-mark=MY-CONN

and not my previous form:

add action=mark-connection chain=prerouting connection-mark=no-mark connection-state=new new-connection-mark=MY-CONN

that is, skip mention 'new', right?

I was care for 'new' only to save some CPU cycles (which is indeed may not be so effective at all).

So, the right way to send traffic OUT to WAN1 or WAN2 depending from which hosts on local LAN it originates should be like this (LAN_ACL_1 holds hosts that will go thru WAN1, LAN_ACL_2 - via WAN2, LAN_IPs holds the whole LAN IP range):

1. Mark connection that is so far without any connection marks and which is from=LAN_IPs and to=LAN_IPs with mark=CONN_LOCAL and set passthrough='no'.
2. Mark connection that is still without any connection marks and which is from=LAN_ACL_1 with connection mark=HOSTS1_OUT, passthrough='yes'.
3. Set route mark for connections with connection mark=HOSTS1_OUT to route mark='HOSTS1_OUT_ROUTE', passthrough='no'.
4. Mark connection that is still without any connection marks and which is from=LAN_ACL_2 with connection mark=HOSTS2_OUT, passthrough='yes'.
5. Set route mark for connections with connection mark=HOSTS2_OUT to route mark='HOSTS2_OUT_ROUTE', passthrough='no'.

This way I'll have all of my connections that are originates from LAN be route-marked the proper way, right?

How I need to route-mark connections that are coming in from WAN1 and WAN2 (so I can 'publish' ports on host with my LAN to the outside):

6. Mark connection that is still without any connection marks and which is not from=LAN_IPs and interface_in='WAN1' with connection mark=WAN1_IN, passthrough='yes'.
7. Set route mark for connections with connection mark=WAN1_IN to route mark='HOSTS2_OUT_ROUTE', passthrough='no'.
8. Mark connection that is still without any connection marks and which is not from=LAN_IPs and interface_in='WAN2' with connection mark=WAN2_IN, passthrough='yes'.
9. Set route mark for connections with connection mark=WAN2_IN to route mark='HOSTS2_OUT_ROUTE', passthrough='no'.

right?

I can even remove lines 3 and 5, move 7 right before 9, and so do route marking only at 7 and 9 at the end of 'Mangle rules' list.

Thu May 14, 2015 2:45 pm

I can even remove lines 3 and 5, move 7 right before 9, and so do route marking only at 7 and 9 at the end of 'Mangle rules' list.

Sounds like you've got it!
You'd need to use the same connection mark name for both the incoming and outgoing connections which you're marking, so that only 2 route mark rules are necessary.

I.e. instead of HOSTS1_OUT / WAN1_IN --> WAN1
Then route mark all packets of connections WAN1 with route mark HOSTS1_OUT_ROUTE

It makes the router use less rules to check when you use the same connection mark for both directions.
Technically, you only need to route-mark the outbound packets anyway, but it's probably just as fast to mark all of them with less checks than to add a check and only actually mark outbound packets.

Another way to speed things up is to move the connection marking rules into a custom chain (newconmark), and making the first preroute chain rule = connection-mark=no-mark action=jump jump-target = newconmark
The last rule of newconmark = return.

This way, only one rule is checked per packet whenever the connection marks exist. (the vast majority of the packets)

upower3 · Thu May 14, 2015 5:12 pm

Sounds like you've got it!

I really hope so!

Thank you for your guidance!

But if I have mangle rules and routing rules to use two WAN, it'll work fine. But what if I have SMTP server inside my LAN? I'll set up NAT to pass packets from 25/tcp of WAN1 or WAN2 to 25/tcp of SMTP_IP. Won't returning packets be routed directly back to the incoming WAN interface since every "in" packets will have record in the NAT table so reply will be on the same interface?

Another way to speed things up is to move the connection marking rules into a custom chain (newconmark), and making the first preroute chain rule = connection-mark=no-mark action=jump jump-target = newconmark
The last rule of newconmark = return.

This way, only one rule is checked per packet whenever the connection marks exist. (the vast majority of the packets)

Great idea and nice example how to use another feature that was known to me but the one I've never imagine how to use it in real life (never needed any)!

But am I right, I'll need to set passthrough to 'yes' in every rule in this extra chain? If I won't do that the rule processing will quit at the rule that match and no 'return' rule be executed?

Thu May 14, 2015 5:30 pm

But am I right, I'll need to set passthrough to 'yes' in every rule in this extra chain? If I won't do that the rule processing will quit at the rule that match and no 'return' rule be executed?

Yes, but the connection mark rules should be passthrough anyway, so nothing's really changed due to this.
In fact, you still need to check "connection-mark=no-mark" on all of them as well - what if rule 1 marks the connection but rule 4 would also match? If rule 4 includes a check for no-mark, then it won't override the rule1 mark. The entire purpose of connection mark is this 'sticky' concept that once marked, that's it. The choice is made.
(of course, as always, you may WANT something to override - so there's no right/wrong here)

Great idea and nice example how to use another feature that was known to me but the one I've never imagine how to use it in real life (never needed any)!

Custom chains are awesome.
If you have 4 lans, and want to give them all the same outbound policy rules, for instance, you can make 4 jump rules in the forward filter table "in-interface=lan1 out-interface=wan1 action=jump jump-target=out_policy" where in out_policy you can allow several ports, put a p2p check rule, etc. Otherwise, each check in out_policy would have to be in forward chain 4 times.

The other nice thing about chains is that they group rules together in a way that makes your firewall more readable. If you have a chain that filters access to NAT pinhole services, then you can ignore those rules when you're thinking about outbound traffic. (the view filter drop-down at the top-right corner of the firewall window is very useful)

upower3 · Thu May 14, 2015 6:00 pm

So if I write (rule numbers are mine, not in export

):

/ip firewall mangle
(0) add action=jump chain=prerouting connection-mark=no-mark jump-target=_new-conn-mark
(1) add action=mark-routing chain=prerouting connection-mark=conn_WAN1 new-routing-mark=route_WAN1 passthrough=no
(2) add action=mark-routing chain=prerouting connection-mark=conn_WAN2 new-routing-mark=route_WAN2 passthrough=no
(3) add action=mark-connection chain=_new-conn-mark dst-address-list=LAN-IPs new-connection-mark=LOCAL-CONN passthrough=no src-address-list=LAN-IPs
(4) add action=mark-connection chain=_new-conn-mark new-connection-mark=conn_WAN1 src-address-list=LAN-WAN1
(5) add action=mark-connection chain=_new-conn-mark new-connection-mark=conn_WAN2 src-address-list=LAN-WAN2
(6) add action=return chain=_new-conn-mark

so very first packet in connection (let's say it's from LAN to LAN) will go 1 -> 3 (the rule will be executed) -> exit (no more rules be executed, both in _new-conn-mark and in prerouting chains).

If the packed be sources from IP on LAN-WAN1 access list then it will go 1 -> 3 -> 4 (executed) -> 5 -> 6 -> return to 2 -> exit.

Do I understand it right?

Or the way in first case be 1 -> 3 -> (exit from _new-conn-mark back to prerouting chain) -> 2 -> "global" exit ?

Thu May 14, 2015 6:25 pm

Bold is my changes to your post. (I think you changed rule numbers while posting, and didn't update your "traces")

so very first packet in connection (let's say it's from LAN to LAN) will go 0 -> 3 (the rule will be executed) -> exit (no more rules be executed, both in _new-conn-mark and in prerouting chains).

Execution stops on rule 3 because rule 3 has passthrough=no. No more rules are checked.
Note that if pass-thru was allowed, then rule 4 or 5 would match (I assume that all of your LAN IPs are in one or the other of your policy lists) and the packets would end up getting route-marked anyway.

If the packed be sources from IP on LAN-WAN1 access list then it will go 0 -> 3 -> 4 (executed) -> 5 -> 6 -> return to 1 (executed)-> exit.

Do I understand it right?

Pretty much! I added the "executed" on rule 1 for clarity.

Or the way in first case be 1 -> 3 -> (exit from _new-conn-mark back to prerouting chain) -> 2 -> "global" exit ?

No, you got it right above.

The last thing I'd say is that on rules 4 and 5, you should probably add "connection-mark=no-mark" as a criteria. Note that if you forgot to disable passthrough on rule 3, rule 4 or 5 would overwrite your LOCAL-CONN mark. With this extra criteria, they won't override rule 3 anyway. It seems more expensive, but remember that this only happens once per new connection. All subsequent packets just go 0 -> 1 -> match+exit, 0 -> 1 -> 2 -> match+exit, or 0 -> 1 -> 2 -> end of chain exit.

upower3 · Thu May 14, 2015 6:55 pm

I think you changed rule numbers while posting, and didn't update your "traces"

Yes, my bad. Thank you!

The last thing I'd say is that on rules 4 and 5, you should probably add "connection-mark=no-mark" as a criteria. Note that if you forgot to disable passthrough on rule 3, rule 4 or 5 would overwrite your LOCAL-CONN mark. With this extra criteria, they won't override rule 3 anyway. It seems more expensive, but remember that this only happens once per new connection. All subsequent packets just go 0 -> 1 -> match+exit, 0 -> 1 -> 2 -> match+exit, or 0 -> 1 -> 2 -> end of chain exit.

Got that too. This is why I put passthrough to "no" on rule 3 (in hope that "execution" of rule 3 would return 'execution point' to 'previous' chain, that is, to rule 1). Since my hope was wrong I'll definitely add "connection-mark=no-mark" as a criteria to 4 and 5.

So new chain won't remove many checks but it surely beautify the listing.

You see I used to treat such a rule lists as kind of 'program' that has some 'execution point', has some jumps between lines, and have some subroutines (which, as I wrongly assumed, would return to the jump point as subroutine 'code' execution breaks). Wrong approach, I see.

Thu May 14, 2015 7:11 pm

So new chain won't remove many checks but it surely beautify the listing.

As Margie says in Fargo: "Uhhhh, not so sure I agree with your police work there, Lou."

Actually, it removes a bazillion checks - because almost all of the packets your router will process are subsequent packets in established connections, and none of those will jump into the chain at all.

Without using the chain, each and every packet would have to go:
am I unmarked and lan to lan? No
am I unmarked and from lan group 1? No
am I unmarked and from lan group 2? No
am I marked wan1? yes (action)

Now they go:
am I unmarked? No
am I marked wan1? yes (waction)

Also - if you're a millisecond miser (heh) you could make rule #1 be route-mark connection-mark=wan1, rule #2 be route-mark connection-mark=wan2, and rule 3 be jump into classify, and then rules 4 and 5 be copies of rule 1 and 2.... which would remove the "am I unmarked" check for every packet - but seriously, is it going to make THAT much difference by shaving off that one more check?

You see I used to treat such a rule lists as kind of 'program' that has some 'execution point', has some jumps between lines, and have some subroutines (which, as I wrongly assumed, would return to the jump point as subroutine 'code' execution breaks). Wrong approach, I see.

Actually, I draw on my comp-sci training quite heavily in my firewall thinking. The difference is that in netfilters, some actions implicitly end processing, and some don't. In the mangle table, it's even more nebulous because you can optionally continue or not. (e.g. If you wanted to do QoS marking as well, you'd have to allow processing to continue after the route marking rules, because you'll also need to packet mark the individual packets.)

I was a tad surprised to find that a custom chain has an implicit "return" rule at the end, though. I still put them just because it's good to be explicit.

upower3 · Fri May 15, 2015 11:11 am

Now, just to finish this Friday lesson with something useful

let me ask the rest of my silly questions.

The general approach to load-balance of dual-home router is this:

1. Set up WAN1, WAN2 and LAN, both interfaces and IPs. In my case, WANs are PPPoE links (which leads to using interfaces names instead of IPs of local and remote link ends).
2. Set up NAT masquerading so traffic from LAN would go outside but no set output interface in this NAT rule.
3. Set up mangle rules like we discussed above to connection-mark traffic from LAN-WAN1 and LAN-WAN2 acls, then route-mark these packets (we do route-mark packets, right?) after these connection marks (say, with route_WAN1 and route_WAN2 marks)
4. Set up route tables so packets marked as route_WAN1 would route to WAN1 as distance=1 and to WAN2 as distance=2, and vise versa table for route_WAN2 packets.
5. Set up another ('default' or 'no mark') routing table and set up it the way we prefer un-route-marked traffic would go outside.
6. Set up if-uplink-alive checks so we'll be sure that we can use WAN1 and/or WAN2 to reach 0.0.0.0/0. If we don't have a BGP (and we don't) I'd prefer to use netwatch and scripts on check success and fail.

At this point we have dual-WAN internet connectivity balanced accordingly to LAN-WAN1 and LAN-WAN2 access-lists.

Now we need to provide the ability to return back traffic from outside that sent to our internal resources (SMTP server as an example) which were made available via NAT. So I mangle in input (or forward?) chain the connections that fro interface-in=WAN1 (or WAN2) and finally route-mark it as route_WAN1 (route_WAN2). This part a bit mysterious for me as I expert it should be done 'automatically' due to NAT connection track table.

Take a look, please, what would you say, where I'm wrong or should extra polish?

Thank you!

pukkita · Fri May 15, 2015 1:00 pm

Great discussion here

2. Set up NAT masquerading so traffic from LAN would go outside but no set output interface in this NAT rule.

You need to specify an output interface for masquerading to be useful in this context. NAT is the last step, routing happens before src-nat, so all the "manipulations" are done previously by modifying routing.

4. Set up route tables so packets marked as route_WAN1 would route to WAN1 as distance=1 and to WAN2 as distance=2, and vise versa table for route_WAN2 packets.
5. Set up another ('default' or 'no mark') routing table and set up it the way we prefer un-route-marked traffic would go outside.
6. Set up if-uplink-alive checks so we'll be sure that we can use WAN1 and/or WAN2 to reach 0.0.0.0/0. If we don't have a BGP (and we don't) I'd prefer to use netwatch and scripts on check success and fail.

No need for any checks or netwatch. You set up already two default routes for each routing mark in step 4. Specify "ping" or ARP depending on situation in the check-gateway tab for ROS to monitor that gateway and it will mark instantly the route as inactive so the next distance route will be used.

Now we need to provide the ability to return back traffic from outside that sent to our internal resources (SMTP server as an example) which were made available via NAT. So I mangle in input (or forward?) chain the connections that fro interface-in=WAN1 (or WAN2) and finally route-mark it as route_WAN1 (route_WAN2). This part a bit mysterious for me as I expert it should be done 'automatically' due to NAT connection track table.

Input is for connections TO the router, forward for traffic flowing through it so it would be forward. Prerouting happens before both, so better use prerouting and you will be including also traffic from outside to the router.

upower3 · Fri May 15, 2015 3:19 pm

Great discussion here

Indeed!

No need for any checks or netwatch. You set up already two default routes for each routing mark in step 4. Specify "ping" or ARP depending on situation in the check-gateway tab for ROS to monitor that gateway and it will mark instantly the route as inactive so the next distance route will be used.

No, I've faced too many time when ISP gw is reachable, the link electrically (logically) is up, but the internet is unreachable. Some ISP also block users (say, for lack of payment) by assigning them private IP instead of 'white' one so this user may only browse ISP site and payment portal.

So unless you check that several remote hosts are reachable you can not assume the link is OK to be used as Internet link

Pinging remote hosts (say, web server of some huge company) is risky (due to fact you need to trust that the host is always up which is not the case in every situation), and have its drawbacks like 'slow reaction' (if you ping once a minute the router won't know that link is dead until next ping), ping lost due to high bandwidth utilization (you can play with priority and so on but you can't control the whole way forth and back to the remote host), but I have no other idea to analyze if the link is alive (if we don't have BGP, and we won't analyze the traffic on the link itself for TCP flags etc.)

P.S. BGP itself is not the solution either but it bring you some information from outside and from remote routers which may be good to use later.
P.P.S. Don't trust pinging public anycast DNS (link Google's one, 8.8.8.8 and 8.8.4.4) since due to its anycast nature sometimes they works but wont' answer the ping.

Fri May 15, 2015 6:04 pm

I'm going to give the extra polish you requested - but first, let me take a moment to just say this one thing:

I HATE - I mean, really HATE pinging things as a routing decision point.

It is necessary in situations (like yours) where dynamic routing is not an option, but it's chatty, in the big picture it wastes tons of bandwidth, and places all kinds of burden on "well-known" addresses. You could be nice and ping something a little less obvious, such as k.root-servers.net or some other IP in their anycasted /24, but I guarantee that hundreds of thousands of other administrators are doing exactly the same thing....

If I had to ping something, it would be things which I know to be anycasted (reliability), and I would only ping 1 per link because hey, if it goes down, you're just going to fail over to another interface, right? The only well-known anycast address that I've ever seen "go dark" was 4.2.2.2 and it's happened on a few occasions. I've never seen google's addresses go dark.

Anyway - time for me to put my helpful, un-grouchy face back on.

The general approach to load-balance of dual-home router is this:

1. Set up WAN1, WAN2 and LAN, both interfaces and IPs. In my case, WANs are PPPoE links (which leads to using interfaces names instead of IPs of local and remote link ends).
2. Set up NAT masquerading so traffic from LAN would go outside but no set output interface in this NAT rule.
3. Set up mangle rules like we discussed above to connection-mark traffic from LAN-WAN1 and LAN-WAN2 acls, then route-mark these packets (we do route-mark packets, right?) after these connection marks (say, with route_WAN1 and route_WAN2 marks)
4. Set up route tables so packets marked as route_WAN1 would route to WAN1 as distance=1 and to WAN2 as distance=2, and vise versa table for route_WAN2 packets.
5. Set up another ('default' or 'no mark') routing table and set up it the way we prefer un-route-marked traffic would go outside.
6. Set up if-uplink-alive checks so we'll be sure that we can use WAN1 and/or WAN2 to reach 0.0.0.0/0. If we don't have a BGP (and we don't) I'd prefer to use netwatch and scripts on check success and fail.

I agree with this list 99%. #3 especially shows you've got the idea right.

Regarding #2, Pukkita pointed out that you should specify the interface in order for masquerade to work. That's not entirely true either. I definitely prefer to use interfaces as the criteria for the masquerade rules, but I would use whatever scheme requires the fewest rules....

If I had only one LAN interface, I would just say out-interface=!lan action=masquerade.
This one rule would work for any arbitrary number of WAN interfaces without ever having to be touched again, even if I were adding and removing WAN interfaces constantly in real time. (the MASQUERADE table would be an entirely different story though - heh)

If I had multiple WAN and LAN interfaces, I would just put out-interface=wan1 action=masquerade / out-interface=wan2 action=masquerade.

Now we need to provide the ability to return back traffic from outside that sent to our internal resources (SMTP server as an example) which were made available via NAT. So I mangle in input (or forward?) chain the connections that fro interface-in=WAN1 (or WAN2) and finally route-mark it as route_WAN1 (route_WAN2). This part a bit mysterious for me as I expert it should be done 'automatically' due to NAT connection track table.

If you're connection-marking in prerouting, then these cases are already taken care of. Forget forward and input chains here - prerouting happens before both of these, no matter which one is coming next. It took me a while to break the habit too.

About the only thing you lack for completeness in the sticky-path solution is connections which the Mikrotik originates itself. This is done with the output chain. Of course packet marking in output is the same as packet marking in prerouting (mark based on connection mark) but there are a few philosophies you could use for your connection marks. The simplest and cleanest (in my way of thinking, anyway) is to use the source IP address as your switch, because the router has actually already chosen the out-interface and thus the appropriate source-IP. The connection mark just makes it set in stone, so if wan1 dies, the connection riding it should die too - not get switched to wan2 and the srcnat screwing things up there.

Here's a link to a complete solution with load sharing, redundancy, etc - the whole works. I actually was in attendance at this MUM presentation, and enjoyed it quite a bit: (it's a shame the talk itself wasn't preserved, there were great explanations and discussions about the various moving parts and what they did and why they are there)
http://mum.mikrotik.com/presentations/US12/tomas.pdf

This wiki article covers the "ping past default GW" case you (correctly) mentioned in the discussion:
http://wiki.mikrotik.com/wiki/Advanced_ ... _Scripting

upower3 · Fri May 15, 2015 7:00 pm

http://mum.mikrotik.com/presentations/US12/tomas.pdf

Just as I reading this good papers I found very unknown thing here (page 38):

/ip firewall mangle 
add chain=prerouting connection-mark=no-mark src-address-list=LAN
dst-address-list=!Connected dst-address-type=!local action=mark-connection
new-connection-mark=LAN->WAN

that is, dst-address-type=!local. Never seen this before, and very interested why it here.

Keep reading...

Fri May 15, 2015 7:26 pm

http://mum.mikrotik.com/presentations/US12/tomas.pdf

Just as I reading this good papers I found very unknown thing here (page 38):
Code: Select all
/ip firewall mangle 
add chain=prerouting connection-mark=no-mark src-address-list=LAN
dst-address-list=!Connected dst-address-type=!local action=mark-connection
new-connection-mark=LAN->WAN
that is, dst-address-type=!local. Never seen this before, and very interested why it here.

Keep reading...

I just learned that on the forums recently as well - it essentially means "any IP address which will end up in the input chain"
i.e. if the dst-address is an IP address of the Mikrotik itself.
(other types are unicast, multicast, broadcast) - unicast = "not me, but a normal IP" - multicast and broadcast = obvious.

pukkita · Sat May 16, 2015 2:33 pm

At first glance one would think in this rule will have the same exact effect using forward, cannot stop thinking there should be a reason for this but cannot find where's the subtlety?

DLNoah · Mon May 18, 2015 3:10 pm

At first glance one would think in this rule will have the same exact effect using forward, cannot stop thinking there should be a reason for this but cannot find where's the subtlety?

The most important reason to do it in pre-routing is that pre-routing mangle rule will fire before NAT, whereas forward will occur after. So, especially for things like "make sure Internal device X goes one specific way" or "load balance traffic for internal device Y", you have to perform your connection marking and route marking before the routing decision is made and the source IP of the packet has been re-written.

pukkita · Mon May 18, 2015 3:41 pm

Guess you mean for dst-nat, AFAIK src-nat happens way after input / forward?

DLNoah · Mon May 18, 2015 3:54 pm

Oops, I guess you're right there. I shouldn't be answering forum posts first thing Monday morning

pukkita · Mon May 18, 2015 4:01 pm

never mind!

upower3 · Mon May 18, 2015 7:06 pm

I HATE - I mean, really HATE pinging things as a routing decision point.

Can say more on my side, either, but I don't know the better way. Yes there are solutions that can "understand" suboptimal link (uplink) work by analyzing tcp/udp traffic/connections by observing link, but this is all beyond our scope of hardware (and money), right?

I just don't know the way to check if some remote hosts are good visible to us. Much like host from US can not 'see' server in EU, but at the same time all other Internet may be visible to the US host. What's worst even ISP don't know if the visibility of Internet is good or not. If we use BGP we can at lest monitor if we have path to some remote AS but this won't answer us if we have good link with hosts on remote, or the link is poor and slow.

Common problem. If we can't get quality metrics from passive observation the only thing we can do is to use active probes like ping. Say but still true. (

upower3 · Tue May 19, 2015 4:48 pm

Here's a link to a complete solution with load sharing, redundancy, etc - the whole works.

It come surprise for me that some approaches include use of /ip route rules, while other are not.

Say, this kind of scheme said to be good (2 LANs and 2 WANs):

# LANs
/ip address add address=192.168.88.1/30 interface=ether3
/ip address add address=192.168.89.1/30 interface=ether4

# WANs
/ip address add address=1.1.1.2/30 interface=ether1
/ip address add address=2.2.2.2/30 interface=ether2

# Outside routes
/ip route add gateway=1.1.1.1 routing-mark=ISP1
/ip route add gateway=2.2.2.1 routing-mark=ISP2

/ip route rule add dst-address=192.168.88.0/24 action=lookup table=main
/ip route rule add dst-address=192.168.89.0/24 action=lookup table=main
/ip route rule add dst-address=1.1.1.0/30 action=lookup table=main
/ip route rule add dst-address=2.2.2.0/30 action=lookup table=main
/ip route rule add src-address=1.1.1.0/30 action=lookup table=ISP1
/ip route rule add src-address=2.2.2.0/30 action=lookup table=ISP2
/ip route rule add routing-mark=ISP1 action=lookup table=ISP1
/ip route rule add routing-mark=ISP2 action=lookup table=ISP2

/ip firewall mangle add chain=prerouting src-address=192.168.88.0/24 action=mark-routing new-routing-mark=ISP1 passthrough=no
/ip firewall mangle add chain=prerouting src-address=192.168.89.0/24 action=mark-routing new-routing-mark=ISP2 passthrough=no
/ip firewall mangle add chain=prerouting action=mark-routing new-routing-mark=ISP2 passthrough=no

/ip firewall nat add chain=srcnat action=masquerade out-interface=ether1 src-address=192.168.88.0/24
/ip firewall nat add chain=srcnat action=masquerade out-interface=ether2 src-address=192.168.89.0/24

And "/ip route rule"s are used to set which route tables to use for a specific traffic. As for me, I used this approach rarely so as I compare this to what you've written then I try to figure out which one is the robust one.

Just to compare, here is the configuration from the PDF you've given me (it was a bit tricky to copy it):

#
/interface ethernet
set 0 name=LAN
set 3 name=ISP_1
set 4 name=ISP_2

/ip address
add address=192.168.22.1/24 interface=LAN
add address=1.1.1.32/24 interface=ISP_1
add address=2.2.2.65/24 interface=ISP_2

/ip firewall nat
add action=masquerade chain=srcnat out-interface=ISP_1
add action=masquerade chain=srcnat out-interface=ISP_2

/ip route
add gateway=1.1.1.1 distance=1
add gateway=2.2.2.1 distance=2
add gateway=1.1.1.1 routing-mark=ISP1_Route distance=1
add gateway=2.2.2.1 routing-mark=ISP2_Route distance=1

/ip firewall address-list
add address=1.1.1.0/24 list=Connected
add address=2.2.2.0/24 list=Connected
add address=192.168.22.0/24 list=Connected
add address=192.168.22.0/24 list=LAN

/ip firewall mangle
add chain=prerouting src-address-list=Connected dst-address-list=Connected action=accept

/ip firewall mangle
add chain=input connection-mark=no-mark in-interface=ISP_1 action=mark-connection new-connection-mark=WAN1->ROS
add chain=input connection-mark=no-mark in-interface=ISP_2 action=mark-connection new-connection-mark=WAN2->ROS
add chain=output connection-mark=WAN1->ROS action=mark-routing new-routing-mark=ISP1_Route 
add chain=output connection-mark=WAN2->ROS action=mark-routing new-routing-mark=ISP2_Route


/ip firewall mangle
add chain=forward connection-mark=no-mark in-interface=ISP_1 action=mark-connection new-connection-mark=WAN1->LANs
add chain=forward connection-mark=no-mark in-interface=ISP_2 action=mark-connection new-connection-mark=WAN2->LANs
add chain=prerouting connection-mark=WAN1->LANs src-address-list=LAN action=mark-routing new-routing-mark=ISP1_Route
add chain=prerouting connection-mark=WAN2->LANs src-address-list=LAN action=mark-routing new-routing-mark=ISP2_Route

/ip firewall mangle
add chain=prerouting connection-mark=no-mark src-address-list=LAN dst-address-list=!Connected dst-address-type=!local action=mark-connection new-connection-mark=LAN->WAN
add chain=prerouting connection-mark=LAN->WAN src-address-list=LAN action=mark-routing new-routing-mark=ISP1_Route comment="Load-Balancing here"
add chain=prerouting connection-mark=LAN->WAN routing-mark=ISP1_Route action=mark-connection new-connection-mark=Sticky_ISP1
add chain=prerouting connection-mark=LAN->WAN routing-mark=ISP2_Route action=mark-connection new-connection-mark=Sticky_ISP2
add chain=prerouting connection-mark=Sticky_ISP1 src-address-list=LAN action=mark-routing new-routing-mark=ISP1_Route
add chain=prerouting connection-mark=Sticky_ISP2 src-address-list=LAN action=mark-routing new-routing-mark=ISP2_Route

Nollitik · Thu May 28, 2015 11:01 am

Awesome discussion...thanks for sharing

Thu May 28, 2015 5:31 pm

And "/ip route rule"s are used to set which route tables to use for a specific traffic. As for me, I used this approach rarely so as I compare this to what you've written then I try to figure out which one is the robust one.

Configuration 1 is basically a hard-wired configuration where LAN1 -> ISP1, and LAN2 -> ISP2, and nobody can use the opposite one. (It's a strict policy-based routing example)

Configuration 2 is a load-sharing configuration. It wants to use ISP1 first and if it's down or above some threshold of load, it will switch to using ISP2. The nice feature of this second configuration is that it can be easily extended to any arbitrary number of ISP links, and also that it easily balances links which have different speeds.

Thanks for posting example 1 - because I just learned something here:
In most examples I've seen in the forums, if packets in policy ISP1 need to ping the default GW on ISP2, it's always been suggested to simply add dst=2.2.2.0/30 gateway=ISP2 routing-mark=ISP1 - which makes a copy of the ISP2 link's addresses available to the ISP1 routing table. This has been the fix for check-gateway in lots of cases....

But route rules are so much more elegant, and I've never even looked in that tab before. I tried to find a decent manual page for this feature, but it's buried in other howtos. Rules let you override the route-mark lookup behavior based on whatever criteria. I'm going to play with this in GNS3 and probably going to emerge as the new advocate of route rules.

h17 · Fri May 29, 2015 10:25 pm

Thank you for this thread. It helped me to optimize my mangle rules,
but I'm still trying to understand how to use policy routing _and_ QoS (queue-tree)
both based on connection marking.

Currently I have policy routing and load-balancing based on conn-marks and route-marks.
Unfortunately, all QoS rules are still based entirely on packet-marking without any help
from conn-marks. Is it even possible in my case?

If I use conn-mark in prerouting, then (based on it) I set route-mark,
is there any way to reuse conn-mark to reduce average number of QoS rules
every packet is tested by?

Assuming I have:
- 2 ISP uplinks
- 2 LAN downlinks
- 4 classes/priorities in my QoS tree per each uplink
- quirky src-routing rules to balance my outgoing traffic
- no dynamic routing

How would you approach to QoS mechanism to make it cpu less intensive?

My first thought was to make 2x4 conn-mark rules to distinguish both WAN port and QoS class,
but then I'd have to make more rules to parse them later. Wrong way to do it, I guess.

Sat May 30, 2015 7:05 am

I am also interested in this. I would like to be able to use more than one mark for packets and connections at once. I will probably make a request for v.7 but I don't believe in it.

MikRu · Wed Apr 06, 2016 2:35 pm

Here's a link to a complete solution with load sharing, redundancy, etc - the whole works.

But where is ROS-WAN1,2 mangles in that manual?

When I apply that config,
1.WAN-ROS and ROS-WAN get the same conn-mark

rules are just the same as in PDF.

ros 6.34.4

 0    ;;; 1.1.1 WAN1->ROS
      chain=input action=mark-connection new-connection-mark=WAN1->ROS passthrough=yes 
      in-interface=ether2-master-local connection-mark=no-mark log=no log-prefix="" 

 1    ;;; 1.1.2 WAN2->ROS
      chain=input action=mark-connection new-connection-mark=WAN2->ROS passthrough=yes 
      in-interface=pppoe-out1 connection-mark=no-mark log=no log-prefix="" 

 2    ;;; 1.2.1 WAN1->ROS
      chain=output action=mark-routing new-routing-mark=ISP1_Route passthrough=yes 
      connection-mark=WAN1->ROS log=no log-prefix="" 

 3    ;;; 1.2.2 WAN2->ROS
      chain=output action=mark-routing new-routing-mark=ISP2_Route passthrough=yes 
      connection-mark=WAN2->ROS log=no log-prefix="" 

 4    ;;; 2.1.1 WAN1->LANs
      chain=forward action=mark-connection new-connection-mark=WAN1->LANs passthrough=yes 
      in-interface=ether2-master-local connection-mark=no-mark log=no log-prefix="" 

 5    ;;; 2.1.2 WAN2->LANs
      chain=forward action=mark-connection new-connection-mark=WAN2->LANs passthrough=yes 
      in-interface=pppoe-out1 connection-mark=no-mark log=no log-prefix="" 

 6    ;;; 2.2.1 WAN1->LANs
      chain=prerouting action=mark-routing new-routing-mark=ISP1_Route passthrough=yes 
      src-address-list=LAN connection-mark=WAN1->LANs log=no log-prefix="" 

 7    ;;; 2.2.2 WAN2->LANs
      chain=prerouting action=mark-routing new-routing-mark=ISP2_Route passthrough=yes 
      src-address-list=LAN connection-mark=WAN2->LANs log=no log-prefix="" 

 8    ;;; 3.1.0 LAN->WAN0
      chain=prerouting action=mark-connection new-connection-mark=LAN->WAN passthrough=yes 
      dst-address-type=!local src-address-list=LAN dst-address-list=!Connected connection-mark=no-mark 
      log=no log-prefix="" 

 9    ;;; 4.1.0.Load-Balancing here
      chain=prerouting action=mark-routing new-routing-mark=ISP1_Route passthrough=yes 
      src-address-list=LAN connection-mark=LAN->WAN log=no log-prefix="" 

10    ;;; 5.1.1. Stick connections after 4.1.0
      chain=prerouting action=mark-connection new-connection-mark=Sticky_ISP1 passthrough=yes 
      routing-mark=ISP1_Route connection-mark=LAN->WAN log=no log-prefix="" 

11    ;;; 5.1.2. Stick connections after 4.1.0
      chain=prerouting action=mark-connection new-connection-mark=Sticky_ISP2 passthrough=yes 
      routing-mark=ISP2_Route connection-mark=LAN->WAN log=no log-prefix="" 

12    ;;; 5.2.1. Stick connections after 4.1.0
      chain=prerouting action=mark-routing new-routing-mark=ISP1_Route passthrough=yes 
      src-address-list=LAN connection-mark=Sticky_ISP1 log=no log-prefix=""

MikRu · Wed Apr 06, 2016 5:15 pm

it seems i've got it. need some time to compare configs.

Bytezone · Mon May 23, 2016 8:11 pm

Guys.. can anybody enlighten me on how to adapt Mr. tomas script to work on pppoe client by bridging modems especially the mangle section

..

Internet connection is so terrible in my country i really suffer from repeatedly internet disconnection from my ISP.

DaKater · Sun Jul 30, 2017 10:49 pm

My head is exploding. I read thru this post like 10x, maybe 20x times. I have a very special setup and I cannot copy any tutorial, because no tutorial will fit.
At the end its a basic static routing problem with mark connection.

I hope ZeroByte or any other high reputated user can help me out. I am almost crying.

R1
ether1 ->
public static ip from ISP: 99.66.55.161 and ofc the default route to come online (outgoing) for any packet to the internet.
routed in another whole net 77.66.40.0/21 (the problem maker!)

***for testing only*** I can map the 77.66.40.1/21 to ether2 and all connected systems can use an ip from 77.66.40.2 up 77.66.47.255***
***but this was just a test, the network is not connected to this port anymore for next hop routing***

R2 (in another place with an ISP Cable modem)
ether 1 ->
public static ip from ISP 99.88.77.38 (whatever) and ofc the default route to come online for any packet to the internet. (we have some more static IPs, but does not matter for this)

SSTP successfull etablished between both sites and tested
sstpr1-r2
ip adresses are local 10.10.10.1 for R1 and 10.10.10.2 for R2 to reach each other.

***tested and ping is working, via LDP neighbours can be seen***

R1
Route 77.66.40.0/21 -> 10.10.10.2 reachable sstpr1-r2

R2
77.66.40.1/21 (ether2)

*** tested and working, I can reach and ping 77.66.40.1 thru the sstp tunnel ***

R2
MANGLE RULE, because I want the packets flow back thru sstp tunnel.
Chain: prerouting (check)
Dst. Adress: 77.66.40.0/21 (check)
In-Interface: sstpr1-r2 (check)
Action: mark conneciton
New Connection Mark: sstpr1-r2mark (check)
Passthrough: Yes (check)

And of course the fitting route:
Dst. Address 0.0.0.0/0
Gateway: 10.10.10.1 (reachable sstpr1-r2)
Distance: 1
Scope: 30
Target Scope: 10
Routing Mark: sstpr1-r2mark

And it seems to work.
I can ping any device connected with a cable to R2 ether2 from R1, the distanced router.

Because I tested the incoming net first on R1 I can be sure its incoming on R1 the right way to use on a LAN port it should be the right way to route it over the sstp tunnel to the endpoint 10.10.9.2, or?
And I can reach any host from R1 over R2 for example 77.66.40.12 or 77.66.41.215 (gw 77.66.40.1 Subnet 255.255.248.0) and so on.
But from outside there is no connection and from inside there is no internet from the hosts of R2 -> ether2 (gw 77.66.40.1)

Torch is not telling me that any incoming traffic for 77.66.40.0 is passing incoming.
Connection mark is telling me that there are outgoing packets with a mark, but of course the test hosts are working and searching from inside.
I more and more think that R1 is the problem and just routing the 77.66.40.0/21 -> sstpr1-r2 reachable is not he right way.

But it feels more like I dont know anything. Name? Planet? ... my head is really exploding.
I get really mad on this and my head is getting dizzy the last days/weeks. What I am doing wrong.
Please get me out of this nightmare.

Mon Jul 31, 2017 4:58 pm

Let's see - tunneling MPLS across the Internet. . . routing tables. . .

Perhaps all of this complexity is necessary, but it sounds like you've got one of these overly-complex mechanisms on your hands.

If the tunnel is up and you can ping across the tunnel, then there's a good chance that there's one of three things happening if you can't reach the Internet via the tunnel:
1- firewall filter rules (make sure the router that gives Internet access to the remote end of the tunnel will allow packets to forward from the tunnel interface to its WAN interface)
2-firewall NAT rules (make sure that router will perform src-nat on packets going to the Internet from the tunnel
3-Routing table entries

I see you point 0.0.0.0/0 through the tunnel - I'm assuming the 10.x.x.x IPs in the tunnel are part of a tunnel-only routing table - make sure that the LAN and Tunnel endpoint IPs are included in the tunnel routing table.

DaKater · Mon Jul 31, 2017 6:19 pm

First of all: Thank you for the reply and website. A first smile after days (and nights) for my colleague and me.
But well --- BGP is an option, too for connecting 2 sites .... how to get RIPE member .... O M G.... (just kidding)

Second: I will shutdown and test everything in the next 2-3 hours and will report in here.

A pitty. I cannot shutdown and test right now what you wrote me. I thought about activating nat on the Internet Router Site, too (once)
The Firewall rule on this router will be hard, but I think we can get it.

Well, the network is runnig with a dirty bridged solution (and 15% packet loss) for now.
Better then 100% loss

Third: I have another solution, but ... wow ... sounds for me again like your website-link in my head. More easy to configure, but can it work ... and if... why?

R1 (Internet Router with the public ip block)
ip routes
77.66.40.0/21 -> 10.10.9.2 reachable SSTP (ip of R2 endpoint of sstp)

R2 (Remote office)
ip routes
defaulte route -> 10.10.9.1 reachable SSTP (ip of R1 endpoint of sstp)
99.66.55.161 (public outside IP from R1) -> 99.88.77.1 (public IP and gateway from ISP)

ok ok ... do I got this right?
with the defaulte route -> 10.10.9.1 it could theoretical work, but the mikrotik will loose its ability to be online.
Thru the second route where I tell him where to find the online way (back) thru the tunnel it maybe could work.

But can I reach this router for configuration anymore?
Is this really a good idea?
No firewall, no mangle rule.

I will test all ways this night again.

Big thanks for the good idea with NAT and the fast answer.

Mon Jul 31, 2017 6:23 pm

Before you do anything based on what I said (which was general thoughts, and not really particular advice for your specific situation), let's take a step back.

What are you trying to do? (in general terms)
- e.g.: "I'm trying to give public IP addresses from Site A to be usable at site B"
What is your goal? What sort of functionality are you trying to support?

Could you post a diagram of your two routers and the connections they have? Be sure to indicate the real interfaces, the tunnel, and the IP addressing on all relevant interfaces.

DaKater · Mon Jul 31, 2017 6:28 pm

Yeah. I wish I had VISO or whatever software (and the ability to use it).

I am allowed to post pictures in here? (never tried before)
I will try to draw with GIMP a bit.
Or the classical way without virtualization. Pen and Paper and a picture of it later.

I think this would be much more easy. I agree.
I have 2-3 hours until next testing. I will try to draw it ...

Mon Jul 31, 2017 6:42 pm

When you post the drawing, don't forget to include an overview of what you're trying to accomplish in very general terms like in my examples previously.

pukkita · Mon Jul 31, 2017 7:07 pm

Yeah. I wish I had VISO or whatever software (and the ability to use it).

I am allowed to post pictures in here? (never tried before)
I will try to draw with GIMP a bit.

There are free online diagraming tools, more on the post in my sig.

If you're handy with it, the dude could also be used for that (network layout), then export the image.

DaKater · Mon Jul 31, 2017 7:18 pm

FOUND A MISTAKE IN THE PICTURE:
THE ROUTE WRITTEN IS TO 10.10.9.2 from R1 to R2 ofc and not to itsself endpoint. (sorry)
###

On the left site you see the Remote office (R2).
It has an own static public IP from ISP and the local sstp address 10.10.9.2

One the right site you see the far away Router (R1). Lets call it Backbone.
The public IP from ISP attached to ether1 is 77.66.44.165
BUT
the incoming PA net (99.88.77.0/21) is routed in via .164 (VRRP).
For our example not important. The second router on Backbone site is just doing a bad bridged solution right now and is not important.

99.88.77.0/21 will be later splitted in 8x /24 --- but for us not important, too. Right now we are fine if we could route this block to R2.
For this we did the routes and the SSTP connection (this is working fine, we know it from LAN configurations - port 443 etc. and bridge for profile etc.)
Ping from R1 to R2 is working, even on single hosts of 99.88.77.15 with Gateway 99.88.7.1 (ether2 - R2)
But 99.88.77.15 is not reachable from the internet and this is the goal. Any host with an IP address connected from the PA net on Remote Office site should be open for the outside.

The problem seems to occur between R1 and Internet on the back route and/or back from R2 in the tunnel or tunnel bridge. (not sure about this)
But because we can ping any PA net address from R1 to the remote site and of course we can ping the PA gateway (99.88.77.1) connected to ether2 from R2, too I am more and more sure the problem is the Backbone site.

If I activate NAT on R1 it could maybe help, but I am not sure what exactly to setup.
And it has to be the right way or maybe the hosts are be online, but the outside IP is a gateway-IP for all (and so on). I am not good with NAT, too :/

Another idea we got is this, but I dont get behind this:

Backbone site (R1):
route 99.88.77.0/21 via SSTP endpoint of R2 (10.10.9.2)

Remote Site (R2)
defaulte route via SSTP endpoint of R1 (10.10.9.1)
*** would cause the router to go offline, I think, but then he is saying ***
77.66.44.165 via 55.70.88.3 (what is the gateway of our ISP of Remote site)

Even in the last point I am not sure if 55.70.88.3 (IP from ISP router) or 55.70.88.9 (the static IP from ISP for Mikrotik on ether1)

I hope anything of this is understandable :/

P.S.: Thank you for the software hint. I will have a look on this portal.

DaKater · Mon Jul 31, 2017 7:43 pm

P.P.S.: I found the VRT extension on OpenOffice. Installed it and I hope to make you are real 100% clear overview, even with all exact configurations.

DaKater · Mon Jul 31, 2017 8:42 pm

Maybe this gives a better overview. Finally I mastered something with a picture. My teacher in art would be suprised

Everything is about: How to get the full routed PA net 99.88.77.0/21 routed to R2 with answers over SSTP and of course reaching the Internet again.
While 99.88.77.1/21 is the gateway on ether2 (R2, Remote Office) for all Hosts.

IDEA 1
Backbone site (R1):
route 99.88.77.0/21 via SSTP endpoint of R2 (10.10.9.2) ***already set like this***

Remote Site (R2)
defaulte route via SSTP endpoint of R1 (10.10.9.1) ***afraid of setting it like this***
77.66.44.165 via 55.70.88.3 (what is the gateway of our ISP of Remote site) ***afraid of setting it like this***

Well, maybe this is working in a way. But I have the feeling ... everything will be routed through the tunnel then on R2 and ... can this work? What happens if sstp fails for a reason (Remote Office site is Client SSTP site; Backbone R1 is SSTP Server).

IDEA 2
Activating NAT on Backbone (R1) site - but we had this idea, but no idea how to.

IDEA 3
Mangle rule with a connection mark first sounds exactly what we need. But I feel more and more, because of the sstpbridge which is necessary ... maybe this is not working with a connection or packet mark. Or its working in a way, but totally like your website link. Overpowered mechanics

TESTED:
SSTP is working fine and we can reach both sites from the other site.
We tested pinging and 99.88.77.1 was reachable on ether2 R2 from R1 (thru SSTP). (R1 -> 10.10.9.2 and R2 -> 10.10.9.1)
99.88.77.1 if it is mapped to ether2 (R1) than this PA net is reachable from outside and working with a cable on ether2 and all IP addresses from PA net are working from inside and outside.
But this is the wrong port on the wrong router. We want it on R2 -> ether2 for the Remote Office site.

99.88.77.1 is reachable on ether2 (R2) from R1. (R2 with ether2 has 99.88.77.1 and we can ping it from R1)
But not from outside. And so, no host like 99.88.77.2 could be online if the gateway cannot be pinged from outside.

Mon Jul 31, 2017 10:21 pm

I'd say to route the IP and really assign it onto hosts at site 2. NAT can get "interesting" with all of the permutations of where it gets translated, where it gets forwarded, etc.
Just use the 99.88.77.x addresses on hosts at site2 where you wish to use public IPs from Site 1's pool.

Then make sure your NAT rules do not process packets entering/leaving the tunnel.

Since R2 only has its ISP-supplied public IP, the only NAT rule you need is a srcnat rule:
/ip firewall nat add chain=srcnat out-interface=ether1 action=src-nat to-address=55.70.88.9

Now, you can do this pretty easily with some route table rules and no packet marking in mangle table.

Add routing rules like this:

/ip route rule
add action=lookup-only-in-table dst-address=10.0.0.0/8 table=main
add action=lookup-only-in-table src-address=99.88.77.0/24 table=pubhosts

Okay - obviously the first rule should be modified to be the actual range of your private IPs at site 2. (if more than one block is required, then add more rules just like it)
The src-address rule should be as specific as possible - if you routed a /29 of public space via the tunnel from R1, then only list that particular range.

Finally, create 2 routes for the routing table pubhosts:
0.0.0.0/0 -> 10.10.9.1
0.0.0.0/0 -> 55.70.88.3 (distance=2)

That should get what you want done.

EDIT:
I forgot to add that you'll need to add locally-connected routes for the SSTP and ether1 interfaces into the pubhosts routing table:
dst=10.10.9.0/24 gateway=sstp-interface routing-mark=pubhosts
dst=55.70.88.3/24 gateway=ether1 routing-mark=pubhosts

(replace the /24 with the appropriate netmasks in use)

DaKater · Tue Aug 01, 2017 1:57 am

Supercalifragilisticexpialidocious ... ok ... I think I got behind what you are doing, but not in all points so maybe I will fill in some things wrong.

I will try to repeat what I should do:
R1 is fine like in the diagram.

R2 -> First

/ip route rule
add action=lookup-only-in-table dst-address=10.10.9.2/32 table=main
add action=lookup-only-in-table src-address=99.88.77.0/21 table=pubhosts

The local address of SSTP is set by the sstp secret and profile of the SSTP Server on R1.
After starting SSTP Server and Client rules with 10.10.9.2/32 (R2) and 10.10.9.1/32 (R1) are added dynamicaly.

R2 -> Second

/ip route
add dst-address=10.10.9.0/24 gateway=sstpbridge routing-mark=pubhosts
add dst-address=55.70.88.3/30 gateway=ether1 routing-mark=pubhosts

R2 -> Third

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.9.1/24
add dst-address=0.0.0.0/0 gateway=55.70.88.3/30 distance=2

Problems:
1) I am not sure about the /24 of the 10.10.9.x net. SSTP is not asking me about a subnet, of course, and its /32 on each tunnel end after connecting. I could use /30 or /28 or whatever smaller, but /24 should be fine as one Class C. Or?
2) I am not sure about all parameters and even if I set them up in the right sector (routes or rules).
3) I tested around with connection mark and was 100% unsuccessfull and I am happy I I dont have to mangle anything and the setup is as small as possible, but just to know.
Do I need those table entries for a connection mark, too? I followed some tutorials and I never used them. (not so important for my problem, but good to know)

####

I have to admit that I have a problem with (all) command lines, scripting --- well syntax at all.
That is my problem since 15-20 years and why I will never be a - professional - in this field.

You could say its something like dyslexic on CLI´s or general written syntax.

I am really trying hard, again and again, to use CLI´s... I think its getting better little bit, but well ... as slow I have to get 500 years old ...

So I have to admit that I use the Webinterface (TCP80) and Winbox (TCP8291) and I once tested the Terminal, a CL Emulation within the Webinterface, for /export hide-sensitive
But the WI and WinBox seems to have the power that everything can be configured and CL is not needed (on this stage of use).
And if somebody is writing the full command from the command line I was able to find eveything (for now) on the WI or Winbox too.

But your learning stuff you gave me is hard to fetch with CL for me even on this basic level.
I think If I ever will try to get deeper in what YOU are setting up every day I will be like this at the end of the day, after failing --> https://www.youtube.com/watch?v=1kwECkqm3VM

DaKater · Tue Aug 01, 2017 11:11 am

Maybe the third code box is better like this?

/ip route
add dst-address=0.0.0.0/0 gateway=10.10.9.1/24 routing-mark=pubhosts
add dst-address=0.0.0.0/0 gateway=55.70.88.3/30 distance=2 routing-mark=pubhosts

DaKater · Tue Aug 01, 2017 12:41 pm

/ip route rule
add action=lookup-only-in-table dst-address=10.10.9.2/32 table=main
add action=lookup-only-in-table src-address=99.88.77.0/21 table=pubhosts

/ip route
add dst-address=10.10.9.0/28 gateway=sstpbridge **routing-mark=pubhosts** (not empty for main?)
add dst-address=55.70.88.3/30 gateway=ether1 routing-mark=pubhosts
add dst-address=0.0.0.0/0 gateway=10.10.9.1/32
add dst-address=0.0.0.0/0 gateway=55.70.88.3/30 distance=2

Maybe with a bit help I got behind this. I adjusted the subnets of the tunnel endpoints in routing and routing rule and I lowered the /24 to /28 of the sstpbridge route.
But before I test it:

I thought I got it now, but then I see the first line of /ip routes and I have the feeling again I didnt get anything.

Wed Aug 02, 2017 4:24 pm

I use Winbox primarily. It's just that the forums are text-based, so it's easier to post configurations using command-line syntax because you can just type it - when explaining winbox, you have to include lots of steps about click this, then this, then that; you sometimes need to make screenshots, etc. Many times posters will give a screenshot of their firewall rules and say "what's this doing wrong?" and the problem is that the window doesn't show all of the fields configured in the rules. I actually improved quite a bit w/ the command line by participating in the forums. I typically have a Mikrotik open in Winbox with the command terminal open to check that I'm giving the correct syntax, especially with commands I don't use as much.

As for the routes, the main routing table will have those connected routes automatically added when you put IP addresses on the interfaces. so you shouldn't need to add anything there for the normal functionality. The policy routes are just there to force the public IP hosts to use the tunnel as default GW. The route rule to use table=main is there to exempt all internal destinations from the special policy.

Who is online