Static Default Route - I'm missing something

Pericynthion · Sat May 05, 2018 12:09 am

So this is a follow on (but discrete) question from another thread - I think I'm missing something obvious.

1) I have 2 dynamic default routes on my Hex. (i) with a metric of 2 via the Eth1 WAN interface (ii) with a metric of 1 via an L2TP interface
2) I have srcnat masquerade NAT rules in place for both out interfaces (the defconf WAN and I added one for the L2TP)

As you would expect, everything goes over the L2TP tunnel, except when it drops - then everything following the direct route out of the WAN Eth1 connection. Exactly as I wanted it to.

Question: How can I identify specific clients (static IP's) to ALWAYS follow the less preferable default route of direct out of the WAN regardless if the L2TP is up or not.

I tried Using a mangle mark-routing 'BYPASSVPN' , and I created a new static default route for this routing mark - 0.0.0.0 with a gateway of Eth1.
It seems that with this in place, everything 'routes' like its supposed to , but if I run a traceroute it goes out the correct interface but never passes the next hop (which I think is the ISP gateway - but thats a guess).

Am I going about this the right way? or am I missing something do with the way the pre-routing mangle and the routing process is working when I specify the Eth1 as the gateway?

anav · Sat May 05, 2018 12:52 am

Well it should work....
IP mangle.
new routing mark-bypassvpn source-address-list =(list created in IP firewall lists to identify which IPs) "bypasslist" chain=prerouting
IP route
destination=0.0.0.0/0 gateway=gateway_IPaddress_ISP1 mark route=bypassvpn

maybe try a two step mangle rule
First step marks new connections for those source LANIP addresses.
Second step route marks them.
and then make the necessary route rule........

Maybe this two step is unnecessary but maybe more efficient??

sindy · Sat May 05, 2018 1:08 am

Question: How can I identify specific clients (static IP's) to ALWAYS follow the less preferable default route of direct out of the WAN regardless if the L2TP is up or not.

The question is whether you really need some clients to send packets to a given destination via the L2TP tunnel and some other clients to send them to the very same destination using the usual gateway or Eth1 instead. If yes, you do need the mangle rule as you have found out. If you just need an exception from the default route for that destination, for any client, just add a dedicated route there to the routing table, as a route with a more precisely defined dst-address (longer prefix, i.e. mask with more 1 bits) always beats a route with less precisely defined dst-address (shorter prefix, i.e. mask with less 1 bits).

It seems that with this in place, everything 'routes' like its supposed to, but if I run a traceroute it goes out the correct interface but never passes the next hop (which I think is the ISP gateway - but thats a guess).
Am I going about this the right way? or am I missing something do with the way the pre-routing mangle and the routing process is working when I specify the Eth1 as the gateway?

From where do you traceroute? From the client or from the Mikrotik?

If from the Mikrotik, bear in mind that for locally originated packets do not pass chain=prerouting; to mark them, you have to place the same marking rules to chain=output. See this picture for details. Don't miss the "routing adjustment" in the end of the Output chain.

If from the client, I don't have any answer.

Pericynthion · Sat May 05, 2018 1:14 am

So I think anav pointed out the minor flaw in my plan - if I point the mark-routing static route at the ISP gateway via the ISP gateway IP address then everything works (thanks anav!!)

I was originally routing the static route at the Eth1 interface, on the understanding that the IP address at the other end might be dynamic. I dont suppose there's some smart way to say 'via whatever the gateway address is on Eth1' from another static route?

sindy · Sat May 05, 2018 1:23 am

So I think anav pointed out the minor flaw in my plan - if I point the mark-routing static route at the ISP gateway via the ISP gateway IP address then everything works (thanks anav!!)

It wasn't clear to me from the wording that "gateway of Eth1" means "Eth1 as route's gateway".

I was originally routing the static route at the Eth1 interface, on the understanding that the IP address at the other end might be dynamic. I dont suppose there's some smart way to say 'via whatever the gateway address is on Eth1' from another static route?

There is but it seems that Mikrotik doesn't use it. It is called ICMP router discovery. This is what permits most linux machines to set an interface as a gateway of a route. In Mikrotik, you can use the interface as a gateway of a route only on point-to-point links such as tunnels.

Pericynthion · Sat May 05, 2018 1:27 am

Got it - thanks for the clarification everyone! We're up and running (with a much simpler config!)

sindy · Sat May 05, 2018 1:50 am

First step marks new connections for those source LANIP addresses.
Second step route marks them.
and then make the necessary route rule........

Maybe this two step is unnecessary but maybe more efficient??

@anav,
two step marking has two uses:

to evaluate complex sets of match conditions only once, when connection-marking the initial packet of the connection, and then route-marking every packet of the connection in a given direction up to the connection mark alone (including the first packet in most scenarios). This is the "efficiency" purpose.
to set routing marks for LAN->WAN packets properly for connections established in WAN->LAN direction, i.e. when your router is doing port forwarding from clients in the internet to some server in LAN and you need to send that server's responses back the right way and src-nated to the right public IP address.

Pericynthion · Sat May 05, 2018 2:07 am

Follow up question Sindy. As I understand it , as soon as I touch policy based routing with a mark-routing mangle rule, I can’t use the fasttrack feature (hence I have that default firewall rule disabled).

Is there anyway I can still leverage fasttrack for the non-marked packets or am I just stuck with CPU handling of every packet to match the mangle rule on the prerouting?

sindy · Sat May 05, 2018 12:09 pm

Follow up question Sindy. As I understand it , as soon as I touch policy based routing with a mark-routing mangle rule, I can’t use the fasttrack feature (hence I have that default firewall rule disabled).

Is there any way I can still leverage fasttrack for the non-marked packets or am I just stuck with CPU handling of every packet to match the mangle rule on the prerouting?

Sure there is. But to do that, it is not sufficient to use only routing marks, you have to use connection marking. And once you do, it is best to base routing marking on connection marking as well.

The action=fasttrack-connection rule in /ip firewall filter matches on both upload (LAN to WAN) and download (WAN to LAN) packets. So you have to mark the whole connection, i.e. all its packets in both directions, as you mangle its initial packet if you want to use non-default handling for that connection. Connections you don't mark when they begin will be handled the default way.

Therefore, start by identifying (using a qualified guess or mangle rule statistics depending on your particular case) the kind of traffic which has the highest volume. This is the kind of traffic you want to use default handling for, so that you wouldn't need to assign routing marks to it, so that you could use fasttracking for it. All other kinds of traffic will get some connection marks and, eventually, also routing marks based on them.

Then, you add a match condition connection-mark=no-mark to the condition list of the action=fasttrack-connection chain=forward connection-state=established,related rule in /ip firewall filter. Mission completed, only packets without any connection mark will now be fasttracked, so once you set up the connection marking rules properly, you can enable this action=fasttrack-connection rule.

Now the details.

You must assign (or not) the connection mark only when mangling the initial packet of each connection, so the action=mark-connection rules must conform to connection-state=new match condition; this can be ensured either by placing that condition into each rule or by preventing non-conformant packet from reaching that rule
for upload packets you need to add a routing mark also to the initial packet which has just been connection-marked. As you cannot assign both routing mark and connection mark using a single rule, the action=mark-connection rules must not stop the handling of the packet in prerouting, hence add passthrough=yes to them
you want to connection-mark each connection's initial packet only once, so if you use more than one non-default handling and hence need more than one connection mark, it is always better to put a connection-mark=no-mark match condition to all action=mark-connection rules except the first one to avoid hard to understand mistakes. Normally you are used to that if rule N matches on src-address=192.168.0.3, rule N+1 just below it may match on src-adddress=192.168.0.1-192.168.0.9 and it will never actually act on packet with source address 192.168.0.3 because rule N won't let the packet with that source address ever reach rule N+1. This usual behaviour doesn't work if rule N has passthrough=yes, which is necessary so that the packet could reach some later action=mark-routing rule.

To speed up the processing as much as possible also for packets which are not fasttracked, you need to process each packet by the least possible number of rules in mangle, which means that the action=mark-routing rules should be as close as possible to the beginning of the chain, but at the same time you need them after the action=mark-connection rules, which simply means you have to put them to the chain twice.

So it's time for an example. Assuming you have one default handling and two non-default ones, your mangle rules would like as follows:

/ip firewall mangle
add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
add chain=prerouting connection-state=established,related in-interface=your-wan action=accept # download packets MUST NOT be routing-marked
add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A # passthrough=no is a default behaviour but you can state it explicitly
add chain=prerouting connection-mark=handling-B action=mark-routing new-routing-mark=handling-B # same like above

#only initial packets of connections (plus some garbage) get here past the rules above
add chain=prerouting ...list of classifying match conditions for handling A... connection-state=new action=mark-connection new-connection-mark=handling-A passthrough=yes
add chain=prerouting ...list of classifying match conditions for handling B... connection-mark=no-mark connection-state=new action=mark-connection new-connection-mark=handling-B passthrough=yes

#initial packets of connections which evaded both the rules above get here with no connection mark; we just repeat the mark-routing rules above
add chain=prerouting connection-state=established,related in-interface=your-wan action=accept # download packets MUST NOT be routing-marked
add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A
add chain=prerouting connection-mark=handling-B action=mark-routing new-routing-mark=handling-B

NB: connection marks, routing marks and packet marks use separate name spaces, so connection mark XYZ has no relationship to routing mark XYZ unless a mangle rule assigns the latter based on the former.

EDIT to have everything in a single post: if the various handlings are used to control multi-WAN arrangements, and if the router itself or devices on its LAN should act as servers accessible by clients in the internet, it is necessary to assign connection marks also to initial packets coming in via the WAN interface, so that the response packets would be routed through the same WAN interface. The default handling may use one of the WAN interfaces as outbound route; in such a case, connections initiated by packets coming in via that WAN interface get no connection mark. EDIT 2: fixed the template rules to properly handle the scenario where connections are initiated from the WAN side - thanks @Braddock. Also, see post #30 for a more compact approach.

Pericynthion · Sat May 05, 2018 6:38 pm

Perfect. Thanks for the detailed response - I’ll try reworking the config today!

Sent from my iPhone using Tapatalk

Pericynthion · Sat May 05, 2018 11:33 pm

So I think its 'sort of working' Sindy - all the fasttrack non-marked connections are working fine, but I'm still seeing some of the connections I expected to be marked as flowing through fasttrack without a connection mark.

The configuration is pretty simple in terms of the mangle - just one non-default handling, and by source IP address;

/ip firewall mangle 
add action=accept chain=prerouting comment="Process mid-connection with no connection mark" connection-mark=no-mark connection-state=established,related

add action=mark-routing chain=prerouting comment="MarkRouting BypassVPN Early" connection-mark=Bypass-VPNconn new-routing-mark=BYPASSVPNroute passthrough=no

add action=mark-connection chain=prerouting comment="Connection-Mark for Clients Bypassing the VPN" connection-state=new new-connection-mark=Bypass-VPNconn passthrough=yes src-address=192.168.118.132

add action=mark-routing chain=prerouting comment="MarkRouting BypassVPN Late" connection-mark=Bypass-VPNconn new-routing-mark=BYPASSVPNroute passthrough=no

So my interpretation is that any new connections from 192.168.113.132, will get a new connection mark which in turn will become a new routing mark. However, for some reason when I run like this, that client works for some sites and then fails on others. If I change it back to a simple mark-routing with the fasttrack disabled then it works. Weird. I might try some reboots in between config changes - it could be that I'm running on the latest RC version of routerOS I guess. More testing needed!! Thanks again.

sindy · Sun May 06, 2018 12:29 am

So my interpretation is that any new connections from 192.168.113.132, will get a new connection mark which in turn will become a new routing mark. However, for some reason when I run like this, that client works for some sites and then fails on others. If I change it back to a simple mark-routing with the fasttrack disabled then it works. Weird. I might try some reboots in between config changes - it could be that I'm running on the latest RC version of routerOS I guess. More testing needed!! Thanks again.

May I see the complete /ip firewall export (with public addresses substituted ny something meaningful if needed)? The marking rules seem fine to me too.

Pericynthion · Sun May 06, 2018 7:37 am

Here you go - thankyou again for taking the time to look this one over!

To clarify the setup I have an L2TP/IPSEC VPN interface which is my preferred default route when it is up and running, and if that interface drops it follows the default route direct via the ISP (so 2 dynamic default routes with the ISP learned one artificially set to distance:2). - hence the 2 masquerade rules.
What I'm trying to do with the connection-marking is fasttrack the majority of the traffic that goes over the default route (L2TP or direct, whichever is the currently preferred seems to be working great on that part- thanks!), but for a handful of specific clients I want them to follow this direct-to-ISP route all of the time , hence the connection and route marking of BYPASSVPN.

I did solve one riddle of why some sites appeared to work - yahoo.com / cnn.com / mikrotik.com etc - they all have IPv6 addresses which of course aren't relevant here!

Also after rebooting, I can confirm that the test client (192.168.118.143 here) no longer shows any connection that do not have the connection mark so it appears to be correctly identifying all new connections and marking them. From what I can see on the outbound connections, the masquerade is working too because the reply-to-address is the ISP-provided address:port for these connections, and not the L2TPendpoint:port (as it is for the VPN connected clients)

/ip firewall filter
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" dst-limit=30,30,dst-address/1m40s limit=30,30:packet protocol=icmp src-address=192.168.118.0/24
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept out ipsec policy" ipsec-policy=out,ipsec
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-mark=no-mark connection-state=established,related
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN

/ip firewall mangle
add action=accept chain=prerouting comment="Process mid connection with no connection mark" connection-mark=no-mark connection-state=established,related
add action=mark-routing chain=prerouting comment="MarkRouting BypassVPN Early" connection-mark=BYPASSVPN-conn new-routing-mark=BYPASSVPN-route passthrough=no
add action=mark-connection chain=prerouting comment="Connection-Mark for Clients Bypassing the VPN" connection-state=new disabled=yes new-connection-mark=BYPASSVPN-conn passthrough=yes src-address=\
192.168.118.143
add action=mark-routing chain=prerouting comment="MarkRouting BypassVPN Late" connection-mark=BYPASSVPN-conn new-routing-mark=BYPASSVPN-route passthrough=no

/ip firewall nat
add action=masquerade chain=srcnat comment="Hairpin NAT" disabled=yes dst-address=192.168.118.0/24 src-address=192.168.118.0/24
add action=masquerade chain=srcnat comment="defconf: masquerade for ISP routed traffic" ipsec-policy=out,none out-interface-list=WAN
add action=masquerade chain=srcnat comment="srcnat for HideIPVPN routed traffic" out-interface=l2tp-out1

For clarity , here is the current routing table (top entry is for the BYPASSVPN-route routing-mark. Public addresses changed to .99;

 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE

 0 A S  ;;; Static route via local ISP for BYPASSVPN-route

        0.0.0.0/0                          70.95.64.99                1
 1 ADS  0.0.0.0/0                          l2tp-out1                 1
 2  DS  0.0.0.0/0                          70.95.64.99                2
 3 ADC  10.0.0.2/32        10.7.4.199      l2tp-out1                 0
 4 ADC  70.95.64.0/19      70.95.93.199    ether1-EXTERNAL           0
 5 ADS  104.237.61.2/32                    70.95.64.99                0
 6 ADC  192.168.118.0/24   192.168.118.1   bridge-LAN                0

sindy · Sun May 06, 2018 8:46 am

Well, I am a bit lost in all that information.

You say that in order to clean up the connection table, you have rebooted the router. Well, you could instead have removed the connections of the test client (and thus broken for the client so it would have had to re-establish them from scratch) using something like

/ip firewall connection remove [find src-address~"192.168.118.143"]

but that's not the point here.

Now after the reboot, you say that all connections from 192.168.118.143 are connection marked, but do you also say that nevertheless some of them (or maybe even all) get fasttracked?

Pericynthion · Sun May 06, 2018 9:33 am

No , sorry for too much info

The connection marking is working correctly for all connections from the client. But despite the connection and routing apparently going through the correct srcnat and routing table entry, the client is unable to reach any external sites.

Back to your original comment, I suspect this is a firewall filter issue - I guess I have to eliminate them one by one.

sindy · Sun May 06, 2018 10:05 am

Back to your original comment, I suspect this is a firewall filter issue - I guess I have to eliminate them one by one.

No, it's my fault.

When assigning routing marks in mangle, you must only mark upload packets. What happens now is that packets which come through WAN get a routing mark too, and the routing table BYPASSVPN-route doesn't contain the routes to local subnets, so the default route of that table sends the responses back to the WAN.

I was explaining this to someone here less than a week ago

So I've edited my example in the post above to accept "established" and "related" download packets before reaching the connection mark -> routing mark translation rules.

Pericynthion · Sun May 06, 2018 9:23 pm

Sindy - do you have an Amazon wish-list or something I can see?

Can't thank you enough - I've been staring at this for a month now, and you've solved it in 2 posts. Thankyou again.

Everything is working exactly as planned - one last and final question... (I promise).
One of the main reasons for these clients bypassing the VPN is they establish some dynamic dstnat for remote access (here is a copy of what they typically do via uPNP);

 3  D ;;; upnp 192.168.118.99: WD2go
      chain=dstnat action=dst-nat to-addresses=192.168.118.99 to-ports=80 protocol=tcp dst-address=70.95.93.99 in-interface=ether1-EXTERNAL dst-port=9091 

 4  D ;;; upnp 192.168.118.99: WD2goSSL
      chain=dstnat action=dst-nat to-addresses=192.168.118.99 to-ports=443 protocol=tcp dst-address=70.95.93.99 in-interface=ether1-EXTERNAL dst-port=443

Obviously in this case, the in-interface is external-Eth1- so do I just need a separate connection-mark for 'new' connections where the internal client address is the destination rather than the source? Or is there something I can easily do to the existing rules to accomodate any dstnat like the above.

sindy · Sun May 06, 2018 9:55 pm

do you have an Amazon wish-list or something I can see?

Well, if you know what is necessary to do to force-synchronize an Oracle database cluster after it has diverged...

Obviously in this case, the in-interface is eternal - so do I just need a separate connection-mark for 'new' connections where the internal client address is the destination rather than the source? Or is there something I can easily do to the existing rules to accomodate any dstnat like the above.

Gadgets opening ports for http access from outside, hmmm

Are they at least decent enough to look like those big hollow wooden animals?

To the question, as you've guessed, it is enough to mark the connections with the same connection-mark if their initial packets come in via the respective WAN port. Just add the rule below right before or right after the other action=mark-connection one.

add action=mark-connection chain=prerouting comment="Connection-Mark for Clients Bypassing the VPN" connection-state=new new-connection-mark=Bypass-VPNconn passthrough=no in-interface=ether1-EXTERNAL

This time it is the passthrough=no which prevents these packets from getting a routing mark.

Of course the decision whether to permit these connections or not is up to the /ip firewall filter.

Pericynthion · Sun May 06, 2018 10:30 pm

Well, if you know what is necessary to do to force-synchronize an Oracle database cluster after it has diverged...

Depends - is it a RAC cluster thats divided, or has Data Guard gone rogue..
.

Gadgets opening ports for http access from outside, hmmm Are they at least decent enough to look like those big hollow wooden animals?

All low risk stuff - I'm trying to become more automated operations driven, so I can live without manually managing port-forwards for a living
.

To the question, as you've guessed, it is enough to mark the connections with the same connection-mark if their initial packets come in via the respective WAN port. Just add the rule below right before or right after the other

Thanks again - I feel like I've actually achieved something today!

sindy · Sun May 06, 2018 10:45 pm

Depends - is it a RAC cluster thats divided, or has Data Guard gone rogue..

I suppose a Data Guard, one Primary and one Standby with all those flashbacks, managed recovery, "restore" and "recover" commands in RMAN which have to be done both (and restore succeeds but recovery fails), and it was enough to shut down everything and then start it again to end up with an "unresolvable gap" and "restore required" (after shutting down the application first so the database didn't actually get updated). For me everything there is new and different from other database engines I've seen so far, so maybe it is all clear and best of breed but I'm totally lost in that and it drives me mad. I will not lose any actual data now, only time, if I reinstall everything from scratch, but I'm afraid what would I do should something similar eventually happen later.

But something is telling me this is not the right channel do discuss this

CZFan · Mon May 07, 2018 12:48 am

Depends - is it a RAC cluster thats divided, or has Data Guard gone rogue..
... but I'm totally lost in that and it drives me mad....

Wow, so you are human after all

mkx · Mon May 07, 2018 11:36 am

Depends - is it a RAC cluster thats divided, or has Data Guard gone rogue..
... but I'm totally lost in that and it drives me mad....
Wow, so you are human after all

My bet is on highly advanced AI

Other than that ... hats off, sindy!

anav · Sun Oct 02, 2022 5:25 pm

Sindy - do you have an Amazon wish-list or something I can see?

On a serious note, the best gift would be fuel/petrol vouchers/coupons if such things exist, otherwise.......
https://www.tolls.eu/czechia

I was thinking Slovenian schnapps, which apparently is not drinkable but could fuel a car. ;-P

As for AI, you do know they are known to inject random statements of fallibility, like "I dont know the answer to that", just to make us think they are human,

Braddock · Tue Oct 18, 2022 11:26 pm

Hi, I have a follow-up question regarding this long-dead this topic...

I have a similar setup with 2 different ISP WANs, and one LAN. I want the majority of LAN clients to go "out" over WAN1, and only a few clients over WAN2.
I also use fasttrack for as much traffic as possible.

With regards to the above scenario, Sindy's solution with connection marks and routing marks works like a charm for me.

However, I have an additional... complication. Select clients that have to use WAN2 are actually mail and web servers, so naturally, their outbound traffic has to be source NATted to a public IP address in WAN2 range, and inbound traffic has to be destination NATted to their respective LAN IPs.
In a single WAN setup with just "main" routing table, this works just fine, but when these servers have to use a different routing table, I can't get it to work.
It seems that outbound traffic is getting through as intended, but inbound traffic doesn't reach the servers.

As far as I know (and I may be wrong here), I have to connection-mark (and route-mark) the incoming destination NATted traffic too in order to get it to work.
I tried to add this rule in mangle (positioned in the section after Sindy's comment only initial packets...):

add chain=prerouting in-interface=WAN2 dst-address=server's_LAN_IP connection-state=new action=mark-connection new-connection-mark=2nd-routing-table-handler passthrough=yes

But this doesn't work. I also tried with server's public IP as dst-address, but it doesn't work either.

I'm especially confused by Sindy's comment in his solution that "download packets MUST NOT be routing-marked". Does this apply to download packets for both WANs? Can I route-mark only download packets with connection mark "new"? What am I getting wrong here?

Please help.

sindy · Wed Oct 19, 2022 12:06 am

As far as I know (and I may be wrong here), I have to connection-mark (and route-mark) the incoming destination NATted traffic too in order to get it to work.

Correct. You need the connection tracking to "remember" that the initial request of a connection came in via WAN 2 and thus response packets within that connection must be routed out via WAN 2.

I tried to add this rule in mangle (positioned in the section after Sindy's comment only initial packets...):
add chain=prerouting in-interface=WAN2 dst-address=server's_LAN_IP connection-state=new action=mark-connection new-connection-mark=2nd-routing-table-handler passthrough=yes

Not correct. In the prerouting chain, mangle is done before nat, so the rule will never match because dst-address is still WAN2 IP when the packet is matched to this rule. But you don't need to match on dst-address at all - for this particular purpose, it is enough to match on in-interface=WAN2 alone.

I also tried with server's public IP as dst-address, but it doesn't work either.

So the reason why it doesn't work must be something else.

I'm especially confused by Sindy's comment in his solution that "download packets MUST NOT be routing-marked".

A connection mark is assigned by a rule handling a single packet, but unless intentionally changed or removed while handling another packet, it is remembered in the connection tracking module and automatically re-assigned to all subsequent packets belonging to the same connection, regardless their direction. Unlike that, a routing mark is assigned to a single packet and not remembered - it just tells the router which routing table to use for that single packet.

Since the routing table "via WAN 2" only contains a default route, if packets coming in via WAN 2 got a routing-mark "via WAN 2", they would get routed via WAN 2 to the internet (and dropped there or delivered to unrelated destination) rather than getting delivered to the proper recipient in LAN.

Can I route-mark only download packets with connection mark "new"?

new is not a connection mark, it is a connection state. This label is assigned to packets that do not match any existing connection, so they initiate a new one.

Braddock · Wed Oct 19, 2022 1:20 am

Not correct. In the prerouting chain, mangle is done before nat, so the rule will never match because dst-address is still WAN2 IP when the packet is matched to this rule. But you don't need to match on dst-address at all - for this particular purpose, it is enough to match on in-interface=WAN2 alone.

Thank you for the clarification.

So the reason why it doesn't work must be something else.

Any idea what else could it be? I've been monitoring packet counts on the before-mentioned mangle rule, and also on the dst-nat rule. Whenever I refresh the web page in the browser, packet counts go up a few ticks on both mangle and dst-nat. Browser says "connection aborted". I also see on connection tracker that server is constantly trying to reach public NTP server on UDP 123, and that connections have the proper connection mark.

Since the routing table "via WAN 2" only contains a default route, if packets coming in via WAN 2 got a routing-mark "via WAN 2", they would get routed via WAN 2 to the internet (and dropped there or delivered to unrelated destination) rather than getting delivered to the proper recipient in LAN.

I see. Makes sense. Thank you again.

new is not a connection mark, it is a connection state. This label is assigned to packets that do not match any existing connection, so they initiate a new one.

Yes, that's what I meant. It just came out wrong.

sindy · Wed Oct 19, 2022 7:54 am

Any idea what else could it be?

Not without seeing your actual configuration. See my automatic signature for a hint on how to export and anonymize it. Remove serial number, keep RouterOS version.

Braddock · Wed Oct 19, 2022 10:24 am

@Sindy:

I was looking again at my configuration, bearing in mind all the things you advised me...

So. let me double-check if I understood everything correctly:
Any connection initiated from WAN must not have the routing mark on it, because it will be routed back to the internet.
However, connections initiated from WAN must be connection-marked so that reply traffic can be route-marked, and thus routed to the proper WAN interface.

I'll use your example as a "template":

/ip firewall mangle
add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
add chain=prerouting connection-state=established,related in-interface=your-wan # download packets MUST NOT be routing-marked
add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A # passthrough=no is a default behaviour but you can state it explicitly

#only initial packets of connections (plus some garbage) get here past the rules above
add chain=prerouting ...list of classifying match conditions for handling A... connection-state=new action=mark-connection new-connection-mark=handling-A passthrough=yes

#this is my part for connection-marking connections coming from WAN2
add chain=prerouting in-interface=WAN2 connection-state=new action=mark-connection new-connection-mark=handling-A passthrough=yes

#initial packets of connections which evaded both the rules above get here with no connection mark; we just repeat the mark-routing rules above
add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A

If I get this correctly, my mistake here is that connections initiated from WAN are getting their connection-mark, but are also getting their route-mark (the next rule applies it).
Am I getting this right?
Will simply putting passthrough=no on my line in the code solve the problem?

Thank you in advance

Braddock · Wed Oct 19, 2022 10:59 am

@Sindy:

Yes, that's it. Simply putting passthrough=no solved the problem. Web server is now accessible from WAN.

Thank you very much for your inputs. They have been most helpful.

sindy · Wed Oct 19, 2022 10:59 am

Code: Select all
#this is my part for connection-marking connections coming from WAN2
add chain=prerouting in-interface=WAN2 connection-state=new action=mark-connection new-connection-mark=handling-A passthrough=yes
Will simply putting passthrough=no on my line in the code solve the problem?

Yes, it will, but there must be something wrong in the subsequent rules as they must also avoid assigning routing-mark to packets arriving via WAN. And indeed, the rules in my example above are wrong in this regard, so I have to edit that post again (they were fine initially, but I forgot to update them when editing the post to include your scenario).

BTW, I currently use a different arrangement of rules:

chain=prerouting connection-state=new action=jump jump-target=mark-connections
chain=prerouting in-interface-list=WAN action=accept
chain=prerouting connection-mark=A action=mark-routing new-routing-mark=A passthrough=no
...
chain=mark-connections ...some match conditions... action=mark-connection new-connection-mark=A

This approach removes the need to duplicate the action=mark-routing rules.

If I get this correctly, my mistake here is that connections initiated from WAN are getting their connection-mark, but are also getting their route-mark (the next rule applies it).
Am I getting this right?

Yes, as said above, the mistake is already in my template rules. Just a remark to the wording as others will read this as well - the connections initiated from WAN do not get a routing-mark, their initial packets do.

Braddock · Wed Oct 19, 2022 12:20 pm

BTW, I currently use a different arrangement of rules:

Tried your new rules. Works without a hitch.

se232 · Wed Oct 19, 2022 2:01 pm

Hello

I tried the rules on my router, but no success...

chain=prerouting action=accept connection-state=established,related connection-mark=no-mark 
chain=prerouting connection-state=established,related in-interface=ether02-WAN 
chain=prerouting action=mark-routing new-routing-mark=rtab-ether02-WAN connection-mark=connection-WAN2 
chain=prerouting action=mark-connection new-connection-mark=connection-WAN2 passthrough=no connection-state=new in-interface=ether02-WAN 
chain=prerouting action=mark-routing new-routing-mark=rtab-ether02-WAN connection-mark=connection-WAN2

Can somebody tell me, where is my mistake?

regards

se232 · Thu Oct 20, 2022 12:37 pm

Hello *.*

regardless how I order/arrange the rules, I do not get a working scenario.
BTW: Why is it necessary to have one rule twice?

regards

se232 · Thu Oct 20, 2022 1:11 pm

Hello *.*

I tried the updated rules by sindy...

Flags: X - disabled, I - invalid; D - dynamic 
 0  D ;;; special dummy rule to show fasttrack counters
      chain=prerouting action=passthrough 

 1  D ;;; special dummy rule to show fasttrack counters
      chain=forward action=passthrough 

 2  D ;;; special dummy rule to show fasttrack counters
      chain=postrouting action=passthrough 

 3    chain=prerouting action=jump jump-target=mark-connections connection-state=new 

 4    chain=prerouting action=accept in-interface=ether01-WAN 

 5    chain=prerouting action=mark-routing new-routing-mark=rtab-ether02-WAN passthrough=no connection-mark=connection-WAN2 

 6    chain=mark-connections action=mark-connection new-connection-mark=connection-WAN2 in-interface=ether02-WAN

but there is no connection from outside possible.
ether01-WAN is the default port into the internet (the default route 0.0.0.0/0 uses this port)
ether02-WAN is the port, I want to use to access an internal server from the internet and the server shall answer via ether02-WAN back into the internet.

What is the missing point?
Can anybody help?

sindy · Thu Oct 20, 2022 3:50 pm

Can anybody help?

Not without seeing the rest of the configuration. If connection from outside is not possible, modification of filter or nat rules may be necessary, not just of mangle ones.

se232 · Thu Oct 20, 2022 4:50 pm

Hello Sindy,

the access from outside works in general. At the moment I have a static routing rule to add the routing mark rtab-ether02-WAN when the internal server wants to answer (back) into the internet.
I want to change this to the mangle rules, but no success so far.

What information do you need to see to find the problem in my router settings?

regards

sindy · Fri Oct 21, 2022 6:41 pm

What information do you need to see to find the problem in my router settings?

As written above - a complete configuration export, anonymized in terms of the serial number (which can be used to find the public IP of the router if you have the Mikrotik cloud DNS service enabled), any passwords/pre-shared keys (these are not exported automatically if you use hide-sensitive when exporting from ROS 6 and id you do not use show-sensitive when exporting from ROS 7), any login names for external VPN servers, and any public addresses. And any comments that may help identify you - bots never sleep. Also you can usually remove any static DHCP leases.

Make sure that the anonymized configuration remains consistent - e.g. if you have a public IP 123.45.67.89/23 and the gateway is 123.45.66.1, use find&replace in text editor to replace the 123.45 by public.subnet1 or a.a everywhere in the export, so that it still remains clear that the two addresses (a.a.67.89 and a.a.66.1) are related in some way.

se232 · Sat Oct 22, 2022 11:28 am

Hello Sindy,

this should be the complete configuration. The public IPs are 1.2.3.4 (gw 1.2.3.1), 2.3.4.5 (gw 2.3.4.1), 3.4.5.6 (gw 3.4.5.1) and 4.5.6.7 (gw. 4.5.6.1)

# oct/22/2022 09:45:01 by RouterOS 7.5
# software id = DIE2-NY0V
#
# model = 1100AHx2
# serial number = <serialnummer>
/interface bridge
add disabled=yes name=bridge-10.6.0.0
add fast-forward=no name=bridge-broadcast
add admin-mac=<aMacAddress> auto-mac=no fast-forward=no name=\
    bridge-company
add fast-forward=no name=bridge-guest
/interface ethernet
set [ find default-name=ether1 ] comment=www.company.eu name=ether01-WAN \
    speed=100Mbps
set [ find default-name=ether2 ] comment=vpn.company.eu name=ether02-WAN \
    speed=100Mbps
set [ find default-name=ether3 ] name=ether03-WAN speed=100Mbps
set [ find default-name=ether4 ] name=ether04-WAN speed=100Mbps
set [ find default-name=ether5 ] name=ether05-bonding56 speed=100Mbps
set [ find default-name=ether6 ] name=ether06-bonding56 speed=100Mbps
set [ find default-name=ether7 ] name="ether07-trunk(10.1.4.2)" speed=100Mbps
set [ find default-name=ether8 ] name="ether08-trunk(10.1.4.3)" speed=100Mbps
set [ find default-name=ether9 ] disabled=yes name=ether09-nc speed=100Mbps
set [ find default-name=ether10 ] name="ether10-company(switch)" speed=\
    100Mbps
set [ find default-name=ether11 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full name=\
    "ether11-trunk(10.1.4.13)"
set [ find default-name=ether12 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full name=\
    "ether12-trunk(10.1.4.4)"
set [ find default-name=ether13 ] advertise=\
    10M-half,10M-full,100M-half,100M-full,1000M-half,1000M-full name=\
    "ether13-trunk(10.1.4.5)"
/interface eoip
add disabled=yes !keepalive mac-address=<aMacAddress> name=eoip-10.2.1.1 \
    remote-address=10.2.1.1 tunnel-id=3
/interface vlan
add interface=ether04-WAN name=vlan04-company vlan-id=1
add interface=ether04-WAN name=vlan04-guest vlan-id=2
add interface="ether07-trunk(10.1.4.2)" name=vlan07-company vlan-id=1
add interface="ether07-trunk(10.1.4.2)" name=vlan07-guest vlan-id=2
add interface="ether08-trunk(10.1.4.3)" name=vlan08-broadcast vlan-id=3
add interface="ether08-trunk(10.1.4.3)" name=vlan08-company vlan-id=1
add interface="ether08-trunk(10.1.4.3)" name=vlan08-guest vlan-id=2
add interface="ether11-trunk(10.1.4.13)" name=vlan11-company vlan-id=1
add interface="ether11-trunk(10.1.4.13)" name=vlan11-guest vlan-id=2
add disabled=yes interface="ether12-trunk(10.1.4.4)" name=vlan12-broadcast \
    vlan-id=3
add interface="ether12-trunk(10.1.4.4)" name=vlan12-company vlan-id=1
add interface="ether12-trunk(10.1.4.4)" name=vlan12-guest vlan-id=2
add interface="ether13-trunk(10.1.4.5)" name=vlan13-company vlan-id=1
add interface="ether13-trunk(10.1.4.5)" name=vlan13-guest vlan-id=2
/interface bonding
add mode=802.3ad name=bonding56-company slaves=\
    ether05-bonding56,ether06-bonding56
/interface list
add name=WAN
/interface lte apn
set [ find default=yes ] ip-type=ipv4 use-network-apn=no
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec proposal
set [ find default=yes ] enc-algorithms=\
    aes-256-cbc,aes-192-cbc,aes-128-cbc,3des lifetime=8h
/ip pool
add name=pool-10.1.254.0 ranges=10.1.254.100-10.1.254.199
add name=pool-10.1.0.0 ranges=10.1.1.100-10.1.1.199
add name=pool-10.254.0.0 ranges=10.254.1.100-10.254.1.199
add name=pool-192.168.1.0 ranges=192.168.1.100-192.168.1.199
add name=pool-10.5.0.0 ranges=10.5.1.100-10.5.1.199
/ip dhcp-server
add address-pool=pool-10.1.0.0 authoritative=after-2sec-delay bootp-support=\
    dynamic interface=bridge-company name=dhcp-10.1.0.0
add address-pool=pool-192.168.1.0 authoritative=after-2sec-delay interface=\
    bridge-guest name=dhcp-192.168.1.0
add address-pool=pool-10.5.0.0 disabled=yes interface=ether02-WAN name=\
    dhcp-10.5.0.0
/port
set 0 name=serial0
set 1 name=serial1
/ppp profile
add local-address=10.1.1.1 name=VPN use-encryption=yes
add local-address=10.1.1.1 name=iPhone remote-address=pool-10.254.0.0
add local-address=192.168.22.1 name=L2TP_IPSEC use-compression=yes \
    use-encryption=required
add local-address=10.1.1.1 name=cczzz01
add local-address=10.1.1.1 name=profile-openvpn
add local-address=10.1.1.1 name=test use-encryption=no
set *FFFFFFFE local-address=10.1.1.1 use-ipv6=no
/interface l2tp-client
add allow=mschap2 connect-to=bbyyy01.company.eu max-mru=1400 max-mtu=1400 \
    name=l2tp-10.2.1.1 profile=VPN use-ipsec=yes user=10.1.1.1
add allow=mschap2 allow-fast-path=yes connect-to=cczzz01.company.eu max-mru=\
    1400 max-mtu=1400 name=l2tp-10.3.1.1 profile=VPN use-ipsec=yes user=\
    10.1.1.1
/routing bgp template
set default disabled=no output.network=bgp-networks
/routing ospf instance
add disabled=no name=default-v2 router-id=10.1.1.1
/routing ospf area
add disabled=no instance=default-v2 name=backbone-v2
/routing table
add disabled=no fib name=rtab-WAN
add disabled=no fib name=rtab-guest
add fib name=masquerade-guest
add fib name=yyyy
add disabled=no fib name=rtab-ether02-WAN
add disabled=no fib name=rtab-ether03-WAN
add disabled=no fib name=rtab-ether04-WAN
/snmp community
set [ find default=yes ] addresses=0.0.0.0/0
/user group
set full policy="local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,pas\
    sword,web,sniff,sensitive,api,romon,dude,rest-api"
/interface bridge port
add bridge=bridge-broadcast disabled=yes ingress-filtering=no interface=\
    ether09-nc
add bridge=bridge-company ingress-filtering=no interface=\
    "ether10-company(switch)"
add bridge=bridge-company ingress-filtering=no interface=vlan07-company
add bridge=bridge-company ingress-filtering=no interface=vlan11-company
add bridge=bridge-guest ingress-filtering=no interface=vlan11-guest
add bridge=bridge-company ingress-filtering=no interface=vlan12-company
add bridge=bridge-guest ingress-filtering=no interface=vlan07-guest
add bridge=bridge-guest ingress-filtering=no interface=vlan12-guest
add bridge=bridge-company ingress-filtering=no interface=vlan08-company
add bridge=bridge-guest ingress-filtering=no interface=vlan08-guest
add bridge=bridge-broadcast ingress-filtering=no interface=eoip-10.2.1.1
add bridge=bridge-broadcast ingress-filtering=no interface=vlan12-broadcast
add bridge=bridge-company ingress-filtering=no interface=vlan13-company
add bridge=bridge-guest ingress-filtering=no interface=vlan13-guest
add bridge=bridge-broadcast ingress-filtering=no interface=vlan08-broadcast
add bridge=bridge-company ingress-filtering=no interface=bonding56-company
/ip neighbor discovery-settings
set discover-interface-list=!dynamic
/ip settings
set max-neighbor-entries=8192
/ipv6 settings
set disable-ipv6=yes max-neighbor-entries=1024
/interface l2tp-server server
set allow-fast-path=yes authentication=mschap2 enabled=yes max-mru=1400 \
    max-mtu=1400 use-ipsec=yes
/interface list member
add interface=ether01-WAN list=WAN
add interface=ether03-WAN list=WAN
add interface=ether02-WAN list=WAN
add interface=ether04-WAN list=WAN
/interface ovpn-server server
set auth=sha1,md5 certificate=OpenVPN_Server.crt_0 cipher=\
    blowfish128,aes128,aes192,aes256 default-profile=profile-openvpn \
    require-client-certificate=yes
/interface pptp-server server
# PPTP connections are considered unsafe, it is suggested to use a more modern VPN protocol instead
set default-profile=VPN
/interface sstp-server server
set authentication=mschap1,mschap2 certificate=\
    letsencrypt-autogen_2022-09-18T13:53:26Z default-profile=VPN port=4443
/ip address
add address=10.1.1.1/16 interface=bridge-company network=10.1.0.0
add address=192.168.1.1/24 interface=bridge-guest network=192.168.1.0
/ip cloud
set ddns-enabled=yes
/ip dhcp-client
add default-route-distance=10 interface=ether01-WAN use-peer-ntp=no
add default-route-distance=40 interface=ether04-WAN use-peer-dns=no \
    use-peer-ntp=no
add default-route-distance=30 interface=ether03-WAN use-peer-dns=no \
    use-peer-ntp=no
add default-route-distance=20 interface=ether02-WAN use-peer-dns=no \
    use-peer-ntp=no
/ip dhcp-server lease
add address=10.1.1.15 client-id=<aClientID> comment=\
    "aaxxx01sv005 - Neatgear NAS" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.1.211 always-broadcast=yes client-id=<aClientID> \
    comment=AAXXX01PR013 mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.5.3 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.18 client-id=<aClientID> comment=\
    "from here aaxxx01vm***" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.2.19 mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.5.5 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.20 mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.5.6 always-broadcast=yes client-id=<aClientID> \
    mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.1.20 client-id=<aClientID> comment=\
    "aaxxx01sv020 - QNAP" mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.1.21 client-id=<aClientID> comment=\
    "aaxxx01sv021 - VmWare Server" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.5.4 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.8.105 always-broadcast=yes client-id=<aClientID> \
    comment=XBox mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.21 mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.6.1 always-broadcast=yes client-id=<aClientID> \
    comment=EdiMax mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.6.2 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.6.3 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.6.4 always-broadcast=yes client-id=<aClientID> \
    mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.23 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.24 mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.26 client-id=\
    <aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.1.33 client-id=<aClientID> comment=\
    "CCU3 Homematic Server" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.2.28 client-id=\
    <aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.1.34 comment="Tradfri Server" mac-address=<aMacAddress> \
    server=dhcp-10.1.0.0
add address=10.1.1.22 client-id=<aClientID> comment=\
    "aaxxx01sv022 - VmWare Server" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.9.20 client-id=<aClientID> comment=\
    "Server aaxxx01sv020 2nd address" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.1.32 client-id=<aClientID> comment=\
    "Raspberry PI - FHEM Server" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.1.35 client-id=<aClientID> comment=\
    "LAN-Gateway Homematic" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.1.36 client-id=<aClientID> comment=\
    "Homematic LAN Gateway (orig. CCU2)" mac-address=<aMacAddress> \
    server=dhcp-10.1.0.0
add address=10.1.1.23 client-id=\
    <aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.29 client-id=\
    <aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.30 client-id=\
    <aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.4.10 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.27 client-id=\
    <aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.25 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=192.168.1.0 comment="from here guest DHCP" disabled=yes \
    mac-address=<aMacAddress>
add address=10.1.5.1 client-id=<aClientID> comment=\
    "from here cameras" mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.1.37 client-id=<aClientID> comment="Fritz! Box 7270" \
    mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.32 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.1.100 comment="from here on DHCP pool" disabled=yes \
    mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.7.1 client-id=<aClientID> comment=iPhones disabled=\
    yes mac-address=<aMacAddress> server=dhcp-10.1.0.0
add address=10.1.7.2 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.5.9 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.5.8 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.5.7 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=192.168.1.203 client-id=<aClientID> disabled=yes \
    mac-address=<aMacAddress> server=dhcp-192.168.1.0
add address=192.168.1.202 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-192.168.1.0
add address=10.1.5.10 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=192.168.1.138 comment="Vasi\?" mac-address=<aMacAddress> \
    server=dhcp-192.168.1.0
add address=192.168.1.147 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-192.168.1.0
add address=192.168.1.200 client-id=<aClientID> comment=\
    "phones guest" mac-address=<aMacAddress> server=dhcp-192.168.1.0
add address=192.168.1.201 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-192.168.1.0
add address=10.1.7.1 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.4.2 client-id=<aClientID> comment=\
    "from here on access points" mac-address=<aMacAddress> server=\
    dhcp-10.1.0.0
add address=10.1.4.3 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.4.5 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.4.6 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.4.11 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.4.13 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.4.113 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=192.168.1.203 client-id=<aClientID> comment=Name \
    mac-address=<aMacAddress> server=dhcp-192.168.1.0
add address=10.1.4.4 client-id=<aClientID> mac-address=\
    <aMacAddress> server=dhcp-10.1.0.0
add address=10.1.2.33 mac-address=<aMacAddress> server=dhcp-10.1.0.0
/ip dhcp-server network
add address=10.1.0.0/16 dns-server=10.1.1.1 domain=company.lan gateway=\
    10.1.1.1 netmask=16 ntp-server=10.1.1.1
add address=10.5.0.0/16 dns-server=10.5.1.1 gateway=10.5.1.1 netmask=16 \
    ntp-server=10.1.1.1
add address=192.168.1.0/24 dns-server=192.168.1.1 gateway=192.168.1.1 \
    netmask=24
/ip dns
set allow-remote-requests=yes
/ip dns static
add address=10.1.2.1 name=companyweb.company.lan
add address=10.1.1.205 name=aaxxx01ph201.company.lan
add address=10.1.1.205 name=aaxxx01ph301.company.lan
add address=10.1.1.205 name=aaxxx01ph401.company.lan
add address=10.1.1.205 name=aaxxx01ph501.company.lan
add address=10.1.4.2 name=aaxxx01ap002.company.lan
add address=10.1.4.3 name=aaxxx01ap003.company.lan
add address=10.1.4.4 name=aaxxx01ap004.company.lan
add address=10.1.4.5 name=aaxxx01ap005.company.lan
add address=10.1.4.6 name=aaxxx01ap006.company.lan
add address=10.1.1.210 name=aaxxx01ca003.company.lan
add address=10.1.1.204 name=aaxxx01ph001.company.lan
add address=10.1.1.206 name=aaxxx01pr002.company.lan
add address=10.1.1.207 name=aaxxx01pr004.company.lan
add address=10.1.1.208 name=aaxxx01pr008.company.lan
add address=10.1.1.209 name=aaxxx01pr009.company.lan
add address=10.1.2.19 name=sip_server.company.lan
add address=10.1.1.1 name=ntp_server.company.lan
add address=10.1.1.1 name=ntp_server
add address=10.1.2.19 name=sip_server
add address=10.1.1.20 name=tftp_server
add address=10.1.1.20 name=tftp_server.company.lan
add address=10.1.1.20 name=ftp_server
add address=10.1.1.20 name=ftp_server.company.lan
add address=10.1.2.18 name=smtp_server
add address=10.1.2.18 name=smtp_server.company.lan
add address=10.1.2.1 name=aaxxx01vm001.company.lan
add address=10.1.2.9 name=aaxxx01vm009.company.lan
add address=10.1.1.207 name=aaxxx01pr004
add address=10.1.1.16 name=aaxxx01sv006.company.lan
add address=10.1.2.12 name=sites.company.lan
add address=10.1.2.12 name=AAXXX01VM012.company.local
add address=10.1.1.20 name=file_server_1.company.lan
add address=10.1.1.20 name=file_server_2.company.lan
add address=216.58.214.3 comment="no steam" regexp=.*.valvesoftware.com
add address=10.1.1.15 name=aaxxx01sv005.company.lan
add address=1.2.3.4 comment="temporary --> to be deleted" disabled=yes \
    name=www.company.eu
add address=10.1.1.20 name=aaxxx01sv020.company.lan
add address=10.1.1.21 name=aaxxx01sv021.company.lan
add address=10.1.2.26 name=deep_learning.company.lan
add address=10.1.2.29 name=7d2d.company.lan
add address=10.1.2.28 name=iobroker.company.lan
add address=10.1.2.27 name=crawler.company.lan
add address=10.1.1.22 name=vmware.company.lan
add address=10.1.2.20 name=plex_server.company.lan
add address=10.3.1.12 name=cczzz01sv002.company.lan.
add address=10.2.1.12 name=bbyyy01sv002.company.lan
add address=10.1.2.20 name=http_server
add address=10.1.2.11 name=aaxxx01vm011.company.lan ttl=59m59s
add address=10.1.2.18 name=aaxxx01vm018.company.lan ttl=59m59s
add address=10.1.2.20 name=aaxxx01vm020.company.lan ttl=59m59s
add address=10.1.2.21 name=aaxxx01vm021.company.lan ttl=59m59s
add address=10.1.2.22 name=aaxxx01vm022.company.lan ttl=59m59s
add address=10.1.2.23 name=aaxxx01vm023.company.lan ttl=59m59s
add address=10.1.2.24 name=aaxxx01vm024.company.lan ttl=59m59s
add address=10.1.2.26 name=aaxxx01vm026.company.lan ttl=59m59s
add address=10.1.2.28 name=aaxxx01vm028.company.lan ttl=59m59s
add address=10.1.2.29 name=aaxxx01vm029.company.lan ttl=59m59s
add address=10.1.2.30 name=aaxxx01vm030.company.lan ttl=59m59s
add address=10.1.1.110 name=aaxxx02vm027.company.lan ttl=59m59s
add address=10.1.1.103 name=aaxxx01ws010.company.lan ttl=59m59s
add address=2.3.4.5 disabled=yes name=www2.company.eu
add address=10.1.7.1 name=Mobile_10.1.7.1 ttl=8m25s
add address=192.168.1.200 name=Mobile_192.168.1.200 ttl=8m1s
add address=10.1.7.2 name=Mobile_10.1.7.2 ttl=9m27s
/ip firewall address-list
add address=193.56.29.119 list=ip_scanner_list
add address=23.101.7.53 list=ip_scanner_list
/ip firewall filter
add action=fasttrack-connection chain=forward comment=fasttrack \
    connection-state=established,related hw-offload=yes
add action=accept chain=input comment="Rule necessary for let's encrypt (disab\
    le NAT forwarding for port 80 in that case)" disabled=yes dst-port=80 \
    in-interface=ether01-WAN protocol=tcp
add action=accept chain=forward comment="Forward all related" \
    connection-state=established,related
add action=accept chain=input comment="Established connections" \
    connection-state=established
add action=accept chain=input comment="Related connections" connection-state=\
    related
add action=drop chain=input comment="drop scanner" in-interface-list=WAN \
    src-address-list=ip_scanner_list
add action=drop chain=input comment="Block Guest network" connection-state=\
    !established,related dst-address=10.0.0.0/8 in-interface=bridge-guest
add action=accept chain=input comment=icmp in-interface=ether01-WAN protocol=\
    icmp
add action=accept chain=input comment="input tcp-ports from ether01-WAN" \
    disabled=yes dst-port=443,80,25,5014,987,1723,4443,1194,321,7999 \
    in-interface=ether01-WAN protocol=tcp
add action=accept chain=input comment="input tcp-ports from ether02-WAN" \
    disabled=yes dst-port=80 in-interface=ether02-WAN protocol=tcp
add action=accept chain=input comment="accepted upd ports from WAN" dst-port=\
    500,4500,1701,1194 in-interface=ether01-WAN protocol=udp
add action=accept chain=input disabled=yes dst-port=1194 in-interface=\
    bridge-guest protocol=udp
add action=accept chain=input comment="accept ipsec protocol from WAN" \
    disabled=yes in-interface-list=WAN protocol=ipsec-esp
add action=drop chain=input comment="drop all the rest from internet" \
    in-interface-list=WAN
add action=drop chain=forward comment="keep Foscam quiet" dst-address=\
    !10.0.0.0/8 src-address=10.1.5.0/24
add action=accept chain=forward comment="forward all the rest"
add action=accept chain=input comment="allow everything"
/ip firewall mangle
add action=mark-connection chain=prerouting connection-mark=connection-WAN2 \
    disabled=yes new-connection-mark=connection-WAN2 passthrough=no
add action=mark-routing chain=prerouting dst-address=!10.0.0.0/8 \
    new-routing-mark=rtab-ether02-WAN passthrough=no src-address=10.1.2.33
add action=mark-routing chain=prerouting connection-mark=connection-WAN2 \
    disabled=yes new-routing-mark=rtab-ether02-WAN passthrough=no
add action=jump chain=prerouting connection-mark=no-mark connection-state=\
    established,related,new disabled=yes jump-target=mark-connections
add action=mark-connection chain=mark-connections connection-state=new \
    disabled=yes in-interface=ether01-WAN new-connection-mark=connection-WAN1 \
    passthrough=yes
add action=mark-connection chain=mark-connections connection-mark=no-mark \
    connection-state=established,related,new disabled=yes dst-port=443 \
    in-interface=ether02-WAN new-connection-mark=connection-WAN2 passthrough=\
    no protocol=tcp
add action=accept chain=prerouting connection-mark=no-mark disabled=yes
add action=mark-routing chain=prerouting connection-mark=connection-WAN2 \
    disabled=yes new-routing-mark=rtab-ether02-WAN passthrough=no
/ip firewall nat
add action=masquerade chain=srcnat comment="masq everything going out on WAN" \
    out-interface-list=WAN
add action=masquerade chain=srcnat comment="Masquerade 10.0.0.0 - guest NW" \
    out-interface=bridge-guest src-address=10.0.0.0/8 to-addresses=\
    192.168.1.1
add action=masquerade chain=srcnat disabled=yes out-interface=bridge-company \
    src-address=!10.0.0.0/8
add action=src-nat chain=srcnat comment="redirect internal requests (10.x.x.x \
    --> 10.1.1.1 -> 10.1.2.24 --> 10.1.1.1 --> 10.x.x.x) " dst-address=\
    10.1.2.24 to-addresses=10.1.1.1
add action=src-nat chain=srcnat dst-address=10.1.2.33 src-address=10.0.0.0/8 \
    to-addresses=10.1.1.1
add action=src-nat chain=srcnat dst-address=10.1.2.18 to-addresses=10.1.1.1
add action=src-nat chain=srcnat dst-address=10.1.1.20 to-addresses=10.1.1.1
add action=dst-nat chain=dstnat comment=\
    "web services with special ports --> send to nginx reverse proxy server" \
    dst-port=237,5115,5311,5312,5212,5122,7999 protocol=tcp to-addresses=\
    10.1.2.24
add action=dst-nat chain=dstnat comment="smtp goes to zarafa server" \
    dst-port=25 in-interface=ether01-WAN protocol=tcp to-addresses=10.1.2.18
add action=dst-nat chain=dstnat comment=\
    "all http and https go to nginx reverse proxy" dst-port=80,443 \
    in-interface=ether01-WAN protocol=tcp to-addresses=10.1.2.24
add action=dst-nat chain=dstnat comment="zarafa access" dst-port=237 \
    in-interface=ether01-WAN protocol=tcp to-addresses=10.1.2.18 to-ports=237
add action=dst-nat chain=dstnat comment=\
    "Forward TCP ports from ether02-WAN to 10.1.2.33" dst-port=80,443 \
    in-interface=ether02-WAN protocol=tcp to-addresses=10.1.2.33
add action=dst-nat chain=dstnat comment=\
    "all https to www.company.eu from ppp goes to nginx reverse proxy " \
    dst-address=1.2.3.4 dst-port=443 in-interface=all-ppp protocol=tcp \
    to-addresses=10.1.2.24
add action=dst-nat chain=dstnat dst-address=2.3.4.5 dst-port=443 \
    in-interface=all-ppp protocol=tcp to-addresses=10.1.2.33
add action=dst-nat chain=dstnat comment=\
    "all https requests forward to nginx reverse proxy server" dst-address=\
    1.2.3.4 dst-port=443 in-interface=bridge-company protocol=tcp \
    to-addresses=10.1.2.24
add action=dst-nat chain=dstnat dst-address=2.3.4.5 dst-port=443 \
    in-interface=bridge-company protocol=tcp to-addresses=10.1.2.33
add action=dst-nat chain=dstnat comment="openvpn gateway to 10.1.1.20" \
    disabled=yes dst-port=1194 in-interface=ether01-WAN protocol=udp \
    to-addresses=10.1.1.20
add action=dst-nat chain=dstnat comment="7 Days 2 Die - Server UDP ports" \
    disabled=yes dst-port=26900-26903 in-interface=ether01-WAN protocol=udp \
    to-addresses=10.1.2.29
add action=dst-nat chain=dstnat comment="7 Days 2 Die - Server TCP ports" \
    disabled=yes dst-port=26900 in-interface=ether01-WAN protocol=tcp \
    to-addresses=10.1.2.29
/ip route
add disabled=yes distance=1 dst-address=10.1.0.0/16 gateway=bridge-company \
    pref-src="" routing-table=*400 scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=yes distance=1 dst-address=192.168.100.1/32 gateway=1.2.3.1 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=2.3.4.1 \
    routing-table=rtab-ether02-WAN suppress-hw-offload=no
add disabled=yes distance=1 dst-address=10.34.0.0/16 gateway=1.2.3.1 \
    pref-src="" routing-table=main scope=30 suppress-hw-offload=no \
    target-scope=10
add disabled=no distance=1 dst-address=0.0.0.0/0 gateway=3.4.5.1 \
    pref-src="" routing-table=rtab-ether03-WAN suppress-hw-offload=no
add disabled=no dst-address=0.0.0.0/0 gateway=4.5.6.1 routing-table=\
    rtab-ether04-WAN suppress-hw-offload=no
/ip service
set www address=10.0.0.0/8
set www-ssl certificate=letsencrypt-autogen_2022-09-18T13:53:26Z
/ip ssh
set allow-none-crypto=yes forwarding-enabled=remote
/ppp secret
add name=scompany profile=iPhone
add name=10.4.1.1 profile=VPN
add name=10.2.1.1 profile=VPN
add name=10.3.1.1 profile=VPN
add name=10.5.1.1 profile=VPN
add name=10.6.1.1 profile=VPN
add local-address=172.16.0.1 name=10.4.1.1_openvpn profile=profile-openvpn \
    remote-address=172.16.0.4 routes=172.16.0.0/24
add name=uid0002 profile=iPhone
add name=10.1.1.1 profile=VPN
/routing ospf interface-template
add area=backbone-v2 auth-id=1 disabled=no interfaces=bridge-company \
    priority=1 type=ptp
add area=backbone-v2 disabled=no interfaces=bridge-guest type=ptp
add area=backbone-v2 cost=20 disabled=no interfaces=dynamic type=ptp
add area=backbone-v2 cost=10 disabled=no interfaces=l2tp-10.2.1.1 type=ptp
add area=backbone-v2 cost=10 disabled=no interfaces=l2tp-10.3.1.1 type=ptp
/routing rule
add action=lookup disabled=yes interface=bridge-guest table=rtab-WAN
add action=lookup disabled=yes dst-address=10.1.0.0/16 interface=bridge-guest \
    table=rtab-WAN
add action=lookup comment="All routes from 10.1.2.21 shall go through routing \
    table rtab-ether02-WAN" disabled=yes src-address=10.1.2.33 table=\
    rtab-ether02-WAN
/system clock
set time-zone-name=Europe/Vienna
/system identity
set name=aaxxx01rt001
/system logging
set 1 action=disk
set 2 action=disk
add disabled=yes topics=ovpn
add action=disk topics=watchdog
/system ntp client
set enabled=yes
/system ntp server
set enabled=yes manycast=yes multicast=yes
/system ntp client servers
add address=0.at.pool.ntp.org
add address=1.at.pool.ntp.org
add address=2.at.pool.ntp.org
add address=3.at.pool.ntp.org
/system scheduler
add interval=30s name=schedule-MobileDNS on-event=script-MobileDNS policy=\
    ftp,reboot,read,write,policy,test,password,sniff,sensitive start-time=\
    startup
/system script
add dont-require-permissions=no name=script-MobileDNS owner=admin policy=\
    read,write source="# Set IP ranges\
    \n:local ipRangeArray {10.1.7.0/24; 192.168.1.200/29};\
    \n:local Prefix \"Mobile_\";\
    \n\
    \n# Remove all Mobile records with less or equal 30 sec\
    \n:foreach i in=[/ip dns static find ((name~\$Prefix) and ttl<=\"00:00:30\
    \")] do={\
    \n\t/ip dns static remove \$i;\
    \n}\
    \n\
    \n# All other Mobiles reduce by 30 sec\
    \n:foreach i in=[/ip dns static find ((name~\$Prefix) and ttl>\"00:00:30\"\
    )] do={\
    \n\t/ip dns static set \$i ttl=([/ip dns static get \$i ttl]-\"00:00:30\")\
    ;\
    \n}\
    \n\
    \n# Add entries at the static dns list\
    \n:foreach ipRange in=\$ipRangeArray do={\
    \n\t:foreach i in=[/ip dhcp-server lease find ((address in \$ipRange) and \
    (expires-after > \"00:00:00\"))] do={\
    \n\t\t:local bFound false;\
    \n\t\t:foreach di in [/ip dns static find name=(\$Prefix.[/ip dhcp-server \
    lease get \$i address])] do={\
    \n\t\t\t:set bFound true;\
    \n\t\t\t/ip dns static set \$di ttl=[/ip dhcp-server lease get \$i expires\
    -after];\
    \n\t\t}\
    \n\t\t:if (\$bFound = false) do={\
    \n\t\t\t/ip dns static add name=(\"Mobile_\".[/ip dhcp-server lease get \$\
    i address]) address=[/ip dhcp-server lease get \$i address] ttl=[/ip dhcp-\
    server lease get \$i expires-after];\
    \n\t\t}\
    \n\t}\
    \n}\
    \n"
add dont-require-permissions=no name=test owner=admin policy=read,write \
    source="# Remove all Mobile records with less or equal 1 min\
    \n:local ipRangeArray {10.1.7.0/24; 192.168.16.200/3}\
    \n\
    \n:foreach ipRange in=\$ipRangeArray do={\
    \n\t:put \$ipRange ;\
    \n}\
    \n"
/system watchdog
set ping-start-after-boot=15m ping-timeout=10m watch-address=8.8.8.8
/tool graphing interface
add
/tool graphing resource
add
/tool romon port
add

What I want to achieve is that every connection entering ether02-WAN shall leave ether02-WAN, the same for ether03-WAN and ether04-WAN.

Hope you have now everything necessary to tell me where is my mistake in the previous posts.

regards and thanks in advance

sindy · Sun Oct 23, 2022 7:59 pm

The mangle rules in your complete export show some signs of nervosity - having both connection-mark=xxx (a match condition) and new-connection-mark=xxx (parameter of the action) in the same action=mark-connection rule makes that rule useless.

In your original mangle rules in post #34, the only mistake was in rule 4 - chain=prerouting action=accept in-interface=ether01-WAN. The purpose of this rule is to prevent packets coming in from the internet from ever reaching the action=mark-routing rules, and so it should either match on in-interface-list=WAN (which contains all WAN interfaces), or at least it should match on ether02-WAN rather than ether01-WAN for the case where you only handle WAN 2.

So assuming that the default route in routing table main uses ether01-WAN, the complete mangle rules handling all the remaining 3 WANs would look as follows:

/ip firewall mangle
add chain=prerouting connection-state=new action=jump jump-target=mark-connections
add chain=prerouting in-interface-list=WAN action=accept
add chain=prerouting connection-mark=connection-WAN2 action=mark-routing new-routing-mark=rtab-ether02-WAN passthrough=no
add chain=prerouting connection-mark=connection-WAN3 action=mark-routing new-routing-mark=rtab-ether03-WAN passthrough=no
add chain=prerouting connection-mark=connection-WAN4 action=mark-routing new-routing-mark=rtab-ether04-WAN passthrough=no

add chain=mark-connections connection-mark=no-mark in-interface=ether02-WAN action=mark-connection new-connection-mark=connection-WAN2 passthrough=yes
add chain=mark-connections connection-mark=no-mark in-interface=ether03-WAN action=mark-connection new-connection-mark=connection-WAN3 passthrough=yes
add chain=mark-connections connection-mark=no-mark in-interface=ether04-WAN action=mark-connection new-connection-mark=connection-WAN4 passthrough=yes

se232 · Mon Oct 24, 2022 9:42 am

Hello sindy,

thank you for the quick answer.
I now implemented the rules exactly according your proposal.
Access from outside can be established, but not predicatable, and the performance is very poor.
A new access takes up to several seconds, when it is established at all (typically it take a few seconds, up to 12-15, too)
But around the half of the accesses from outside lead into a timeout (20 seconds).
In comparison, access to ether01-WAN (no mangle rules), the answer is received on the other side immediately (less than a second).
When I disable the mangle rules and enable the one to mark-routing everything from the dedicated internal server to ether02-WAN, then the connection is much faster and stable.
Testing scenario:
Remote PC (IP is 62.x.x.x Webbrowser is Edge), trying to open a webpage on ether02-WAN (IP is 80.y.y.y) via the internet.

What is missing?

sindy · Mon Oct 24, 2022 9:55 am

What is missing?

Add a match condition connection-mark=no-mark to the action=fasttrack-connection rule in chain forward in /IP firewall filter to prevent marked connections from getting fasttracked. Most, but not all, packets belonging to fasttracked connection bypass mangle.

se232 · Mon Oct 24, 2022 10:27 am

Hello sindy,

I was playing around and had to found out that the fasttrack causes this instability.
Once disabled, the mangle rules work as expected and the respond is received immediately.
With fasttrack on, the problem appears again.

Is the fasttrack rule wrong?

 1 X  ;;; fasttrack
      chain=forward action=fasttrack-connection hw-offload=yes connection-state=established,related log=no

regards

sindy · Mon Oct 24, 2022 10:29 am

See my post above.

se232 · Mon Oct 24, 2022 12:01 pm

Hello Sindy,

thanks a lot for the support!
Now it's up and running

Who is online