Adding static route won't bypass nat

nagylzs · Tue Jan 05, 2021 9:54 pm

I have two routers connected via IPSEC/IKEv2 in tunnel mode. They don't have explicit firewall NAT bypass rules added. They have this instead:

/interface bridge
add name=vpn-blackhole
/ip route
add comment="Blackhole for RFC 1918" distance=5 dst-address=10.0.0.0/8 gateway=vpn-blackhole
add comment="Blackhole for RCF 1918 class B" distance=5 dst-address=172.16.0.0/12 gateway=\
    vpn-blackhole
add comment="Blackhole for RCF 1918 class C" distance=5 dst-address=192.168.0.0/16 gateway=\
    vpn-blackhole

And it works! I have another two devices where the same thing does not work, and I cannot figure out why. I was so frustrated that I went into big depths with this problem. I have installed GNS3, created virtual CHR routers and virtual machines. All of this just to find what the problem is, but I still don't know.

Here is my test network:

https://imgur.com/a/yoAlWON

The internet is simulated with the top router. The two sites are connected to it with two more routers. All of the routers are running RouterOs CHR in VMWare.

Since this environment was created just to find the solution to this problem, I'm going to post full unaltered configs here.

Here is the config for the "office" site:

/interface bridge
add name=bridge-office
add name=vpn-blackhole
/interface ethernet
set [ find default-name=ether1 ] name=ether1-internet
/interface list
add name=LAN
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec policy group
add name=branch01.myserver.hu
/ip ipsec profile
add dh-group=modp2048 enc-algorithm=aes-256 hash-algorithm=sha256 name=myserver.hu proposal-check=strict
/ip ipsec peer
add exchange-mode=ike2 local-address=100.3.3.10 name=peer-branch01 passive=yes profile=myserver.hu
/ip ipsec proposal
add auth-algorithms=sha256 enc-algorithms=aes-256-cbc name=proposal-myserver.hu pfs-group=modp2048
/ip pool
add name=pool-office ranges=172.16.2.100-172.16.2.200
/ip dhcp-server
add address-pool=pool-office disabled=no interface=bridge-office name=dhcp-office
/interface bridge port
add bridge=bridge-office interface=ether2
add bridge=bridge-office interface=ether3
/interface list member
add interface=ether1-internet list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
/ip address
add address=172.16.2.1/24 interface=bridge-office network=172.16.2.0
add address=100.3.3.10/24 interface=ether1-internet network=100.3.3.0
/ip dhcp-server network
add address=172.16.2.0/24 dns-server=172.16.2.1 gateway=172.16.2.1
/ip dns
set allow-remote-requests=yes servers=100.3.3.3
/ip firewall filter
add action=accept chain=input comment="Allow UPD 500,4500 for IKEv2" dst-address=100.3.3.10 dst-port=500,4500 in-interface=ether1-internet protocol=udp
add action=accept chain=input comment="Allow IPSEC/ESP" dst-address=100.3.3.10 in-interface=ether1-internet protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=accept chain=forward in-interface-list=LAN src-address=172.16.2.0/24
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
/ip firewall mangle
add action=change-mss chain=forward comment="IKE2: Clamp TCP MSS from 172.16.1.0/24 to ANY" ipsec-policy=in,ipsec new-mss=1360 passthrough=yes protocol=tcp src-address=172.16.1.0/24 \
    tcp-flags=syn tcp-mss=!0-1360
/ip firewall nat
add action=accept chain=srcnat comment="bypass nat for VPN tunnel between internal addresses" disabled=yes dst-address=172.16.0.0/12 src-address=172.16.0.0/12
add action=masquerade chain=srcnat out-interface=ether1-internet
/ip ipsec identity
add auth-method=digital-signature certificate=office.myserver.hu match-by=certificate my-id=fqdn:office.myserver.hu peer=peer-branch01 policy-template-group=branch01.myserver.hu \
    remote-certificate=branch01.myserver.hu remote-id=fqdn:branch01.myserver.hu
/ip ipsec policy
add dst-address=172.16.1.0/24 peer=peer-branch01 proposal=proposal-myserver.hu sa-dst-address=100.2.2.10 sa-src-address=100.3.3.10 src-address=172.16.2.0/24 tunnel=yes
/ip route
add distance=1 gateway=100.3.3.3
add comment="Blackhole for RCF 1918 class B" distance=1 dst-address=172.16.0.0/12 gateway=vpn-blackhole
/system identity
set name=office.myserver.hu

Here is the unaltered config for the "branch01" site:

/interface bridge
add name=bridge-branch01
add name=vpn-blackhole
/interface ethernet
set [ find default-name=ether1 ] name=ether1-internet
/interface list
add name=LAN
add name=WAN
/interface wireless security-profiles
set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec policy group
add name=office.myserver.hu
/ip ipsec profile
add dh-group=modp2048 enc-algorithm=aes-256 hash-algorithm=sha256 name=myserver.hu proposal-check=strict
/ip ipsec peer
add address=100.3.3.10/32 exchange-mode=ike2 local-address=100.2.2.10 name=peer-office profile=myserver.hu
/ip ipsec proposal
add auth-algorithms=sha256 enc-algorithms=aes-256-cbc name=proposal-myserver.hu pfs-group=modp2048
/ip pool
add name=pool-branch ranges=172.16.1.100-172.16.1.200
/ip dhcp-server
add address-pool=pool-branch disabled=no interface=bridge-branch01 name=dhcp-branch01
/interface bridge port
add bridge=bridge-branch01 interface=ether2
add bridge=bridge-branch01 interface=ether3
/interface list member
add interface=ether1-internet list=WAN
add interface=ether2 list=LAN
add interface=ether3 list=LAN
add interface=ether4 list=LAN
add interface=ether5 list=LAN
/ip address
add address=172.16.1.1/24 interface=bridge-branch01 network=172.16.1.0
add address=100.2.2.10/24 interface=ether1-internet network=100.2.2.0
/ip dhcp-server network
add address=172.16.1.0/24 dns-server=172.16.1.1 gateway=172.16.1.1
/ip dns
set allow-remote-requests=yes servers=100.2.2.2
/ip firewall filter
add action=accept chain=input comment="Allow UPD 500,4500 for IKEv2" dst-address=100.2.2.10 dst-port=500,4500 in-interface=ether1-internet protocol=udp
add action=accept chain=input comment="Allow IPSEC/ESP" dst-address=100.2.2.10 in-interface=ether1-internet protocol=ipsec-esp
add action=accept chain=input comment="defconf: accept established,related,untracked" connection-state=established,related,untracked
add action=drop chain=input comment="defconf: drop invalid" connection-state=invalid
add action=accept chain=input comment="defconf: accept ICMP" protocol=icmp
add action=drop chain=input comment="defconf: drop all not coming from LAN" in-interface-list=!LAN
add action=accept chain=forward comment="defconf: accept in ipsec policy" ipsec-policy=in,ipsec
add action=accept chain=forward comment="defconf: accept established,related, untracked" connection-state=established,related,untracked
add action=accept chain=forward in-interface-list=LAN src-address=172.16.1.0/24
add action=drop chain=forward comment="defconf:  drop all from WAN not DSTNATed" connection-nat-state=!dstnat connection-state=new in-interface-list=WAN
add action=drop chain=forward comment="defconf: drop invalid" connection-state=invalid
/ip firewall mangle
add action=change-mss chain=forward comment="IKE2: Clamp TCP MSS from 172.16.2.0/24 to ANY" ipsec-policy=in,ipsec new-mss=1360 passthrough=yes protocol=tcp src-address=172.16.2.0/24 tcp-flags=syn tcp-mss=\
    !0-1360
/ip firewall nat
add action=accept chain=srcnat disabled=yes dst-address=172.16.0.0/12 src-address=172.16.0.0/12
add action=masquerade chain=srcnat out-interface=ether1-internet
/ip ipsec identity
add auth-method=digital-signature certificate=branch01.myserver.hu my-id=fqdn:branch01.myserver.hu peer=peer-office policy-template-group=office.myserver.hu remote-id=fqdn:office.myserver.hu
/ip ipsec policy
add dst-address=172.16.2.0/24 peer=peer-office proposal=proposal-myserver.hu sa-dst-address=100.3.3.10 sa-src-address=100.2.2.10 src-address=172.16.1.0/24 tunnel=yes
/ip route
add distance=1 gateway=100.2.2.2
add comment="Blackhole for RCF 1918 class B" distance=1 dst-address=172.16.0.0/12 gateway=vpn-blackhole
/system identity
set name=branch01.myserver.hu

I'm testing this from PC1=172.16.1.10 and PC2=172.16.2.10 - they are pinging each other with TCP ping continuously.

This is what I see on PC2:

Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
*100.3.3.10 tcp_seq=7142 ttl=64 time=162.140 ms (ICMP type:3, code:1, Destination host unreachable)
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout
Connect   7@172.16.1.10 timeout

Most of the time I just get timeout messages, but sometimes I get a "Destination host unreachable" from the 100.3.3.10 test router.

The same thing happens on PC1:

Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
*100.2.2.10 tcp_seq=6193 ttl=64 time=156.605 ms (ICMP type:3, code:1, Destination host unreachable)
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
*100.2.2.10 tcp_seq=6207 ttl=64 time=161.157 ms (ICMP type:3, code:1, Destination host unreachable)
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout
Connect   7@172.16.2.10 timeout

It proves that these packages are not going through the IPSEC tunnel. They are masqueraded and sent out on ether1-internet to the default gateway.

If I enable this first NAT bypass rule under /ip firewall nat

[admin@office.myserver.hu] /ip firewall nat> print 
Flags: X - disabled, I - invalid, D - dynamic 
 0 X  ;;; bypass nat for VPN tunnel between internal addresses
      chain=srcnat action=accept src-address=172.16.0.0/12 dst-address=172.16.0.0/12 

 1    chain=srcnat action=masquerade out-interface=ether1-internet 
[admin@office.myserver.hu] /ip firewall nat> set 0 disabled=no

Then PC2 still cannot ping PC1 for about 10-30 seconds, then it starts to work. The same is true for the branch01 router - if I enable the manually added NAT bypass rule, then it works. Otherwise it doesn't.

Here is what I don't understand. These routers already have blackhole/fake routes to all B class networks:

/ip route
add distance=1 gateway=100.3.3.3
add comment="Blackhole for RCF 1918 class B" distance=1 dst-address=172.16.0.0/12 gateway=vpn-blackhole

I think this general masquerade NAT rule should not match:

/ip firewall nat
add chain=srcnat action=masquerade out-interface=ether1-internet

But it does! In other words, I'm not able to bypass NAT by adding routes to portless bridges inside the test environment.

But if I do the same in production, then it works!

I spent two days putting together GNS3 and the whole test environment and I still don't understand what is happening here. Pulling my hair out. :-)

I apologize, it was a bit long introduction...

The basic question is this:

* On the office router, I have this static route to the portless vpn-blackhole bridge
* But the packets that should be forwarded from 172.16.2.0/24 to 172.16.1.0/24 are getting maqueraded, like if they are meant to be sent out on the ether1-internet interface. Why?
* Not that important but why does it work on other devices, and not on some others? (But I guess once I know the answer to the first one, I'll also know the answer to the second one too...)

Sob · Tue Jan 05, 2021 11:55 pm

I don't see it, but srcnat's condition is out-interface=ether1-internet, so it will match only connections going out via ether1-internet. That will happen when router thinks that route to destination leads there. And that should only happen when your static route is either not active, or if there's another better route. Start by checking "/ip route print".

nagylzs · Wed Jan 06, 2021 9:24 am

I don't see it, but srcnat's condition is out-interface=ether1-internet, so it will match only connections going out via ether1-internet. That will happen when router thinks that route to destination leads there. And that should only happen when your static route is either not active, or if there's another better route. Start by checking "/ip route print".

That's what I thought, but the routing table is as simple as it can be.

Office:

[admin@office.myserver.hu] > /ip route print 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          100.3.3.3                 1
 1 ADC  100.3.3.0/24       100.3.3.10      ether1-internet           0
 2 A S  ;;; Blackhole for RCF 1918 class B
        172.16.0.0/12                      vpn-blackhole             1
 3 ADC  172.16.2.0/24      172.16.2.1      bridge-office             0
[admin@office.myserver.hu] >

Branch01:

[admin@branch01.myserver.hu] > /ip route print 
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme, B - blackhole, U - unreachable, P - prohibit 
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 A S  0.0.0.0/0                          100.2.2.2                 1
 1 ADC  100.2.2.0/24       100.2.2.10      ether1-internet           0
 2 A S  ;;; Blackhole for RCF 1918 class B
        172.16.0.0/12                      vpn-blackhole             1
 3 ADC  172.16.1.0/24      172.16.1.1      bridge-branch01           0
[admin@branch01.myserver.hu] >

Also tried to increase distance for the default route but it did not help. It should not matter anyway: the 172.16.0.0/12 destination is much more specific than 0.0.0.0/0, the router should always use that regardless of the distance. But it doesn't.

But why? Is there a way to debug the routing decision inside RouterOS?

Sob · Thu Jan 07, 2021 6:27 am

It's something with IPSec, I can reproduce it. When I add logging rule in forward chain, then with peer disabled it shows vpn-blackhole as outgoing interface, but with peer enabled it changes to ether1-internet. It's kind of right, because it's actually where packets go to, but I'm not sure if firewall should see it like this, I never checked this before.

You can try the same logging rule on router where it works. Also, what about RouterOS versions? Is there perhaps something else (and probably older) on working router?

nagylzs · Thu Jan 07, 2021 6:12 pm

It's something with IPSec, I can reproduce it. When I add logging rule in forward chain, then with peer disabled it shows vpn-blackhole as outgoing interface, but with peer enabled it changes to ether1-internet. It's kind of right, because it's actually where packets go to, but I'm not sure if firewall should see it like this, I never checked this before.

The IPSEC packets are going out on ether1-internet but layer 3 routing for dst-address=172.16.0.0/12 should never think that they will go out on ether1-internet.

You can try the same logging rule on router where it works. Also, what about RouterOS versions? Is there perhaps something else (and probably older) on working router?

I'm using CHR 6.46.8 on them. I'll do some logging and get back with the results.

nagylzs · Thu Jan 07, 2021 6:27 pm

Test on branch01 router.

First disabled the NAT masquerade bypass rule and added logging to the general masquerade rule:

/ip firewall nat
add action=accept chain=srcnat disabled=yes dst-address=172.16.0.0/12 log=yes src-address=172.16.0.0/12
add action=masquerade chain=srcnat log=yes out-interface=ether1-internet

Then I started ping from internal lan PC1 to remote lan PC2 address:

17:22:28 firewall,info srcnat: in:(unknown 0) out:ether1-internet, src-mac 00:50:79:66:68:00, proto ICMP (type 8, code 0), 172.16.1.10->172.16.2.10, len 84 
17:22:30 firewall,info srcnat: in:(unknown 0) out:ether1-internet, src-mac 00:50:79:66:68:00, proto ICMP (type 8, code 0), 172.16.1.10->172.16.2.10, len 84 
17:22:32 firewall,info srcnat: in:(unknown 0) out:ether1-internet, src-mac 00:50:79:66:68:00, proto ICMP (type 8, code 0), 172.16.1.10->172.16.2.10, len 84 
17:22:34 firewall,info srcnat: in:(unknown 0) out:ether1-internet, src-mac 00:50:79:66:68:00, proto ICMP (type 8, code 0), 172.16.1.10->172.16.2.10, len 84

So it executes the masquerade rule.

Then I disabled the peer:

/ip ipsec
set 0 disabled=yes

Then I started the same ping again and nothing is logged.

So yes, you were right: enabling the ipsec peer will change the route that is applied for a firewall NAT rule.

I believe this is a bug in RouterOS.

If there is anybody here who can come up with an explanation for this (other then being buggy) then please don't hesitate to post it.

nagylzs · Thu Jan 07, 2021 6:39 pm

Okay here is the production network that works.

Office side has LAN address 192.168.13.0/24 running on RouterBOARD 750G r3 (6.46.8), branch01 has LAN address 192.168.14.0/24 running on HAP AC2 (6.46.8)

On the office site I have these static routes:

/ip route
add comment="Blackhole for RCF 1918 class A" distance=10 dst-address=10.0.0.0/8 gateway=vpn-blackhole
add comment="Blackhole for RCF 1918 class B" distance=10 dst-address=172.16.0.0/12 gateway=vpn-blackhole
add comment="Blackhole for RCF 1918 class C" distance=10 dst-address=192.168.0.0/16 gateway=vpn-blackhole

And this single NAT rule:

/interface list
add comment=defconf name=WAN
add comment=defconf name=LAN
/interface list member
add comment=defconf interface=bridge list=LAN
add comment=defconf interface=ether1-internet list=WAN
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN

And it works, without the extra accept nat rule!

[gandalf@no.telling.com] > /tool traceroute 192.168.14.117
 # ADDRESS                          LOSS SENT    LAST     AVG    BEST   WORST STD-DEV STATUS
 1 192.168.14.1                       0%    9  12.3ms    18.1    12.3    34.3     6.5
 2 192.168.14.117                     0%    9  12.5ms    16.7    12.3      28     5.1

It works on this particular RG750 router and does not work on the CHR router???

nagylzs · Thu Jan 07, 2021 6:42 pm

Also tried to add "ipsec-policy=out,none" on the CHR (that was the only notable difference that I could see) but it did not help for about one minute.

Then after a minute, it started to work.

Doing some more tests...

nagylzs · Thu Jan 07, 2021 6:53 pm

All right, so the production version was working because of the extra "ipsec-policy=out,none" condition was added to the masquerade rule. If I remove that on the production system then it also fails.

But here is the interesting part: it only works if ipsec-policy=out,none is specified AND a blackhole static route is added. If I remove either one of them, then it fails again.

Is it how it is supposed to be? I do not fully understand what happens here.

Here is what I think:

* BEFORE the package is encapsulated, the out interface is not WAN, so we don't need the "ipsec-policy=out,none". Or do we? If we need it then can somebody explain why?
* AFTER the package is encapsulated, the IPSEC package already has src-address=<router's wan address>, so it should not matter if the masquerade rule matches or not.

Now I know how to overcome the problem. But I don't understand WHY do I need to add "ipsec-policy=out,none" ?

Sob · Thu Jan 07, 2021 7:54 pm

I assumed that you used same rules, so it was strange why it would work on one device and not on another. If the working one has ipsec-policy=out,none, then it explains it. This condition matches only when there's no IPSec policy for packets. And when there is (like in this case), it doesn't match and srcnat is not applied.

That said, the seemingly ignored route and firewall seeing ether1-internet as outgoing interface is something I didn't expect, but I can't say that it's wrong, maybe it was always like this, as part of how IPSec works internally.

But here is the interesting part: it only works if ipsec-policy=out,none is specified AND a blackhole static route is added. If I remove either one of them, then it fails again.

That shouldn't happen. You must have some route to remote subnet, but even default one is enough (which is always present, unless you have some very special config). You do need to add extra route only in case you make the target subnet unavailable. For example, I like to add unreachable routes for whole private ranges to prevent unnecessary leaking packets, if something decides to send some to private subnet I don't have. But this breaks IPSec. To fix it, I have to add fake routes to remote subnets (same like you have here, with empty bridge as gateway).

nagylzs · Thu Jan 07, 2021 9:46 pm

I assumed that you used same rules, so it was strange why it would work on one device and not on another. If the working one has ipsec-policy=out,none, then it explains it.

Yes I'm sorry I did not notice that difference at first. But I still don't understand why ipsec-policy=out,none is needed.

Given this routing table:

[admin@not.telling.com] /ip route> print
Flags: X - disabled, A - active, D - dynamic, C - connect, S - static, r - rip, b - bgp, o - ospf, m - mme,
B - blackhole, U - unreachable, P - prohibit
 #      DST-ADDRESS        PREF-SRC        GATEWAY            DISTANCE
 0 ADS  0.0.0.0/0                          100.2.2.254              10
 1 A S  ;;; Blackhole for RCF 1918 class A
        10.0.0.0/8                         vpn-blackhole            10
 2 ADC  100.2.2.0/24       100.2.2.188     ether1-internet           0
 3 A S  ;;; Blackhole for RCF 1918 class B
        172.16.0.0/12                      vpn-blackhole            10
 4 A S  ;;; Blackhole for RCF 1918 class C
        192.168.0.0/16                     vpn-blackhole             5
 5 ADC  192.168.13.0/24    192.168.13.254  bridge                    0

How can RouterOS think that these packets need to go out on ether1-internet? The 192.168.0.0/16 destination is more specific than the 0.0.0.0/0. Why the default route is selected instead of a much more specific static route? There must be some kind of mechanism that I don't see.

I can live without an explanation, because now I have a recipe that works... but I would really like to know the exact reason. I don't like to be in the dark.

Sob · Thu Jan 07, 2021 10:25 pm

There are two things:

1) Routing and outgoing interface. Based on routes, outgoing interface should be vpn-blackhole. And that's true when IPSec is not active. Active IPSec clearly changes routing decision in some way. Again, it's not completely wrong, because it reflects where those packets really go. Only it's confusing, because now routes seem ignored.

There isn't even just some simple hidden route added by IPSec, because if it was, I wouldn't need those extra routes in my example (unreachable routes for preventing packet leaks). It would be interesting to know what exactly happens, maybe it would be possible to find out by studying Linux kernel internals, but for now just take it as fact, outgoing interface is ether1-internet.

2) Given that outgoing interface is ether1-internet, the need for ipsec-policy=out,none is obvious.

nagylzs · Thu Jan 07, 2021 11:08 pm

There are two things:

1) Routing and outgoing interface. Based on routes, outgoing interface should be vpn-blackhole. And that's true when IPSec is not active. Active IPSec clearly changes routing decision in some way. Again, it's not completely wrong, because it reflects where those packets really go. Only it's confusing, because now routes seem ignored.

But they are not ignored! If I remove the dummy rule (dst-address=192.168.0.0/16 gateway=vpn-blackhole) then it fails to work again. If you don't believe me then try it, you will see. So that blackhole rule is not ignored and it is needed. I would like to know why it is needed, and how it is used.

I see a big contradiction here.

* the original packet has destination address something like 192.168.14.117
* it is consumed by encapsulation, and a new IPSEC packet created with a totally different destination address = sa-dst-addres (e.g. something like 100.3.3.161 )
* the new ipsec packet is created in the "local-out" internal router process and it goes over the whole routing - mpls - switching process again before it is sent out on ether1-internet

There are two packets: the original (later encapsulated) and the ipsec packet, and both of them are routed.

We must be specific when we talk about "where they go" - let's talk about the original packet:

* If the default route is selected for the original packet, then we wouldn't need the blackhole route at all. But apparently that is not the case - without the blackhole route, the original packet gets masquerdade, it is not encapsulated, and goes out on ether1-internet, unencrypted. That is a fact.
* If the vpn-blackhole route is selected for the original packet (instead of the default route), then the masquerade rule should not match, because the vpn-blackhole bridge has no WAN interface, and the masquerade rule has an "out-interface-list=WAN" condition. But apparently that is not the case either, because the masquerade rule does match (unless I add the ipsec-policy=out,none).

So neither of them can be the case, and that is a contradiction. The most probable explanation is that I'm wrong about how packets are processed, in what order etc.

I went through the packet flow diagram ( https://wiki.mikrotik.com/wiki/Manual:Packet_Flow ) several times. The order of things should be:

* routing decision (selecting the route) should be the first
* then comes postrouting, it contains the srcnat chain and the masquerade rule (by the time the packet arrives there, the decision about the output interface was already made)
* the ipsec encapsuation is the last thing that should happen (and then a new IPSEC packet is created and the routing starts again)

I'm not able to find any packet flow scenario that would explain the actual behaviour.

There isn't even just some simple hidden route added by IPSec, because if it was, I wouldn't need those extra routes in my example (unreachable routes for preventing packet leaks). It would be interesting to know what exactly happens, maybe it would be possible to find out by studying Linux kernel internals, but for now just take it as fact, outgoing interface is ether1-internet.

The original (later encapsulated) packet does not go out on any interface, and the ipsec packet goes out on ether1-internet.

2) Given that outgoing interface is ether1-internet, the need for ipsec-policy=out,none is obvious.

Yes, I agree on this part. I just don't understand, if the route to vpn-blackhole is not selected/used, then what is it used for? It must be used for something, because if I remove it, then unencrypted packets start to leak on ehter1-internet.

Sob · Fri Jan 08, 2021 1:58 am

But they are not ignored! If I remove the dummy rule (dst-address=192.168.0.0/16 gateway=vpn-blackhole) then it fails to work again. If you don't believe me then try it, you will see. So that blackhole rule is not ignored and it is needed. I would like to know why it is needed, and how it is used.

Let's say they are "half-ignored". Valid route to destination must exist, it fails when it doesn't, but default route is enough for this. But it seems that beyond check for existence, it's not really used for routing decision. When I remove dummy/blackhole route, it works fine without it, as expected.

As for packet flow:

- device 172.16.2.10 sends packet to 172.16.1.10
- router sees it in prerouting with in-interface=bridge-office
- router checks if there's route to 172.16.1.10 (again, even default one to 0.0.0.0/0 would be fine)
- packet goes in forward with in-interface=bridge-office and out-interface=ether1-internet <- this is what suprised me, because it looks like routing decision ignored the route pointing to vpn-blackhole, I don't know what's the exact machanism that causes it
- packet goes in postrouting with out-interface=ether1-internet (that can't change now)
- srcnat is part of postrouting, and it can optionally change source address; when it does, packet may no longer match policy
- if packet matches policy, it's "snatched away" by IPSec, encrypted and send out; it's different packet and we don't need to worry about this one, because it's like any other output from router, there's no problem with that

I just don't understand, if the route to vpn-blackhole is not selected/used, then what is it used for? It must be used for something, because if I remove it, then unencrypted packets start to leak on ehter1-internet.

With the config you posted, with existing default route, it's either:
a) useless
b) serves as anti-leaking protection for unused subnets

Unencrypted packets will go to internet, if they don't match any IPSec policy. Your posted config included policy only for 172.16.1.0/24 <-> 172.16.2.0/24, so these packets can't leak, but packets to e.g. 172.16.100.x would and your rules with gateway=vpn-blackhole prevent that.

nagylzs · Fri Jan 08, 2021 9:58 pm

Let's say they are "half-ignored". Valid route to destination must exist, it fails when it doesn't, but default route is enough for this. But it seems that beyond check for existence, it's not really used for routing decision. When I remove dummy/blackhole route, it works fine without it, as expected.

Not for me. I have uploaded a video to demonstrate it. It contains live IP addresses but I don't care anymore.

https://www.youtube.com/watch?v=bWH7q8A ... e=youtu.be

* there is a default route
* the firewall masquerade rule has ipsec-policy=out,none condition added
* if I remove the dummy/blackhole route, then it does not work

This is a fact. I don't know why it is different for you.

Sob · Fri Jan 08, 2021 10:21 pm

Two things:

1) I'm not sure why you have sa-src-address=192.168.13.254, which is part of LAN subnet. But that probably don't have any negative effect.

2) There's a catch with accessing remote subnet from router itself over plain IPSec tunnels. It doesn't work with default config, because router chooses wrong source address not covered by policy. To make it work, it's necessary to do one of the following:

a) set the right source address, e.g. ping has src-address parameter
b) use srcnat to change source to right address
c) add a route (even fake one) to remote subnet and set its pref-src to router's LAN address covered by policy

My guess is that your fake routes influence address selection and make router choose 192.168.13.254 as source, even though it's not set as pref-src. You can test it with this:

/ip firewall mangle
add action=log chain=output dst-address=192.168.14.117 protocol=icmp

Keep the ping running, disable the fake route and see if source address changes for logged packets.

nagylzs · Fri Jan 08, 2021 11:15 pm

1) I'm not sure why you have sa-src-address=192.168.13.254, which is part of LAN subnet. But that probably don't have any negative effect.

It is coming from /ip ipsec peer menu, local-address attribute was set to it. I set it to 0.0.0.0 - after I did that, the sa-src-address attribute was changed to 0.0.0.0 on the policy. (By the way, sa-src-address is not an attribute of the policy. But it appears as such, even in "/ip ipsec policy export "output.)

2) There's a catch with accessing remote subnet from router itself over plain IPSec tunnels. It doesn't work with default config, because router chooses wrong source address not covered by policy. To make it work, it's necessary to do one of the following:

a) set the right source address, e.g. ping has src-address parameter
b) use srcnat to change source to right address
c) add a route (even fake one) to remote subnet and set its pref-src to router's LAN address covered by policy

My guess is that your fake routes influence address selection and make router choose 192.168.13.254 as source, even though it's not set as pref-src. You can test it with this:
Code: Select all
/ip firewall mangle
add action=log chain=output dst-address=192.168.14.117 protocol=icmp
Keep the ping running, disable the fake route and see if source address changes for logged packets.

You hit the nail on the head. /ping 192.168.14.117 src-address=192.168.13.254 did the trick, it works without the blackhole route. I also tried the mangle rule and the logging, and in the router uses its WAN address as the source address for ping.

The conclusion for me is that adding any kind of static route should not be used alone to bypass masquerade rules. (It may be possible, but I must be extremely cautious.) I'll always be using regular NAT rules to bypass masquerade rules, but I'll also always add blackhole rules to prevent leaking packets with private LAN dst-addresses.

I finally have answers to all of my questions. I couldn't have done this without your help.

Thank you! :-)

Sob · Fri Jan 08, 2021 11:47 pm

The conclusion for me is that adding any kind of static route should not be used alone to bypass masquerade rules.

Correct, not in combination with IPSec.

For the record, what I do about leaking packets is:

/ip route
add dst-address=10.0.0.0/8 type=unreachable
add dst-address=172.16.0.0/12 type=unreachable
add dst-address=192.168.0.0/16 type=unreachable

Then if there's more specific route (connected subnets, static routes), target will be available. But everything else is blocked. And client trying to connect to something unavailable will be properly notified (icmp "destination unreachable").

Problem with IPSec is what I wrote, active route must exist, but these say it doesn't. And IPSec itself doesn't add any routes to remote subnets. To fix it, I need to manually add routes to existing remote subnets:

/interface bridge
add name=ipsec protocol-mode=none
/ip route
add dst-address=192.168.89.0/24 gateway=ipsec pref-src=192.168.80.5
add dst-address=192.168.254.0/24 gateway=ipsec pref-src=192.168.80.5

I'm not excited about it, I'd prefer if it could be done automatically, but so far it doesn't seem possible (I don't count any asynchronous scripts). Bonus is that pref-src=<router's address> fixes communication with remote subnets from router itself (it also fixes "holes" in traceroute).

Optional finishing touch is:

/ip firewall filter
add action=reject chain=forward out-interface=ipsec reject-with=icmp-network-unreachable

to return more proper "network unreachable" intead of "host unreachable" when tunnel is down, but it's just a tiny little detail for perfectionists. Also looking at this, it shouldn't have surprised me that out-interface is WAN when tunnel is up, this older rule is proof that I knew it. :)

nagylzs · Sat Jan 09, 2021 3:11 pm

I already spent days with this. I'm putting together a site-to-site VPN tutorial (in Hungarian). If you don't mind then I would share some of your ideas there.

Sob · Sat Jan 09, 2021 6:18 pm

Sure, use anything you like, it's not like any of this would be personal secret of mine. If it was, I wouldn't share it. :)

Adding static route won't bypass nat

Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat - RouterOS bug?

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Re: Adding static route won't bypass nat

Who is online