Thank you, everything is clear/perfect now.
That link misses the most key points
where is the direction for input chain rules
If needed where is the direction for forward chain rules.
But what I find hardest is…Where is the exit/entry point of the tunnel (behind the router) in relation to interfaces, or faux IP address or IP subnet etc…
Folks,
I’m finetuning this solution, and something came across to me:
I have an active peer in IPsec/Active Peers tab, for my phone when it’s connected onto VPN:
I have a webserver running in the LAN (192.168.2.100, port 80), and I added 2 logs into “Firewall/Filter rules” at the very top:
- One with “RX” prefix: forward chain, TCP, dst port=80, action=log
- One with “TX” prefix: forward chain, TCP, src port=80, action=log
(And also, I have a dstnat: TCP, dstport=80, from ether1_wan (with in interface list), action=dst-nat to 192.168.2.100)
However investigating the log, when my phone is on VPN, and accesses this webserver, it seems exactly the same as if the whole traffic is not going through the VPN but a normal port fw:
RX from WAN:
1.) in:ether1_wan out:bridge 4.3.2.1:10000->192.168.2.100:80, NAT 4.3.2.1:10000->(192.168.1.10:80->192.168.2.100:80)
TX from LAN:
2.) in:bridge out:ether1_wan 192.168.2.100:80->4.3.2.1:10000, NAT (192.168.2.100:80->192.168.1.10:80)->4.3.2.1:10000
Could you please validate this? I think it’s going through the VPN, but these log lines makes some fear for me.
I don’t understand why I don’t see any mention for the “Dynamic address” of 192.168.200.200 anywhere in the chain.
Thank you!
There is a common problem - when the VPN tunnel becomes the default gateway for the VPN client, you have to make sure that it is not used for routing the transport packets, for obvious reasons. And whilst on Mikrotik, you must take care about this manually for all types of VPN except bare IPsec where RouterOS does some magic on its own (exepmting IPsec transport packets from matching IPsec traffic selectors), the Android implementation may have chosen to exempt the whole server address. If you have another public IP somewhere, on which you could sniff, you can verify this assumption.
Actually, if I check “myip” from Android during VPN, I can see TIK’s public address. So I believe it all goes through VPN, just I had some fear because of the packet logs.
So you are saying, that this implementation I’ve done here in this thread can be considered as “bare IPsec”, and RouterOS is just doing the magic for me?
Also, I’ve came across another thing, which I really need to solve.
It’s kind of merging two threads, Sindy. We have created a nice EoIP tunnels with bonding over IPsec earlier.
I’m marking some packets to go into this tunnel instead getting out on public WAN.
This also works for this VPN clients, ie:
Phone~~~TIK0~~~TIK1~~~WebServer
My issue is that I can see the packets in TIK1 (coming through the tunnel between TIK0 and TIK1), but TIK1 can’t send their replys back.
Why? Because:
- Step1 (TIK0):
in:wan out:bonding 192.168.200.200:10000->9.10.11.12:80 (9.10.11.12 is TIK1)
- Step2 (TIK1):
in:bonding out:wan 192.168.200.200:10000->9.10.11.12:80
in:wan out:bridge LASTIP_BEFORE_TIK1:10000->WEBSERVER_IN_TIK1_LAN:80, NAT LASTIP_BEFORE_TIK1:10000->(TIK1_WANIP:80->WEBSERVER_IN_TIK1_LAN:80)
- Step3 (TIK1), reply:
in:bridge out:wan WEBSERVER_IN_TIK1_LAN:80->LASTIP_BEFORE_TIK1:10000, NAT (WEBSERVER_IN_TIK1_LAN:80->TIK1_WANIP:80)->LASTIP_BEFORE_TIK1:10000
!!!! in:wan out:wan 9.10.11.12:80->192.168.200.200:10000, NAT 9.10.11.12:80->(TIK1_WANIP:10000->192.168.100.200:10000)
This last line is the problem, as it puts the reply to WAN, instead of bonding. Now I could create a static route, to explicitly show TIK1, that it can reach 192.168.200.0/24 on the bonding, but it feels a hack (this IP Pool can be changed in TIK0, and it would break TIK1’s config), thus I’ve tried, and it makes replys reaching back TIK0, but TIK0 can’t forward those replys to Phone over IPsec VPN.
So I’m considering something more elegant. Like some masq, or srcnat in TIK0? Or anything? Do you maybe have some ideas on this? How is this to be solved elegant in MikroTik’s beautiful world? 
Thank you so much!
Yes, all goes through VPN, except traffic to the public address of the responder (VPN server).
RouterOS is doing the magic in terms that even if the src-address of a policy includes the WAN IP from which the IPsec transport packets are sent, and dst-address of the same policy is 0.0.0.0/0 (or at least includes the address to which the IPsec transport packets are sent), i.e. the transport packets would normally match that policy, the fact that they are IPsec transport packet prevents RouterOS from matching them to the policy, so they take the regular route.
Whereas the Android doesn’t discriminate at such a deep level, and simply exempts all packets with dst-address of the responder, even other ones than IPsec transport ones, from the traffic selector matching.
The solution is included in this post. Start reading it from the last paragraph, which explains the relationship to your context.
Sindy, thank you so much!
At first, I didn’t get this.
Yes, that’s crystal clear I already knew this.
And wow, I can feel how my “language interpretation” evolves after drinking morning coffee: I can finally understand what’s happening under the hood in this case. This was the reason why my WebServer in Tik’s LAN (which is accessed via VPN) made those traffic logs, as they are indeed simple port forwards.
By any chance do you know how Windows is working in this term? Is it also not discriminating IPsec transport to dst-address of repsonder vs anything else to dst-address of responder?
Regarding this:
I got this, but I think I didn’t describe my issue with enough details:
Please consider I’ve a working tunnel between 2 Tiks (TIK0 and TIK1). There is a service in TIK1’s LAN, which I need to access from TIK0 (one lan client of it), and I want this traffic to go through the tunnel.
Therefore I have these mangle’s in TIK0:
/ip firewall mangle
add action=accept chain=prerouting comment="est. rel. connections without conmark are accepted to prevent next rules" connection-mark=no-mark connection-state=established,related
add action=accept chain=prerouting comment="download packets must not be routing-marked" connection-state=established,related in-interface-list=wan|tunnel
add action=mark-routing chain=prerouting comment="connection marked packets marked for route also" connection-mark=into-tunnel new-routing-mark=into-tunnel passthrough=no
add action=mark-connection chain=prerouting comment="only initial packets of connections get here past the rules above" connection-state=new dst-address-list=to-tunnel-dsts new-connection-mark=into-tunnel passthrough=yes
add action=mark-routing chain=prerouting comment="initial packets of connections which evaded the rules above get here with no connection mark; we just repeat the mark-routing rules above" connection-mark=into-tunnel new-routing-mark=into-tunnel passthrough=no
Obviously,
to-tunnel-dsts
contains the address of TIK1, therefore traffic to TIK1 address will go through the tunnel. This works perfectly (I have static routes defined in both TIKs to the other TIK: reachable via tunnel). So, all traffic directed to
to-tunnel-dsts
go into the tunnel, and all the rest just simply go out on the default gateway.
However, the picture is now a bit more complex as I’ve introduced a remote VPN client to TIK0. This remote VPN client still need to use the tunnel (between TIKs) if it wants to communicate with the service in TIK1’s LAN.
This doesn’t mean a problem until a point:
- in TIK0: above mangles are redirecting the traffic into the tunnel, even if it comes from the remote VPN client
- in TIK1: I can see the traffic coming through the tunnel, and it also successfully reaches the service on TIK1’s LAN
- in TIK1: reply’s route is not defined in ip/route, as the original packet contains the src address as the imaginary 192.168.200.200 (defined only in TIK0), and TIK1 has no idea on where shall it send this response.
I don’t want to create a new static route in TIK1 for 192.168.200.0 via tunnel, as it might be changing if I decide to change that imaginary pool in TIK0. Plus, I might want to use the same imaginary pool in TIK1 for something else later.
Sorry for this, but I really don’t see how mangles can help me more with this, as that imaginary IP seems making my situation hard.
So the problem is not in TIK1, I have many options to send back the reply:
- Static route
- Mangle
- NAT
The problem is that after TIK0 receives the reply, it tries to send it to 192.168.200.200, which is the phone’s Dynamic Address in IPsec, and this packet doesn’t seem arriving into the phone.
So I might have to srcnat this IP pool into phone’s Remote IP? Or IPsec shall simply take care of that?
The essence of the setup outlined in the post I’ve linked is “send the response packets of a connection via the same interface through which the initial request of that connection has arrived to you”, and it doesn’t matter much whether that interface is a real WAN or a tunnel. So at the router to which the server is connected, you don’t need dedicated routes to particular destinations.
When Mangle marks the connection, does that mean for MikroTik, that it will have to use the “incoming interface of the packet” when sending back the response?
It’s just one of the components of the complete setup. You also need another mangle rule, assigning a routing-mark (in fact, a routing table name) to packets sent from LAN side depending on the connection-mark value, and the routing table itself, typically consisting of just a single default route via the gateway reachable via that interface.
So you assign the connection-mark as the initial client->server packet arrives, making the connection context “remember” the interface to use, and you use the connection-mark when sending the server->client packets, translating it to a routing table name.
Ok, I got this, thank you.
Please consider this other scenario:
Phone ~~~ VPN ~~~ TIK0 ~~~ TUNNEL ~~~ TIK1 ~~~ WAN
Let’s say phone first joins to TIK0 via VPN, and TIK0’s mangle rules marks some connections and packets, to routing them (instead TIK0’s default gateway) towards the TIK1 through the TUNNEL, letting my phone browsing the net if it was part of TIK1’s LAN.
I can do mangle-rules in TIK1 for marking the connections (coming from TUNNEL), but as they have to go towards the default gateway (of TIK1), it feels unnecessary (or I don’t really know how to do it properly).
Here (https://wiki.mikrotik.com/wiki/Manual:IP/Firewall/Mangle) it’s written:
The mangle marks exist only within the router, they are not transmitted across the network
And indeed, if the packet leaves TIK1, and then later a response for this packet arrives back, shall those reply packets contain my original marks? If so - I can do a route towards TUNNEL if has my connection-mark, and originating from WAN. (Actually, not exactly, as routes can’t have source addresses or interfaces, so I’d need another mangle in TIK1: if originating from WAN, and has my earlier connection-mark, replace the connection-mark to something else, and for this something else connection-mark, I can add my static route towards TUNNEL)
But it seems I can’t solve this.
The statement regarding the mangle marks being virtual and not being added to the actual packet data is absolutely correct, but you miss some information.
The stateful firewall is built around a key component called connection tracker (conntrack module of Linux netfilter). This component maintains a database of “connections”, which are TCP or SCTP sessions, UDP flows, ICMP echo/echo reply conversations, GRE or IPIP flows…
Let’s take an example of UDP, as it requires a more complex handling than GRE or IPIP but a less complex one than TCP or SCTP. When a UDP packet arrives, the conntrack checks whether a UDP connection with the same source and destination address and port exists in the database. If it does, it determines the direction of the packet within that connection: if the source of the packet matches the one stored in the connection database, it is the “originating” direction, if the destination of the packet matches the source one stored in the connection database, it is the “reply” direction. In fact, it is a bit more complex as there is also src-nat and dst-nat, so four address:port tuples in total are stored per connection.
If no matching connnection is found, it gets created.
The conntrack module automatically attaches connection-state and connection-nat-state attributes to each packet, so that subsequent stages of firewall processing could use them to treat the packet specifically, depending on these attributes. Also the connection-mark attribute may get attached to the packet if the one being currently processed or some previous packet belonging to the same connection has hit an action=mark-connection rule, because the connection-mark value is stored as yet another attribute of the connection in the database.
So the fact that the response packet arriving from the server to TIK1 belongs to a particular tracked connection is determined from its source and destination address and port; unless you actively rewrite the connection-mark when processing that packet, the pre-existing value of the connection-mark attribute is retrieved from the database.
If you look at the rules in the linked post, you’ll see that packets with connection-state value established, which means almost every mid-connection packet, are accepted in mangle/prerouting before they can reach any of the mark-connection rules. So as a result, the connection-mark is assigned only when processing the initial packet of the connection and remains unchanged until the connection ends.
Sindy, thank you so much for summarizing this for me, I can’t be thankful enough.
Although, I have the intention that I won’t need mangles: as for me it seems they are extremely helpful if (for example) you need to direct specific packets from the same subnet to not go towards default route(s).
However in my case, I have different subnets and different interfaces, so Tik anyway will be able find the reply-path, as if some packets arrived through the tunnel (on distinct interface and on distinct address), reply-packets will go back through that interface and address.
I’ve validated this, my packets are going back and forth, without adding any mangle-rules into TIK1.
I’m however still stuck at one point, I guess this will be the last, because everything else is working now:
In TIK0:
I’m logging “RX”: packets received with dst-address of “whatismyip.com” (RX from client)
I’m logging “TX”: packets received from src-address of “whatismyip.com” (TX to client)
In this scenario (Scenario1):
Phone ~~~ VPN ~~~ TIK0 ~~~ WAN
RX forward: in:wan out:wan src-mac aa:bb:cc:dd:ee:ff, proto TCP (SYN), PHONE_DYNAMIC_ADR:10000->WHATSMYIP:443, len 60
TX forward: in:wan out:wan src-mac aa:bb:cc:dd:ee:ff, proto TCP (SYN,ACK), WHATSMYIP:443->PHONE_DYNAMIC_ADR:10000, NAT WHATSMYIP:443->(WANIP_OF_TIK0:10000->PHONE_DYNAMIC_ADR:10000), len 52
This works as intended! Only little detail here is that I have no idea on why “RX” shows “wan” for “in” interface if that packet is clearly coming from the phone over IKE2 VPN.
However in this scenario (Scenario2):
Phone ~~~ VPN ~~~ TIK0 ~~~ TUNNEL ~~~ TIK1 ~~~ WAN
RX forward: in:wan out:tunnel, src-mac aa:bb:cc:dd:ee:ff, proto TCP (SYN), PHONE_DYNAMIC_ADR:10000->WHATSMYIP:443, len 60
TX forward: in:tunnel out:wan, src-mac aa:bb:cc:dd:ee:ff, proto TCP (SYN,ACK), WHATSMYIP:443->PHONE_DYNAMIC_ADR:10000, NAT WHATSMYIP:443->(TUNNEL_ADDRESS_ENDPOINT_IN_TIK0:10000->PHONE_DYNAMIC_ADR:10000), len 52
My phone is constantly retrying, it seems no TX packets are arriving back to it, which is strange as it seems the only difference is in NAT (WANIP_OF_TIK0 vs TUNNEL_ADDRESS_ENDPOINT_IN_TIK0).
Here are my NAT-rules:
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface=wan //by default for outgoing packets, this makes first scenario working
add action=src-nat chain=srcnat comment="conf: for tunneled connections" connection-mark=please-go-through-tunnel to-addresses=TUNNEL_ADDRESS_ENDPOINT_IN_TIK0 //I've created this for 2nd scenario working
Strange facts in order:
- Scenario2 is not working
- If I add “ipsec-policy=out,none” to my src-nat action, there are no “TX” logs printed from that point on.
Note: I have tried with masquerading instead of src-nat, but as that’s just a special srcnat, not surprisingly, it didn’t work.
Could you please help me with this NAT-thing, I think this is the final piece missing.
Thank you!
Well - you’ve said before you wanted something “future-proof”, i.e. something that would work even if you change the IP addresses used at the remote end of the tunnel. So the use of connection tracking to clip a connection to the in-interface of the initial packet addresses exactly this request. Of course, where plain routing based on destination addresses is enough, policy routing (mangle rules & multiple routinig tables) are clearly an overkill. So if you use src-nat on TIK0 also when sending packets to TIK1 via the tunnel, whatever actual address the phone has, its packets are seen as coming from TIK0’s address inside the tunnel, so a route to TIK0’s tunnel address at TIK1 is sufficient for everything to work.
Because the payload packet extracted from the IPsec transport one inherits the in-interface attribute from the transport one. In vanilla IPsec as implemented by Mikrotik, there is no Virtual Tunnel Interface for the IPsec payload. So to distinguish packets that came as payload ones of an IPsec SA from those that didn’t, the ipsec-policy matcher is used - in,ipsec matches on packets that came within an SA, whereas in,none matches on packets that came outside an SA, like e.g. the IPsec transport packets.
The behavious is confusing, and so is the description. The second nat rule refers to some connection-mark but I don’t know where that connection-mark is assigned. But the fact that adding ipsec-policy=out,none to the rule changes the behaviour indicates that the connection-mark is asigned indeed.
Since you can see the response packet to arrive from TIK1 via the tunnel, it means everything is fine all the way from TIK0 via TIK1 to whatsmyip and back to TIK0. In what you have posted, I can see no reason why the response packet should not make it back to the phone. But as the ipsec-policy=out,none in the srcnat rule has an impact, I assume that the request packet matches to some other IPsec policy. So if the srcnat rule changes its source address, this policy doesn’t match it later on, the packet is really sent via the tunnel with the IP address of the local tunnel interface, and the response comes back via the tunnel. As the response gets un-src-nated, it starts inverse-matching that policy again, and because packets that should have come via an SA but didn’t are silently dropped, it is not delivered to the phone. But that’s a speculation, I have never tested this. When you exempt the packet from being srcnated by adding the ipsec-policy=out,none to the srcnat rule, it doesn’t get srcnated, so although the filter rule log still says that the packet has been routed via the tunnel, in fact the policy matching supersedes the routing decision, so the packet is sent down the SA associated to that other policy, and gets lost somewhere on the remote peer, so no response comes back.
Sorry for being unclear about “future-proof” I meant for that only for the IKE2VPN in TIK0. What you wrote here is completely clear and true, thank you so much.
This was my idea too, thank you very much for confirming!
Sorry for made this confusing: both NAT rules shown here are in TIK0 (in both scenarios as-is, no changes in NAT rules, they are constant in both scenarios). Also, in both scenarios I’m showing the RX and TX in TIK0.
Actually there are only 2 differences between those 2 scenarios:
- In Scenario1 Phone browses the internet as if it was part of TIK0’s LAN, whereas in Scenario2 Phone browses the internet as if it was part of TIK1’s LAN.
- Scenario1 is working, whereas Scenario2 is not.
“please-go-through-tunnel” mangle is in TIK0, based on your earlier topic suggestion (here I felt those mangles are not needed to share in detail). It really just marks the connection when it needs to go towards TIK1, and also does the routing marks accordingly. I just tried using something as descriptive as possible, and this is what I found out: all packets belong to “please-go-through-tunnel” in TIK0 will use TIK1 as gateway instead of TIK0’s default gateway.
I’m so happy for at least not being alone with this 
This can really be a possible case! This would be a triple20 in darts Sindy: as the current request packet is coming from IKE2 IPsec VPN from the phone, it is actually part of another IPsec policy. OH! And if this is true, then the “masquerade” NAT-rule is also not applied to the request coming from the phone in Scenario1! So in Scenario1, there is no NAT-rule based masquerading or src-natting at all. I shouldn’t do that in Scenario2 either, but then I’ll definitely have to use either Mangles or other basic routing in TIK1.
Is srcnat-ing IPsec connections, even possible at all?
I’ve remove the “src-nat” from TIK0 (to not hurt any IPsec policies), and I added a new route “192.168.200.0/24” in TIK1 (to have the response packets chance to come back, i.e. 192.168.200.0/24 is the “Dst. Address” of the IPsec Policy that handles the Phone in TIK0):
Phone ---> IKE2_IPSEC ---> TIK0 ---> IKE2_IPSEC ---> TIK1 ---> MYIP
|
v
Phone ---- IKE2_IPSEC x--- TIK0 <--- IKE2_IPSEC <--- TIK1 <--- MYIP
Checking my earlier RX/TX log in TIK0 for this, I can see the packets are both receiving and transmitting:
RX forward: in:wan out:tunnel, src-mac aa:bb:cc:dd:ee:ff, proto TCP (SYN), PHONE_DYNAMIC_ADR:10000->WHATSMYIP:443, len 60
TX forward: in:tunnel out:wan, src-mac aa:bb:cc:dd:ee:ff, proto TCP (SYN,ACK), WHATSMYIP:443->PHONE_DYNAMIC_ADR:10000, len 52
I.e. here is no NAT involved in TIK0. Nothing touches the IPsec packets’ addresses, however in TIK1, it obviously must do the masquerading when forwarding the original request to MYIP. From the view of TIK0, everything feels OK, but my phone is still not receive one single response over this complex multi-IPsec tunnel setup.
Do you maybe have any idea on what would worth checking, why this response packet is not able to fulfill IPsec’s requirements?
Thank you so much!
Show me the complete anonymized configuration from TIK0. As adding the route to phone’s address at TIK1 was enough to make the responses reach TIK0, the idea with another IPsec policy at TIK0 is clearly not the answer, so it must be some misconfiguration of the firewall at TIK0.
Okay, what infos do you need exactly? Firewall only? Or Firewall+IPsec? Or Firewall+Route+IPsec?
Thank you!