I’m trying to set up a IPSec connection between AWS (10.5.0.0/16, test machine at 10.5.1.14) and my company network (10.0.10.0/24, test machine at 10.0.10.2, router is at 10.0.10.254). I’ve set up everything and routes are propagated correctly.
What I’m seeing: from 10.0.10.2 I’m pinging 10.5.1.14. On 10.5.1.14: packet is received (icmp echo request), response is sent (icmp echo reply).
On my router, the icmp reply is received in the prerouting chain:
in:ether1-gateway out:(none), src-mac ::**, proto ICMP (type 8, code 0), 10.5.1.14->10.0.10.2, len 84
Then, I can see it in the dst-nat chain (action=accept, no rewriting is done). After that, I cannot find the packet anymore. I’ve put passthrough (with src=10.5.0.0/16) rules on my fw input, fw forward and postrouting chain, but it never shows up anymore.
Just right now, while logging both the prerouting and mangle input chain, saw these two lines:
prerouting in:ether1-gateway out:(none), src-mac e4:48:ab:ab:ab:ab, proto ICMP (type 8, code 0), 10.5.1.14->10.0.10.2, len 84
mangle inp input: in:ether1-gateway out:(none), src-mac e4:48:ab:ab:ab:ab, proto ICMP (type 8, code 0), [aws wan]->[my wan ip], len 84
So the src-mac addresses are the same, but the ip addresses are translated to wan ips. Wouldn’t expect that, as I have a accept rule for traffic from 10.5.0.0/16 in my /ip firewall nat.
But these are not your IPSec peers according to “/ip ipsec peer” export.
As for mangle and nat, you can remove mangle rules entirely, and you can remove 2nd nat rule.
Regarding the packets you notice, those are not a problem, those are just the Amazon VPN gateways checking if you are alive with ICMP (ping).
These packets are fine and should be like that.
IPsec is working (but only one tunnel, the .111 one), AWS is reporting that the tunnel is up (and routes to 10.0.10.0/24 are advertised via BGP). The addresses being incorrect and missing incoming ipsec firewall rule was a copy-paste mistake on my end. Sorry for the confusion.
I can also confirm traffic is arriving at 10.5.1.14 when pinging from 10.0.10.2. Replies from 10.5.1.14 arrive at the router again, but never find their way to 10.0.10.2.
The policies are correct afaik. AWS uses a double-endpoint for their VPN connections (in different availability zones), so there’s a redundant connection between the mikrotik and AWS. There’s a limitation in RouterOS (last paragraph) which prevents ipsec policies with the same src- and dst-addresses from being active at the same time (but their sa-dst-address is different). I guess this isn’t a problem though, at least one connection should be up.
That is why I said it would not work with the previously posted config.
There must be other config that prevents this, firewall and NAT and mangle are OK from what you posted.
This is exactly what I meant. 2nd peer will never come up, as the 2nd policy will never be used.
If you are aware of this, then ok. Just know that this is not failover, nor redundancy.
Simply, the 2nd policy will NEVER be used, and therefore the 2nd peer will NEVER be used.
I removed the superfluous rules from my nat/filter chains, still no luck though.
There must be other config that prevents this, firewall and NAT and mangle are OK from what you posted.
And they should be in either the /ip firewall mangle or filter, right? I have no rules in those chains that explicitly drop this traffic, and if I turn on logging for my drop rules, I do not see the dropped packets either. If I add rules such as chain=input|forward|output src=10.5.0.0/16 action=accept (and put them first in the list), it doesn’t catch any traffic. The counters just stay 0. I would expect, based on the packet flow diagram, packets to be either in the input or forward chain (second diagram, enter at I, after routing decision packets should be in input or forward).
So what could be happening is that there is some sort of masq/…? going on that rewrites traffic coming from 10.5.0.0/16 to a wan ip, and therefor the counters do not increase. Maybe I can accept/log by mac address to see if that triggers.
P.s.: as for the second peer/policy: you’re totally right. I’ve just added these so when Mikrotik decides to fix this, I can just enable them in my config.
routing engine dropping packets because of no route or a blackhole route
packets are arriving with TTL of 1, therefore are being dropped
I also advise sniffing the traffic (there is an action in prerouting that can do that for you) and looking at the packets directly in wireshark to analyze more.
/ip route
add distance=15 gateway=[next hop wan 2] routing-mark=to-xs4all
add check-gateway=ping distance=5 gateway=[next hop wan 1]
add check-gateway=ping distance=10 gateway=[next hop wan 2]
There is also a dynamicly generated route for 10.0.10.0/24 (pref source 10.0.10.254 with gateway bridge-vlan10). I sniffed the traffic in Wireshark, TTL is 255 for the incoming packets.
After a short screen sharing session with tomaskir, it turned out AWS relies on IPSec VTI being available when using your VPN connection with BGP instead of static routes (I hope I’m saying this correctly ).
After I reconfigured the VPN connection to use static routes, everything started working as expected. I just needed to add a simple rule that allows input traffic with protocol 50 from AWS IPs.
Issue resolved, hopefully this helps somebody in the future who’s seeing the same issues as I did.
> ab:ab, proto ICMP (type 8, code 0), 10.5.1.14->10.0.10.2, len 84