I find wireshark captures helpful to debugging some fundamental protocol misunderstanding (it's also very educational), since they can be straightforwardly mapped into actual RFCs. Logs are, after all, results of Mikrotik's processing and may hide the issue.
Wireshark is perfect to tell you
what has happened, provided that you are able to decrypt the protocol messages. Phase 1 becomes encrypted quite soon. So to see the actual contents of these packets in plaintext, you may still need the log which dumps the Phase 1 packet contents in decrypted form. There is a topic here on the forum on how to read these using Wireshark. Not sure it works for both directions of Phase 1, though.
Log is the best source to learn
why something happened, i.e. why the side generating the log has taken a particular decision.
However, I don't know exactly how I have to setup the "action=none" policies, to avoid ordinary traffic ending up in the tunnel.
If I understand @sindy correctly, just add more specific policies (i.e. specific source addr. and/or destination addr. and/or protocols) and the same group and proposal, but set the action to none (perhaps you also need to set level to "use"?). Note that the list of policies is an an ordered one, so put them before "0.0.0.0/0 <=> 0.0.0.0/0". The net result should be:
1. During the negotiation "0.0.0.0/0 <=> 0.0.0.0/0" will be found which will let phase 2 to complete and establish communication SAs
2. The none policies will, hopefully, prevent RouterOS from routing through IPsec
Yes, that's the idea. There is no need for a proposal or a peer in
action=none policies.
Example - you want that only packets from a local subnet 10.0.0.0/8 to a remote subnet 11.0.0.0/8 reach the 0.0.0.0/0 <=> 0.0.0.0/0 policy. This is intentionally simplified to keep the number of policies low for the purpose of the example.
So you have to use
action=none policies with the following traffic selectors:
src-address=128.0.0.0/1 dst-address=0.0.0.0/0
src-address=64.0.0.0/2 dst-address=0.0.0.0/0
src-address=32.0.0.0/3 dst-address=0.0.0.0/0
src-address=16.0.0.0/4 dst-address=0.0.0.0/0
src-address=0.0.0.0/5 dst-address=0.0.0.0/0
src-address=12.0.0.0/6 dst-address=0.0.0.0/0
src-address=8.0.0.0/7 dst-address=0.0.0.0/0
src-address=11.0.0.0/8 dst-address=0.0.0.0/0
src-address=0.0.0.0/0 dst-address=128.0.0.0/1
src-address=0.0.0.0/0 dst-address=64.0.0.0/2
src-address=0.0.0.0/0 dst-address=32.0.0.0/3
src-address=0.0.0.0/0 dst-address=16.0.0.0/4
src-address=0.0.0.0/0 dst-address=0.0.0.0/5
src-address=0.0.0.0/0 dst-address=12.0.0.0/6
src-address=0.0.0.0/0 dst-address=8.0.0.0/7
src-address=0.0.0.0/0 dst-address=10.0.0.0/8
The blue ones catch packets from any source except 10.0.0.0/8 to any destination; the green ones catch packets to any destination except 11.0.0.0/8.
So for a /8 source or destination network, you need 8 policies; for a /24 one, you need 24 policies. And if there are more subnets to let pass, it becomes a nightmare that can only be generated using a script.
Worse than that, in some cases Mikrotik acting as a responder rejects 0.0.0.0/0 at destination side as a "safety measure". I'm not sure whether it is the case also for a statically configured policy or only for ones generated from a template.
Note that on RouterOS you can control even further what can go through IPsec via ipsec-policy parameter of a filter rule. I have not tried it, but looks like it allows to describe very complex setups by the means of marks.
Unfortunately, it's the other way round. The
ipsec=in|out,ipsec|none match conditions only allow the firewall rules to treat received packets depending on whether they came in via an IPsec policy or not, and packets being sent or forwarded out depending on whether they will match a traffic selector of an IPsec policy if they ever reach it. So basically you can use
in-interface-list=WAN ipsec-policy=in,ipsec to match only packets that came in IPsec-encrypted via WAN (e.g. to accept them for management access) or
ipsec-policy=out,ipsec to match packets that will get encrypted, e.g. to exempt them from being src-nated (which would prevent them from matching the traffic selector and thus get encrypted).
In summary - in RouterOS there is no way to protect a packet on its way to be sent out from being diverted by a policy except two things - another policy or src-nat.