Hello,
I’ve just replaced my main router from pfSense to a RB5009 with RouterOS 7.16.1 but I have a weird behaviour with an IPSEC Site2Site with a Chateau 4G with RouterOS 7.16.1
With default configuration, VPN it’s really sluggish: ping is ok, HTTPS is poor, RDP doesn’t work at all.
At the moment it’s working bypassing Fasttrack with following rules:
No mangling involved because IPSEC rules are above FastTrack one, but I read that for performance it’s better to keep frequently used rules at the top and less used at the bottom in order to match earlier and don’t check all rules. In that case FastTrack should be almost at the top.
Do you think that your two mangle rules have zero costs ? They will be checked against every packet (before they are fasttracked) in the forward chain too, and before the fasttrack rule is reached (3B and 3C steps):
Not only that, they make modifications to ipsec connections (by setting the mark). That’s memory write operations. The two rules from defconf are read-only (apart from increasing the counters). Once a connection is fasttracked, most of the packets will bypass the firewall and are not affected by the two defconf filter rules.
Additionally, your fasttrack rule now has an extra check to make (compare the connection mark).
In conclusion, you have spared no rules, added more memory writes, and added more checks.
The most processing it’s possible to bypass/skip is to fasttrack packets, as already explained further packets belonging to fasttracked connections skip lots of processing. So no need to bother with connection-state for this reason. However, packets, which should be processed in a special way (tunneled through IPsec tunnel, mangled, etc.), should not be fasttracked. And the most “kosher” way of doing it is to accept them by filter rules, placed above fasttrack rule. Another possible way is to make exclusion “matchers” on fasttrack rule, but this option doesn’t scale.
And then what @GabrieleV wrote: less firewall rules (filter, raw, whatever) means less processing … because some CPU cycles are spent on every rule even if it doesn’t execute. So the fastest rules are the non-existing ones.
IPsec traffic gets looped around the boxes. Here’s general packet flow diagram:
When a packet is about to be sent through IPsec tunnel, it’ll first pass points A and B (bridging decision), then points I and L (routing decision, forward path … during which usual firewall rules for forward will be evaluated) … and then it’ll proceed to the “ENCAPSULATE?” parallelogram where it’ll take “YES” branch … after encapsulating in IPsec tunnel wrapper, it’ll pass points K and L (during which it’ll pass firewall rules again, but this time they should be handled as IPsec payload, hence not fasttracked) … and after bypassing points C and D it’ll get ejected via physical out-interface (unless your WAN is PPPoE tunnel in which case packet will pass the lower portion of diagram again, this time being encapsulated into PPPoE envelope, which is not IPv5 or IPv6 content and will hence bypass points K and L).
Similarly ingress IPsec packet will loop over the upper part of the diagram: after passing input firewall filters as part of IPsec tunnel it’ll get decapsulated and will return to top left part where it’ll be again processed as a “usual” IPv4/IPv6 packet …
You’re saying IPsec traffic goes through the forward rules both as “IPsec policed” and not? (order depending on direction) Your explanation makes sense, but how does this manifest itself? I added a log rule like so at the top of my forward chain at site A:
Packets with encrypted payload, belonging to IPsec tunnel (don’t know how to describe them better) will use chains input and output … Establishment of IPsec tunnel between two peers will usually start by sending packets (and thus using chain=output, default config allows all) and then return packets are handled by chain=input connection-state=related. This may not work at the first moment before local IPsec process starts to establish connection with peer … which means that some peer’s packets (which may have tried to establish tunnel before) get dropped by chain=input (because packets from peer don’t seem to be “related” to any of connections). But as soon as local IPsec process sends out first tunnel establishment packet, the return packets will start to be accepted.
Packets with “normal payload”, which are (due to IPsec policy, either static or dynamic) marked to pass through IPsec tunnel, will receive “ipsec-policy” state. And firewall rules can use it as additional matcher. Default rules (placed above fasttrack rules) use those properties to accept those packets before fasttrack rule could “taint” them (as you noticed. fasttracked packets don’t pass IPsec tunnel). If there was no fasttrack rule, then those special rules with ipsec-policy set wouldn’t be necessary, if none of other rules would drop them, they would be accepted by implicit ultimate “accept” rule. If you want to filter some traffic passing through IPsec tunnel and still want to keep the fasttrack rule, then you’d have to implement needed “action=drop ipsec-policy=,ipsec” rules and have them placed above the default “accept ipsec-policy=,ipsec” packets. Or you could implement “more traditionally looking” drop rules which would match against IP address (src- or dst-) but again placing them above the “accept ipsec” rules.
So in short: property ipsec-policy is optional property, just like e.g. connection-nat-state property or connection-state property (the former is set if packet is subjected to any kind of NAT and the later is set if packet is not exempt from connection tracking). And this property is set to packets which are otherwise payload of IPsec tunnel (either direction).
That’s not correct because default rules don’t have “connection state new only” so they will always match before FastTrack because they match established sessions as well.
Reading official Mikrotik documentation, they suggest to use RAW to bypass connection tracking:
FastTrack packets > bypass firewall, connection tracking> , simple queues, queue tree with parent=global, ip traffic-flow, IP accounting, IPSec, hotspot universal client, VRF assignment, so it is up to the administrator to make sure FastTrack does not interfere with other configuration!
Only occasionally, some packets will still be sent through the whole firewall chains:
Currently, only IPv4 TCP and UDP connections can be fast-tracked and to maintain connection tracking entries > some random packets > will still be sent to a slow path. This must be taken into consideration when designing firewalls with enabled “fasttrack”.
Which means, as I wrote above, for the majority of the packets of a fasttracked connection (from the 3rd packet on), the two defconf filter rules will have no effects and are completely ignored. No processing is spent on the firewall tables for those packets. Only a small part of the connection’s packets will occasionally travel the slow way and hit the two filter rules. But if you had the two mangle rules instead of the two filter rules, the mangle rules will need to process those packets too. And they “costs” more as I wrote in my previous post.
In short: after connection tracking machinery determines, that packet belongs to fasttracked connection, it bypasses almost all other packet processing steps (including prerouting, firewall filter, etc.). Meaning that those packets don’t (or very rarely) even enter raw firewall and filter firewall. And hence the rules set there don’t have chance to act on those packets.
Normally fasttracked packets are only “established and related”, which means that according to this rule, packets opening new connections (or are invalid) are not fasttracked, hence they hit the normal firewall filter processing. The fasttrack rule is then only hit by packets which belong to yet not-fasttracked connections … and if connection is elligible for fasttracking, only very few packets belonging to it will go through normal firewall filtering (for TCP that’s initial 3 packets of the 3-way handshake - at which point connection gets “established” status - and additional one which is the first one belonging to this connection after connection state transitioned to established and thus trigger fasttrack rule). And if fasttrack rule is wide enough, then it’ll handle almost all packets with status establishe or related. Remaining are a few packets of “new” connections, the “invalid” packets which should be dropped as soon as possible. And then the remaining are “new” which should be subject to more or less specific accept/drop rules.
The exception are connections which must not be fasttracked for some reason. And that’s why one needs firewall filter rules above fasttrack rule … so that packets belonging to those connections would not trigger fasttrack rule. One example is IPsec-bound packets, another example is packets which should get mangled in any way (e.g. queuing using other than hardware queues).
Sometimes it’s easiest to simply disable fasttrack rule … one looses (lots of) performance, but with fasttrack and it’s gotchas removed from the picture, things may start to behave as expected.
So if I’ve understood correctly, the FastTrack firewall rule is not an “accept FastTracked connections” but something more like “accept that this new connections will be Fasttracked”, is it correct?
In that case, everything make sense, sorry for the misunderstanding.
Correct. With a side note: it’s not “new connections”, it’s “established,related” … and that’s default. According to how rule works and how fasttrack works it’s obvious that only packets not fasttracked yet[*] will pass by this rule.
One could set matcher of fasttrack rule to “connection-state=new” … and likely trigger havoc in firewall. Rule would still be syntactically correct though.
If I understood correctly, the rule with action=fasttrack-connection is actually a pass-through rule, so even packet that does go via “slow track” and does trigger fasttrack rule, doesn’t get accepted by that rule … but rather by another rule below fasttrack (default setup has an accept rule right after fasttrack and which accepts all packets which are supposed to be fasttracked but somehow are not (yet) … plus untracked packets).
[*] it was mentioned that small portion of packets, belonging to connections marked for fasttracking, still pass “slow path” … which makes the “normal” “accept established,related” rule necessary. And this also explains why connections, which should not be fasttracked but erroneously are, still somehow work, but painfully slowly.
? They will be checked against every packet (before they are fasttracked) in the forward chain too, and before the fasttrack rule is reached (3B and 3C steps):
