FastTrack Causing Slow HTTPS Loads

RouterOS General
1

I have been playing with WireGuard a lot as of late and am a bit confused as to exactly how FastTrack rules are considered. Details regarding my setup are here. Additionally, I have implemented MSS clamping as described here.

This leaves me with a functional connection, except that TCP connections seem to take a very long time to initialize and begin transferring data. This can be seen in Firefox’s network monitor as a very long wait period (5 - 50 seconds) before receiving data. That said, once data reception starts everything is snappy and usually downloads in a few milliseconds.

I was able to solve this by disabling the FastTrack rule in the forward chain. What has me confused is why this wasn’t working in the first place.

Consider the following firewall rules:

[admin@MikroTik] > /ip/firewall/filter/print
Flags: X - disabled, I - invalid; D - dynamic 
 0  D ;;; special dummy rule to show fasttrack counters
      chain=forward action=passthrough 

 1    ;;; FastTrack
      chain=forward action=fasttrack-connection hw-offload=no connection-state=established,related 

 2    ;;; Established, Related
      chain=forward action=accept connection-state=established,related 

      <truncated for brevity>
      
[admin@MikroTik] > /ip/firewall/mangle/print
Flags: X - disabled, I - invalid; D - dynamic 
 0  D ;;; special dummy rule to show fasttrack counters
      chain=prerouting action=passthrough 

 1  D ;;; special dummy rule to show fasttrack counters
      chain=forward action=passthrough 

 2  D ;;; special dummy rule to show fasttrack counters
      chain=postrouting action=passthrough 

 3    chain=prerouting action=mark-connection new-connection-mark=pia_wireguard_conn src-address=192.168.0.0/16 dst-address=!192.168.0.0/16 connection-mark=no-mark 

 4    chain=prerouting action=mark-routing new-routing-mark=routes-pia src-address=192.168.0.0/16 connection-mark=pia_wireguard_conn 

 5    chain=forward action=change-mss new-mss=clamp-to-pmtu passthrough=yes tcp-flags=syn protocol=tcp routing-mark=routes-pia

As can be seen, I have two mangle rules to mark VPN traffic and another mangle rule to clamp the MSS. Additionally, I have a general FastTrack rule in the forward chain. So far, so good.

Now, according to the official documentation (here):

3. The packet enters forward process;
    a. check TTL value;
    b. process packet through Mangle forward chain;
    c. process packet through Filter forward chain;
    d. send the packet to accounting processes;

If I understand this correctly, since the packet passes through the Mangle forward chain before the Filter forward chain, the MSS should be clamped before any FastTracking is considered. Despite this, I have the issues described above.

Now, if I disable the FastTrack rule in the forward chain (via /ip/firewall/filter/set disabled=yes 1), everything works exactly as expected without delay.

In short, why is this? Where is my understanding in this process going off the rails?

Fastpath and Mangle rules incompatibality
2

You’ve kept the action=fasttrack-connection rule from factory default configuration unchanged, so it fasttracks everything. To make it selective, you have to add e.g. connection-mark=no-mark to it, to make it ignore packets belonging to any marked connection.

3

Sindy,

Thanks for the info and that makes sense.

That said, what I am struggling with is why FastTracking is detrimental in the first place. Shouldn’t all of the mangle rules and such be done by the time the packet is FastTracked? At this point, isn’t the wireguard traffic basically the same as any other forward traffic, at least from the point of view of the firewall?

4

The very essence of fasttracking is that most packets belonging to fasttracked connections skip a lot of stages of packet processing - mangle, filter, queues except those parented by interface queues, and IPsec traffic selector matching. So the packet arrives, gets matched against the list of all active connections maintained by the conntrack module of the firewall, and if found to match an existing connection and that connection has the fasttrack flag set, the packet goes straight to routing, bypassing all the stages listed above. NAT processing is done even for fasttracked packets of course.

What may be interesting for you is that on top of mangle rules, a routing table may be chosen also using /ip route rule. The assortment of match conditions is much smaller there: the source interface, the source prefix and the destination prefix, and a routing mark eventually previously assigned by mangle (no address ranges, no address lists, no interface lists, no matching on protocol/port, no matching on connection mark) but these rules are taken into account even for fasttracked packets. So if you want e.g. to route a whole local subnet via a Wireguard tunnel, this may be an efficient way to do that.

If actually the bulk of your traffic should go via Wireguard, you can make Wireguard the default route in routing table main and handle the rest of the traffic by another routing table, which allows you to fasttrack the Wireguard traffic instead of the rest.

5

Oh, I think I see.

So the very first packet for this connection traverses the entire “normal” data path including MSS fixing and eventually hits the FastTrack rule. This doesn’t do anything per se but does add the connection to the FastTrack list. Then, when the next packet is seen its connection identifier is automatically matched against the FastTrack list and the rest of the traffic for this connection bypasses the rest of the processing chain as you describe above.

In short, FastTrack is connection-based rather than packet- or fame-based.

Is this correct?

6

Absolutely correct. There are minor details - the second packet also takes the full path, and every n-th packet of a fasttracked connection also takes the full path (don’t ask me what the n is, though), which explains why it was causing just a slowdown of connections relying on mangle, not breaking them completely.

7

Good to know about the n-th full traversal, that explains a lot.

Also, thanks for the tips on the routing rules. For now, I am running everything through the VPN but this may be handy in the future for “unmodified” subnets which poke out to the internet directly.

Thanks again for the assistance understanding this, it is a big help!

8

If you run everything through VPN, you don’t need multiple routing tables (aka routing-mark values) at all. Just add a dedicated route towards the VPN server address into the main routing table (i.e. no routing-mark), so that the router would not loop the transport packets of the tunnel back to the tunnel, and set the default route’s gateway to the Wireguard interface.

9

For now I am using a second routing table as I have written a special script to handle re-registration periodically required by the VPN vendor. The second routing table allows me to, on successful registration (and receipt of a new default gateway), regenerate the entire table without having to worry about breaking other parts of the device. Specifically, I don’t have to touch any of the non-VPN routes, which I keep available in case I ever have to re-register after the tunnel has gone down.

That said, your approach makes a lot of sense for less convoluted setups that mine. :laughing:

10

I see from your rules that you have a wire guard connection to Private Internet Access VPN, if the PIA naming is not leading somewhere else.

Would you be willing to share your setup? I am a beginner regarding Wireguard, and having something like a template would be invaluable.

11

Apologies for the thread necro, but searching for PIA and wireguard lead me here.

It doesn’t look like dromon ever shared his RouterOS script, unfortunately, and he hasn’t been back to this forum since shortly after his posts here.

I’ll share what I know, since I’ve been tinkering with this and it’s fresh in my mind. If you download https://github.com/pia-foss/manual-connections, and read through connect_to_wireguard_with_token.sh, you’ll see what it’s doing, and that’s essentially:

  1. generate a wireguard private key and public key pair
  2. contact a PIA meta server and give it the public key from above
  3. from the above request, it’ll receive the interface IP and endpoint IP:port and public key for the peer, and generates an /etc/wireguard/pia.conf that looks something like:
[Interface]
Address = 10.7.199.123
PrivateKey = <privKey>

[Peer]
PersistentKeepalive = 25
PublicKey = <pubKey>
AllowedIPs = 0.0.0.0/0
Endpoint = 91.90.120.123:1337

The data above can be translated directly into RouterOS as:

/interface wireguard add mtu=1420 name=wg-pia private-key="<privKey>"
/interface wireguard peers add allowed-address=0.0.0.0/0 endpoint-address=91.90.120.123 endpoint-port=1337 interface=wg-pia persistent-keepalive=25s public-key="<pubKey>"
/ip address add address=10.7.199.123/17 interface=wg-pia

And that’s pretty much the one-to-one equivalent between the wireguard config on unix vs in RouterOS.

Then to make it work for your LAN, you’ll need to NAT it:

/ip firewall nat add action=masquerade chain=srcnat out-interface=wg-pia

Clamp the MSS going out of it:

/ip firewall mangle add action=change-mss chain=postrouting new-mss=clamp-to-pmtu out-interface=wg-pia protocol=tcp tcp-flags=syn

And route traffic out of it. There are lots of ways on doing this, but here’s an example using routing marks to selectively send traffic out of it:

/routing table add fib name=pia
/ip firewall address-list add address=dns.google list=vpn
/ip firewall mangle add action=mark-routing chain=prerouting dst-address-list=vpn new-routing-mark=pia passthrough=no
/ip route add distance=1 dst-address=0.0.0.0/0 gateway=wg-pia pref-src=0.0.0.0 routing-table=pia scope=30 suppress-hw-offload=no target-scope=10

That should send all traffic to dns.google out the PIA wireguard interface.

Now, here’s the part I don’t quite understand: in iperf3 testing, inbound (-R) through the VPN is fine, but outbound is basically 0 kbps, even though I can ping and make outgoing connections. With experimentation, I’ve found that I need to disable ipv4 fasttrack in order for outbound to work correctly. I don’t understand why this is necessary. I’ve set up site-to-site wireguard, road-warrior wireguard, and wireguard in VMs as gateways with similar routing, and all those have worked fine with ipv4 fasttrack enabled. Can someone shed some light on why disabling ipv4 fasttrack is needed for this?

12

http://forum.mikrotik.com/t/fastrack-and-mark-routing/122888/1

13

Thanks for the pointer. The post you linked points towards this: http://forum.mikrotik.com/t/static-default-route-im-missing-something/119183/2

From that, I get that fasttrack-connection fast tracks the connection based on the incoming packet, which causes the outgoing packet to skip mangle processing and thus being marked for the routing mark, correct? Thus, for whatever logic I used to put a routing mark on the packet, I should switch to marking the connection (but only on the first/new packet). Then I should match on that connection mark to apply the routing mark. I can then re-enable the fasttrack-connection rule with the condition that it doesn’t apply to any packet with a connection mark. I get all that, but I have some questions about the example config given:

/ip firewall mangle

  1. add chain=prerouting connection-state=established,related connection-mark=no-mark action=accept # if a mid-connection packet has no connection mark, it needs the default handling
  2. add chain=prerouting connection-state=established,related in-interface=your-wan # download packets MUST NOT be routing-marked
  3. add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A # passthrough=no is a default behaviour but you can state it explicitly
  4. add chain=prerouting connection-mark=handling-B action=mark-routing new-routing-mark=handling-B # same like above

#only initial packets of connections (plus some garbage) get here past the rules above
5. add chain=prerouting …list of classifying match conditions for handling A… connection-state=new action=mark-connection new-connection-mark=handling-A passthrough=yes
6. add chain=prerouting …list of classifying match conditions for handling B… connection-mark=no-mark connection-state=new action=mark-connection new-connection-mark=handling-B passthrough=yes

#initial packets of connections which evaded both the rules above get here with no connection mark; we just repeat the mark-routing rules above
7. add chain=prerouting connection-mark=handling-A action=mark-routing new-routing-mark=handling-A
8. add chain=prerouting connection-mark=handling-B action=mark-routing new-routing-mark=handling-B

Aren’t lines #3-4 the same as lines #7-8? Do they need to be listed before and after the connection mark lines (#5-6)?

Line #2 doesn’t have an action, what’s it supposed to do? Was that supposed to have action=accept? Assuming that’s the case, this line’s purpose seems to be such that incoming packets from the WAN will have connections marks, but this prevents routing marks from being applied to those packets, so that they don’t get routed back out the tunnel?

Line #1 doesn’t really seem necessary, but more of a performance optimization, correct? And would be bad if I needed to mark connections/packets in other ways?

14

They are, yet there is a reason to have them “twice”. You need to save CPU cycles, so rules #3 and #4 handle mid-connection packets, which come already marked with connection-mark, and packets matching these rules do not continue further down the rule chain as passthrough=no is set for these rules. But you need to assign the routing-mark also to the initial packet of each connection, that has just been marked with a connection-mark, hence the same rule appears once again after the action=mark-connection ones.


All correct - the default action for a rule is accept (yet it was my mistake not to put it there explicitly, sorry), and yes, the purpose is to prevent the packets coming from outside from getting a routing-mark and being routed out via WAN again rather than reaching their actual destination in LAN.


Also correct.

In the meantime, I’ve switched to a different structure, where I mark all connections using a dedicated chain:
chain=prerouting connection-mark=no-mark action=jump jump-target=mark-conns
chain=prerouting connection-mark=CM1 in-interface-list=LAN action=mark-routing new-routing-mark=RM1 passthrough=no
…
chain=prerouting connection-mark=CMn in-interface-list=LAN action=mark-routing new-routing-mark=RMn passthrough=no

chain=mark-conns ..some match conditions… action=mark-connection new-connection-mark=CM1 passthrough=yes
chain=mark-conns ..some match conditions… connection-mark=no-mark action=mark-connection new-connection-mark=CM2 passthrough=yes
…
chain=mark-conns connection-mark=no-mark action=mark-connection new-connection-mark=use-main

So an average mid-connection packet still traverses only (N+1)/2 rules where N is the number of routing marks, but there are no duplicate rules so the eventual mistakes are conveniently centralized.

action=jump should actually be named call - unless some rule in the chain provides a definitive verdict (passthrough=no, action=accept etc.), the packet processing returns to the calling chain after passing the last rule in the called one.

15

Ok, I get it now. At first I thought it was a mistake or an unnecessary optmiization, since they appear again after only a few connection mark lines, but if the average connection has say 10000 packets, then 99.99% of them would end processing here instead of after a few more lines.


Ok, I have new questions about your new method then:


  1. chain=prerouting connection-mark=no-mark action=jump jump-target=mark-conns
  2. chain=prerouting connection-mark=CM1 in-interface-list=LAN action=mark-routing new-routing-mark=RM1 passthrough=no
    …
  3. chain=prerouting connection-mark=CMn in-interface-list=LAN action=mark-routing new-routing-mark=RMn passthrough=no
  4. chain=mark-conns ..some match conditions… action=mark-connection new-connection-mark=CM1 passthrough=yes
  5. chain=mark-conns ..some match conditions… connection-mark=no-mark action=mark-connection new-connection-mark=CM2 passthrough=yes
    …
  6. chain=mark-conns connection-mark=no-mark action=mark-connection new-connection-mark=use-main

So the passthrough=yes in the mark-conns chain really applies towards the jump rule where it came from? And if they return to that jump rule when a verdict is reached, then the connection-mark=no-mark shouldn’t be necessary in lines (e)-(f), right? And why is (f) needed at all? With that, you’d need to change the fasttrack-connection rule to only work on packets with the connection mark ‘use-main’, right?

I get that the in-interface-list=LAN in lines (b)-(c) make it so that line 2 in the previous example isn’t necessary anymore, but line 1 is still a useful optimization if you got rid of line (f), isn’t it?

Shouldn’t line (a) have connection-state=new, which should make connection-mark=no-mark unnecessary (as well as line (f))? That way, it only tries to mark connections on new connection packets, and those can’t already have a connection mark at this point?

So, to summarize with your examples, why not this?

chain=prerouting connection-state=established,related connection-mark=no-mark action=accept
chain=prerouting connection-state=new action=jump jump-target=mark-conns

chain=prerouting connection-mark=CM1 in-interface-list=LAN action=mark-routing new-routing-mark=RM1 passthrough=no
...
chain=prerouting connection-mark=CMn in-interface-list=LAN action=mark-routing new-routing-mark=RMn passthrough=no

chain=mark-conns ..some match conditions... action=mark-connection new-connection-mark=CM1 passthrough=yes
...
chain=mark-conns ..some match conditions... action=mark-connection new-connection-mark=CMn passthrough=yes
16

There are many ways to do the same thing :slightly_smiling_face: If you add a connection-mark to every connection, the jump rule can match only on connection-mark and it does not need to match on connection-state; vice versa, if you can handle a failure to assign a connection-mark to some connections (or if it is an intention), it is enough that the jump rule matches on connection-state=new alone. So that also determines whether you need line (f) or not.

But it seems to me you’ve misunderstood one thing - after passing through the last rule of chain prerouting, the packet does not continue to chain mark-conns; the only way for a packet to enter mark-conns is by being sent there from some other chain using jump.

As for the inside of mark-conns, the classifying match conditions may be partially overlapping, so their order may matter, and for such cases, the connection-mark=no-mark is there to avoid rewriting an already assigned connection-mark. You cannot set passthrough=no for these rules, because after passing through mark-conns, the packet has to be further processed in prerouting in order to (eventually) get a routing-mark.

17

No, I assumed that part. What I’m trying to understand is exactly how the jump rule is evaluated. I think my misunderstanding was assuming that the jump rule implicitly continued to the next line after returning from the user-defined chain. For example, with these rules:


  1. chain=prerouting connection-state=established,related connection-mark=no-mark action=accept
  2. chain=prerouting connection-state=new action=jump jump-target=mark-conns
  3. chain=prerouting connection-mark=CM1 in-interface-list=LAN action=mark-routing new-routing-mark=RM1 passthrough=no
  4. chain=prerouting connection-mark=CM2 in-interface-list=LAN action=mark-routing new-routing-mark=RM2 passthrough=no
  5. chain=prerouting connection-mark=CM3 in-interface-list=LAN action=mark-routing new-routing-mark=RM3 passthrough=no
    …
  6. chain=prerouting connection-mark=CMn in-interface-list=LAN action=mark-routing new-routing-mark=RMn passthrough=no
  7. chain=mark-conns ..some match conditions… action=mark-connection new-connection-mark=CM1 passthrough=yes
  8. chain=mark-conns ..some match conditions… action=mark-connection new-connection-mark=CM2 passthrough=yes
  9. chain=mark-conns ..some match conditions… action=mark-connection new-connection-mark=CM3 passthrough=yes
    …
  10. chain=mark-conns ..some match conditions… action=mark-connection new-connection-mark=CMn passthrough=yes

Lets say we have a packet for a new connection, and it would match CM2, what’s the order of processing?

I had thought it would be: 1, 2, 7, 8, 3, 4
Such that once line 8 matches, it would return, and the “passthrough=yes” from line 8 applied to the behavior of the jump action in line 2, thus continuing to lines 3 and 4.

But if I understand what you’re saying, it should be: 1, 2, 7, 8, 9, 10, 3, 4 ?
If so, then I should have a line like:

chain=mark-conns connection-mark=no-mark action=accept

At the end of the mark-conns chain, to shortcut having to evaluate lines 3-6?

18

passthrough=yes changes the behaviour of the rule where it is stated, so in your example, only of rule 8. And yes, the longer path will be taken.



Yes, you can do that, since if no connection-mark is assigned, there is nothing to translate to a routing-mark. Of course, in general you can translate an absence of a connection-mark to some routing-mark as well, but it is not a commonly used approach.

19

Thank you for the confirmation, and explaining it all!

20

Sindy, is there a final summary of latest rules that you apply along with commentary on fast-track setting if any(apart from applying it to only to things with no connection marks.Seems like you have iterated over it..) . i had a separate thread going just for this.Mikrotik ProtonVPN Wireguard Setup