i’m seeing a very bizarre performance issue on my mikrotik hEX PoE. running latest firmware 6.45.5. (issue happened before this firmware too)
environment:
simple ipsec vpn between 2 network. i’m going to spare the details until proven needed.
the traffic flows, but under normal condition, the connection init is super slow (~10 seconds to open ssh channel, then lags even typing the command)
here’s an example:
erick@Erick$ time ssh 10.10.3.163 cal
real 0m15.421s
user 0m0.016s
sys 0m0.016s
Here comes the odd part. if i turn on packet sniffer to watch 10.10.3.163 packets, everything end up fast!
real 0m0.511s
user 0m0.016s
sys 0m0.016s
Can someone help me understand what’s going on? what would change with the packet sniffer ON ?
does it bypass firewall? does it bypass NAT?
i know if i do notrack between the 2 network, it’d be fast. but i do need to netmap the subnet, so i can’t do that with this scenario.
i have fasttrack on, but i suspect these packets don’t hit it?
help?
bingo! when i disable fasttrack, the connection is fast again.
i’m assuming this is the packet flow you’re talking about? https://wiki.mikrotik.com/wiki/Manual:Packet_Flow
it doesn’t quite mention fasttrack, and my understanding of fasttrack is limited.
as far as i understand, fasttrack is a tcp header tag that skip certain feature, right? it allows SNAT/DNAT. does that include snat nmap and dnat nmap?
my tcpdump on the server show that the return packets were timing out and had to be resent. so i can start there.
so i’m doing packet sniffer again, here’s a snapshot of the first packets.
i highlighted the reverse nmap from 192.168.5.128 to 192.168.6.128.
this is done automatically by mtik i assume on bridge dst-nat.
is this the one that would get missed if fasttrack was enabled?
i can’t quite tell what it would be because packet sniffer turn off fasttrack, so how does one debug fasttrack?
fasttrack is a certain feature of connection tracked firewall … when conditions are met and connection gets fasttrack mark, then (majority of) packets belonging to such connection skip firewall processing altogether. And mangling and IPsec policy matching and … Only one of many packets gets treated in the usual way by usual rules. If there’s a configuration error at fasttrack rule making that one a tad too greedy, most packets of such connections get mistreated, but some of them arrive at destination … with TCP this means a lot of retransmissions for a sluggish connection.
I can not comment on packet sniffer results, to do that I’d have to see complete /ip firewall configuration.
this is exactly what happen if i turn on fasttrack.
fasttrack has mostly default rules that user can’t change, right?
how can i diagnose/troubleshoot fasttrack? i even tried to do a forwarding accept before fasttrack rule, no avail.
Any firewall filter rule can be changed, including the default ones.
As to rule trouble shooting … I don’t know any explicit way of doing it … other than to thoughtfully consider how a particular packet is processed by consecutive rules and why things go differently than expected. wireshark is your friend, but in this case you’d have to capture packets on the “outer” side of router … to check what kind of packets exit when things are going wrong. And obviously packet capture won’t help here.
A hint: firewall filter rule order matters. It is essential that you have a rule that accepts packets which should not be fasttracked and this rule has to be above the (general) fasttrack rule. (The other possibility being overloading fasttrack rule with all the exceptions whish is sometimes hard or impossible to do.)
At the same time you have to be aware that when a connection gets fasttracked, it can not be un-fasttracked. Hence whenever you change some rules which might affect fasttracking of a connection, you have to test it by initiating a completely new connection.
You may need to reboot the router ftom time to time to clear connection tracking state just to be sure it’s not way off.
i finally have time to go back to this issue. here’s a snippet of one of my most simplistic mtik setup.
/ip firewall filter
add action=accept chain=forward comment=“defconf: accept in ipsec policy” ipsec-policy=in,ipsec log=yes log-prefix=ipsecIN
add action=accept chain=forward comment=“defconf: accept out ipsec policy” ipsec-policy=out,ipsec log=yes log-prefix=ipsecOUT
add action=fasttrack-connection chain=forward comment=“defconf: fasttrack” connection-state=established,related
add action=accept chain=forward comment=“defconf: accept established,related” connection-state=established,related
so i moved the ipsec in/out portion of the forward on top. these are the ones that i don’t want fasttracked.
even after i reboot, i still get missing packets, sign of fasttrack. when i disable fasttrack rule, it works flawlessly.
