Community discussions

 
esunarto
just joined
Topic Author
Posts: 17
Joined: Wed Jul 18, 2018 11:42 pm

bizarre performance issue with packet sniffer

Fri Sep 06, 2019 5:54 pm

i'm seeing a very bizarre performance issue on my mikrotik hEX PoE. running latest firmware 6.45.5. (issue happened before this firmware too)
environment:
simple ipsec vpn between 2 network. i'm going to spare the details until proven needed.
the traffic flows, but under normal condition, the connection init is super slow (~10 seconds to open ssh channel, then lags even typing the command)
here's an example:
erick@Erick$ time ssh 10.10.3.163 cal
real 0m15.421s
user 0m0.016s
sys 0m0.016s

Here comes the odd part. if i turn on packet sniffer to watch 10.10.3.163 packets, everything end up fast!
real 0m0.511s
user 0m0.016s
sys 0m0.016s

Can someone help me understand what's going on? what would change with the packet sniffer ON ?
does it bypass firewall? does it bypass NAT?
i know if i do notrack between the 2 network, it'd be fast. but i do need to netmap the subnet, so i can't do that with this scenario.
i have fasttrack on, but i suspect these packets don't hit it?
help?
 
mkx
Forum Guru
Forum Guru
Posts: 2829
Joined: Thu Mar 03, 2016 10:23 pm

Re: bizarre performance issue with packet sniffer

Fri Sep 06, 2019 6:04 pm

Packet sniffer disables fast-track. Try disabling the firewall filter rule which enables fast-track to see if this makes any difference.

If it does, then you'll have to check the packet flow and which rule grabs or misses the initial packets causing them to be misrouted ...
BR,
Metod
 
esunarto
just joined
Topic Author
Posts: 17
Joined: Wed Jul 18, 2018 11:42 pm

Re: bizarre performance issue with packet sniffer

Fri Sep 06, 2019 6:29 pm

Packet sniffer disables fast-track. Try disabling the firewall filter rule which enables fast-track to see if this makes any difference.

If it does, then you'll have to check the packet flow and which rule grabs or misses the initial packets causing them to be misrouted ...
bingo! when i disable fasttrack, the connection is fast again.
i'm assuming this is the packet flow you're talking about? https://wiki.mikrotik.com/wiki/Manual:Packet_Flow
it doesn't quite mention fasttrack, and my understanding of fasttrack is limited.
as far as i understand, fasttrack is a tcp header tag that skip certain feature, right? it allows SNAT/DNAT. does that include snat nmap and dnat nmap?
my tcpdump on the server show that the return packets were timing out and had to be resent. so i can start there.
 
esunarto
just joined
Topic Author
Posts: 17
Joined: Wed Jul 18, 2018 11:42 pm

Re: bizarre performance issue with packet sniffer

Fri Sep 06, 2019 6:44 pm

so i'm doing packet sniffer again, here's a snapshot of the first packets.
i highlighted the reverse nmap from 192.168.5.128 to 192.168.6.128.
this is done automatically by mtik i assume on bridge dst-nat.
is this the one that would get missed if fasttrack was enabled?
i can't quite tell what it would be because packet sniffer turn off fasttrack, so how does one debug fasttrack?

Thanks again
You do not have the required permissions to view the files attached to this post.
 
mkx
Forum Guru
Forum Guru
Posts: 2829
Joined: Thu Mar 03, 2016 10:23 pm

Re: bizarre performance issue with packet sniffer

Fri Sep 06, 2019 7:57 pm

fasttrack is a certain feature of connection tracked firewall ... when conditions are met and connection gets fasttrack mark, then (majority of) packets belonging to such connection skip firewall processing altogether. And mangling and IPsec policy matching and ... Only one of many packets gets treated in the usual way by usual rules. If there's a configuration error at fasttrack rule making that one a tad too greedy, most packets of such connections get mistreated, but some of them arrive at destination ... with TCP this means a lot of retransmissions for a sluggish connection.

I can not comment on packet sniffer results, to do that I'd have to see complete /ip firewall configuration.
BR,
Metod
 
esunarto
just joined
Topic Author
Posts: 17
Joined: Wed Jul 18, 2018 11:42 pm

Re: bizarre performance issue with packet sniffer

Sat Sep 07, 2019 12:47 am

If there's a configuration error at fasttrack rule making that one a tad too greedy, most packets of such connections get mistreated, but some of them arrive at destination ... with TCP this means a lot of retransmissions for a sluggish connection.
this is exactly what happen if i turn on fasttrack.
fasttrack has mostly default rules that user can't change, right?
how can i diagnose/troubleshoot fasttrack? i even tried to do a forwarding accept before fasttrack rule, no avail.
 
mkx
Forum Guru
Forum Guru
Posts: 2829
Joined: Thu Mar 03, 2016 10:23 pm

Re: bizarre performance issue with packet sniffer

Sat Sep 07, 2019 1:43 pm

fasttrack has mostly default rules that user can't change, right?
how can i diagnose/troubleshoot fasttrack? i even tried to do a forwarding accept before fasttrack rule, no avail.
Any firewall filter rule can be changed, including the default ones.

As to rule trouble shooting ... I don't know any explicit way of doing it ... other than to thoughtfully consider how a particular packet is processed by consecutive rules and why things go differently than expected. wireshark is your friend, but in this case you'd have to capture packets on the "outer" side of router ... to check what kind of packets exit when things are going wrong. And obviously packet capture won't help here.

A hint: firewall filter rule order matters. It is essential that you have a rule that accepts packets which should not be fasttracked and this rule has to be above the (general) fasttrack rule. (The other possibility being overloading fasttrack rule with all the exceptions whish is sometimes hard or impossible to do.)
At the same time you have to be aware that when a connection gets fasttracked, it can not be un-fasttracked. Hence whenever you change some rules which might affect fasttracking of a connection, you have to test it by initiating a completely new connection.
You may need to reboot the router ftom time to time to clear connection tracking state just to be sure it's not way off.
BR,
Metod
 
mcrobertg7
just joined
Posts: 1
Joined: Wed Sep 18, 2019 10:00 am

Re: bizarre performance issue with packet sniffer

Wed Sep 18, 2019 10:23 am

bingo! when i disable fasttrack, the connection is fast again.
i'm assuming this is the packet flow you're talking about? https://wiki.mikrotik.com/wiki/Manual:Packet_Flowwritemyessaytoday
it doesn't quite mention fasttrack, and my understanding of fasttrack is limited.
as far as i understand, fasttrack is a tcp header tag that skip certain feature, right? it allows SNAT/DNAT. does that include snat nmap and dnat nmap?
my tcpdump on the server show that the return packets were timing out and had to be resent. so i can start there.
Try the following rule to log packets:
/ip firewall mangle
add action=log chain=postrouting dst-port=50325 protocol=tcp

Who is online

Users browsing this forum: No registered users and 18 guests