Community discussions

MikroTik App
 
milosm
just joined
Topic Author
Posts: 4
Joined: Wed Jan 26, 2022 2:29 pm

Slow IPsec VPN in one way

Wed Jan 26, 2022 2:57 pm

Hi,
I have strange problem. I have IPsec VPN tunnel between 2 buildings. In past (firmware version about 6.45) I was able transfer data via tunnel about 6 MB/s = 48 mbit/s in both directions.
After upgrade of firmware (I don´t remember version, but it was 6.46 or 6.47) I started see, that speed in one way decreased significantly to 1,7 - 2 MB/s. Speed in opposite way is still same (about 6 MB/s). Details are on attached picture. No other configuration change was performed except of firmware upgrade.
I tried perform bandwidth tests and it looks like MTU issue. If default MTU 1500 bytes is used, packets need fragmentation what result in lower speed. (there are PPPoE and IPsec header). If I decreased MTU in bandwidth test to some lower value like 1300 bytes, I was able measure higher speed.
(Zyxel is using "MSS Adjustment Auto" option, so I guess this is reason why one way is not affected. Is there similar option in MikroTik ?)
I would like to ask you for your opinion on this issue. And if this is caused by MTU as I suspect, is there any option to lower MTU on input interface to prevent fragmentation ?
(I tried change/lower MTU on interface ether1 where internet line is connected, but it didn´t help).
Any advise is appreciated.
Thank you in advance
Best regards
Milos
You do not have the required permissions to view the files attached to this post.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Slow IPsec VPN in one way

Thu Jan 27, 2022 2:10 pm

I'm afraid lowering MTU at ether1 affects how Mikrotik will fragment packets it sends out via ether1, but not how devices connected to ether1 will fragment what they send via the Mikrotik.

If the IPsec policy at the "PC" site (the right house at the drawing) has 0.0.0.0/0 as dst-address, it is possible that the ICMP "fragmentation needed" messages sent by the hAP ac to the PC may get intercepted and sent to the "Synology" site as well, is that the case?
 
milosm
just joined
Topic Author
Posts: 4
Joined: Wed Jan 26, 2022 2:29 pm

Re: Slow IPsec VPN in one way

Thu Jan 27, 2022 3:17 pm

I'm afraid lowering MTU at ether1 affects how Mikrotik will fragment packets it sends out via ether1, but not how devices connected to ether1 will fragment what they send via the Mikrotik.
I wanted force MikroTik RB760iGS to send smaller packets. From this router point of view sending is slow. But you are right, it affected not only IPsec traffic, but all packets going via ether1 to internet.
If the IPsec policy at the "PC" site (the right house at the drawing) has 0.0.0.0/0 as dst-address, it is possible that the ICMP "fragmentation needed" messages sent by the hAP ac to the PC may get intercepted and sent to the "Synology" site as well, is that the case?
IPsec policy is not 0.0.0.0/0. I use policy (source) 172.27.10.0/24 <--> (destination) 192.168.1.0/24 and everything else is send directly to internet via ether1 (by default route).
Moreover I worry that message "ICMP fragmentation needed" is send only if packet is too big and "Don´t Fragment" bit is set. Otherwise fragmentation will happen, packet is send in more fragments and no ICMP message is generated. At least no such message is captured by Wireshark on "PC".

Note: I just suspect that MTU may be an issue (based on bandwidth tests with different size), but I am not 100% sure. Maybe there is something else what I overlooked. I only know that this config was working before and without any change (except of firmware upgrade) it slowed down.
But if you think it can be something different I can test. Also if you need more details, just let me know what I should attach to this post to help you.
Thanks
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Slow IPsec VPN in one way

Thu Jan 27, 2022 4:14 pm

But you are right, it affected not only IPsec traffic, but all packets going via ether1 to internet.
That's not what I had in mind, I had in mind that information about MTU is not communicated to the adjacent device over the cable, because I've missed you wrote that ether1 was WAN. So I had in mind that if you set MTU to 1400 at an interface to which a PC is connected, the PC still uses 1500 for the packets it sends to Mikrotik.

Since ether1 is the WAN here, reducing its MTU just means that you also reduce the MTU for the IPsec transport packets, so even smaller payload packets create transport packets that need to be fragmented.

Moreover I worry that message "ICMP fragmentation needed" is send only if packet is too big and "Don´t Fragment" bit is set. Otherwise fragmentation will happen, packet is send in more fragments and no ICMP message is generated. At least no such message is captured by Wireshark on "PC".
Yes, that ICMP message is only sent if the packet has the DF bit set, but that's normally the case for all TCP packets, as TCP optimizes packet size this way (it is more bandwidth efficient to slice the TCP payload stream into portions that are small enough to fully use the available MTU than to fragment the TCP packets).

Note: I just suspect that MTU may be an issue (based on bandwidth tests with different size), but I am not 100% sure. Maybe there is something else what I overlooked. I only know that this config was working before and without any change (except of firmware upgrade) it slowed down.
But if you think it can be something different I can test. Also if you need more details, just let me know what I should attach to this post to help you.
You can use the action=change-mss rules in mangle, but these also only affect TCP - this time, they modify the advertised maximum payload segment size in the SYN packets. So if your UDP tests show that with smaller packets the speed is not reduced, I'd be searching why the real connections do not adjust packet size. Are they TCP? If so, is the DF bit set? If not, it is strange. If yes, something must prevent the ICMP "fragmentation needed" from reaching the PC.
 
milosm
just joined
Topic Author
Posts: 4
Joined: Wed Jan 26, 2022 2:29 pm

Re: Slow IPsec VPN in one way

Thu Jan 27, 2022 5:22 pm

I tried use mangle rules to change MSS. See attached picture. But there is no hit on rule even if I decreased it more than enough.
ip-firewall-mangle test.png
Connection is TCP - file transfer via SMB

DF bit is set as you mentioned
DF bit.png
I looked deeper into packet capture. Most of packets have Length 1428 (-14 bytes Ethernet header - 20 bytes IP header = 1394 bytes). Looks correct.
wireshark.png
Just note if I try "ping -l 1394 -f 192.168.1.253"... 1394 is maximum for successful ping. With 1395 it needs fragmentation.
Ping.png
From time to time I see Length = 2802 but there is no "fragmentation needed" message after this packet.
Also there are some error rows in packet capture (they appear only rarely). I put example of them on screenshot.

I think nothing is preventing reach "fragmentation needed" to computer because with ping I am getting this message fine.
You do not have the required permissions to view the files attached to this post.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Slow IPsec VPN in one way

Thu Jan 27, 2022 5:45 pm

Just for the case, show me /ip firewall filter export - maybe your firewall rules are based on default ones from an old RouterOS version and thus they do not exempt packets matching IPsec policies from fasttracking. But if so, the speed should be reduced much more.

Regarding change-mss mangle rules not counting - all TCP SYN packets would have to have MSS < 1301, at least while passing through the forward chain, to get ignored by these rules, and this is unlikely as your capture shows the packets to be larger; the IP addresses are also correct in these rules as seen in the capture. So I can see no reasou why they should not count - TCP SYN packets are not fasttracked yet. Which leads me to a suspicion that there are some hidden ipfilter rules not shown in the RouterOS configuration, which would suggest that some malware is running on your device.
 
milosm
just joined
Topic Author
Posts: 4
Joined: Wed Jan 26, 2022 2:29 pm

Re: Slow IPsec VPN in one way

Thu Jan 27, 2022 6:45 pm

At first it is not related to firmware upgrade as I suspected initially. I tried downgrade to 6.46.7 (lowest firmware which I was able download and it is near factory firmware 6.46.3). And speeds are same i.e. upload is slow. So now upgraded back to 6.48.6

Firewall policies
Firewall_policies.png
There is no rule allowing access from Internet so nobody should be able login and install some malware (even if password would be brute-forced)
Is there any way to check/detect it ? (I checked scripts and scheduler and they are empty)
Is there any 100% way to remove malware if there is any ?
You do not have the required permissions to view the files attached to this post.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: Slow IPsec VPN in one way

Thu Jan 27, 2022 8:09 pm

Is there any way to check/detect it ? (I checked scripts and scheduler and they are empty)
A malware capable of inserting own firewall rules invisible in configuration and surviving a reboot would not leave traces in configuration, no point in searching for it this way. Sniffing the WAN traffic using another device could reveal something but you'd have to disconnect everything from the LAN side, otherwise it would be hard to distinguish malware traffic from the traffic of the LAN clients.

Is there any 100% way to remove malware if there is any ?
Only netinstall.

Who is online

Users browsing this forum: Amazon [Bot], astelsrl, CGGXANNX, en1gm4, eworm, Kanzler and 88 guests