Community discussions

MikroTik App
 
acrophobic
newbie
Topic Author
Posts: 31
Joined: Fri Jan 04, 2013 3:56 pm

IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Wed Nov 09, 2022 6:31 am

Hi!

I have two hEX's in two locations, with site-to-site IPsec VPN configured between them. At site A, there is a 250/250 Mbps internet connection (about 240 Mbps down and 230 Mbps up according to Speedtest) via fiber. At site B, I have a 600/50 Mbps connection (about 620 Mbps down and 52 Mbps up on Speedtest) via the cable TV network (with the modem in bridge mode). Now, I have no real use of 600 Mbps, but had to choose it to get any reasonable speed up. Anyway, speed from site A to site B was about 180 Mbps and in the other direction pretty much 50 Mbps.
Until a few days ago, both routers were running RouterOS 6.49.6. Now this is a bit silly, but I hadn't realized that I needed to change the channel to upgrade to RouterOS 7. I simply thought that version 7 wasn't available to my devices. But then I started Googling around a bit, because I really wanted to try WireGuard (not to replace the IPsec tunnel, but to replace L2TP when connecting from my phone, etc), so when I realized I indeed could upgrade to RouterOS 7, I got a bit excited and didn't really think things through... I upgraded the router at site A, set up WireGuard and was amazed how well WireGuard worked, compared to L2TP, so without any further testing, I upgraded the router at site B as well. No other changes were made (such as altering FW rules, etc). The IPsec traffic is BTW handled through the FW, like any other traffic. Somehow, I wasn't expecting a big performance penalty from upgrading to RouterOS 7. Thing is, when I'm at site B, I regularly remote into a VM at site A (with VNC), and before upgrading, performance was basically just like being at site A. But when I got to site B after the upgrade, I noticed that things weren't at all as smooth as they used to and after testing, the speed from site A to site B had dropped from 180 Mbps to about 115 Mbps. I have tested a lot now, and I've noticed that the CPU cores in the site A router doesn't max out, but that one core at site B pretty much does, so I assume it's more CPU intensive to encrypt traffic, than to decrypt it.. But then again, isn't it supposed to be HW encryption? I do route all internet traffic from site B, through site A, but I don't notice any improvement when not doing so. When I check Tools - Profile, what is using most CPU is "networking". Since the router at site B pretty much maxes out one core, but site B doesn't, I've thought of maybe downgrading the site B router to RouterOS 6, to hopefully get a few Mbits back (since I really only need WireGuard at site A).

I know this is a lot of text, but I have a few questions...

1. Is an IPsec performance drop of this magnitude normal when going from RouterOS 6 to RouterOS 7?
2. Are there any good ways to "cure" or mitigate this?
3. Since the IPsec encryption is HW based, why is it consuming so much CPU?
4. What is the correct (and safe) way to downgrade from RouterOS 7 to RouterOS 6?
5. Thinking a little bit into the future, what is cheapest wired Mikrotik router after the hEX, which has better IPsec performance?

Thank you and all the best!
 
User avatar
Znevna
Forum Guru
Forum Guru
Posts: 1347
Joined: Mon Sep 23, 2019 1:04 pm

Re: IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Wed Nov 09, 2022 7:59 am

Yet another performance drop topic.
Use the search button next time.
viewtopic.php?p=966544
viewtopic.php?p=966120
And .. others.
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Wed Nov 09, 2022 10:18 am

I would assume that the higher load/performance drop on the affected site is not related to IPsec encryption/decryption but to the fact that the complexity of the configuration is higher there than on the unaffected site.

Regarding hardware encryption, what does /ip/ipsec/installed-sa/print show? There should be a H flag in the second column, but ROS 7 seems to only explain the flags in the header row of a print if a given flag is present in at least one data row so maybe some other letter is used - I don't have any device supporting hardware encryption and running ROS 7. So another way to do the same - the SAs shown by /ip/ipsec/installed-sa/print where hw-aead are hardware-encrypted, those shown by /ip/ipsec/installed-sa/print where !hw-aead are not.
 
acrophobic
newbie
Topic Author
Posts: 31
Joined: Fri Jan 04, 2013 3:56 pm

Re: IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Wed Nov 09, 2022 2:06 pm

Yet another performance drop topic.
Use the search button next time.
viewtopic.php?p=966544
viewtopic.php?p=966120
And .. others.
Believe me, I have searched and read both of those threads and a few others. I probably should have mentioned this, but thought I'd written enough text already. But I didn't think they answered my questions in a satisfactory way. BTW, some of the specific questions I have, I guess could have been asked in separate threads, but I chose to ask them all in one thread, since they are related.
I would assume that the higher load/performance drop on the affected site is not related to IPsec encryption/decryption but to the fact that the complexity of the configuration is higher there than on the unaffected site.

Regarding hardware encryption, what does /ip/ipsec/installed-sa/print show? There should be a H flag in the second column, but ROS 7 seems to only explain the flags in the header row of a print if a given flag is present in at least one data row so maybe some other letter is used - I don't have any device supporting hardware encryption and running ROS 7. So another way to do the same - the SAs shown by /ip/ipsec/installed-sa/print where hw-aead are hardware-encrypted, those shown by /ip/ipsec/installed-sa/print where !hw-aead are not.
The configuration on both sites are basically mirrored, if anything, the router at site A, does more than the one at site B. BTW, traffic in the opposite direction, where the 50 Mbps upload is the bottleneck, neither of the routers max out, but site A's router then gets a higher CPU load than site B, which strengthened my theory that the decryption is somehow more intensive.

Second letter is an H. And if I check under "Installed SAs" in Winbox and hover the mouse pointer over the H, it says "E - ESP, H - Hardware AEAD"
 
sindy
Forum Guru
Forum Guru
Posts: 10205
Joined: Mon Dec 04, 2017 9:19 pm

Re: IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Wed Nov 09, 2022 6:07 pm

Second letter is an H. And if I check under "Installed SAs" in Winbox and hover the mouse pointer over the H, it says "E - ESP, H - Hardware AEAD"
OK, so encryption/decryption is not the source of additional load as it runs in hardware.

The configuration on both sites are basically mirrored, if anything, the router at site A, does more than the one at site B. BTW, traffic in the opposite direction, where the 50 Mbps upload is the bottleneck, neither of the routers max out, but site A's router then gets a higher CPU load than site B, which strengthened my theory that the decryption is somehow more intensive.
Post both configurations - a supposedly minor difference may change a lot. Also, other traffic (not transported using IPsec) may cause the CPU load.
 
acrophobic
newbie
Topic Author
Posts: 31
Joined: Fri Jan 04, 2013 3:56 pm

Re: IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Wed Nov 16, 2022 4:52 am

Hi and thank you for your replies!

Just a quick update. I've now tested A LOT! I'm way too tired to write everything down right now. But I've googled through many many posts here on the forum and tested things that seemed even remotely relevant, but to no help. Tonight I even tested setting up site-to-site VPN with WireGuard instead of IPsec. Wasn't expecting great speeds, since there is no hardware encryption support with WireGuard (at least not with my devices), but I just wanted to try it out and do some experimenting, to see how things would behave compared with IPsec. But what I was also not expecting was pretty lousy speeds WITHOUT any core on either device maxing out. Maximum speed reached over the tunnel was a bit over 70 Mbps, but no core ever maxed out. Why would this be?
Before trying WireGuard site-to-site, I did a Btest directly from one router to the other over the IPsec tunnel and then reached over 200 Mbps, unfortunately I don't get this speed other than directly between the devices. An odd thing was that this was with UDP checked, if I tested with TCP, I only got like 15 Mbps (despite low CPU usage)...
Haven't posted my configs yet, as I find that a bit scary:P
I'm now seriously contemplating downgrading both routers to ROS6 and buy an extra RB750Gr3, upgrade the new one to ROS7 and use that purely as a WireGuard server. Sure, that could be done with a Raspberry Pi, but a new RB750Gr3 will probably be somewhere in the same price range and then I won't have to spend additional time to learn how to set it up on the Pi.
 
acrophobic
newbie
Topic Author
Posts: 31
Joined: Fri Jan 04, 2013 3:56 pm

Re: IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Thu Nov 17, 2022 2:58 pm

Hi!

So, I downgraded the router at site B to ROS6 and now IPsec speeds are almost back to the speeds I had before upgrading either of the routers. Now it's the router at site A that is maxing out, but that's fine, considering the speeds I'm getting. Now, iperf between two clients on either side is still a bit slow (for whatever reason), at about 100 Mbps, but internet speeds (since I'm pushing all traffic through site A), are reaching about 200 Mbps (according to speedtest.net) and the feeling when using VNC from site B to site A is almost the same as when both routers were running ROS6. I have now rethought things a bit since my previous post and plan now is to be happy with things as they are now and when it's available (and I can afford it, even though the list price seems to be absolutely amazing value for money), get one of these puppies for site A: https://mikrotik.com/product/hap_ax3 This way (even though I in general prefer separate gear), I get a neater solution at site A, which will definitely not have any bottlenecks and I free up a router for fun projects or experimentation (and I can move the AP at site A, to site B where a better AP is needed anyway).
Anyway, to answer my own question number 4, the answer was obviously Netinstall, which I've never done before and which caused a bit of headache. I'm a Linux user and never got the router to connect to the Netinstall program running in Wine. I thought it was due to some bridging I have going on on the computer, but no matter how I tried, it wouldn't work. In the end I tried the Linux CLI Netinstall tool and then it finally worked. Now, I generally like CLI tools over GUI ones, but since this was the first time, I thought Netinstall in Wine would be easier.
I now consider this solved, yet not solved, so if anyone have anything to add, feel free to do so! (For example theories about my slow iperf speeds)
Thank you all!
 
gabacho4
Member
Member
Posts: 331
Joined: Mon Dec 28, 2020 12:30 pm
Location: Earth

Re: IPsec performance problem after upgrading to RouterOS 7 on the RB750Gr3

Thu Nov 17, 2022 4:31 pm

I could have sworn I saw a post earlier this week from someone at MikroTik saying that they generally recommended keeping the hex and similar devices on ROS 6, i believe due to storage limitations or something. Perhaps this is yet another reason? Seems the downgrade to 6 clearly fixed things for OP.

Who is online

Users browsing this forum: DanMos79, dozer46, godel0914, GoogleOther [Bot] and 59 guests