Community discussions

 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Please add performance results for IPsec tunnel!

Mon Jun 22, 2015 11:28 am

Hello,

My suggestion:
Please add performance results for IPsec tunnel (AES) to "Performance test results" table on each product page.
Interested the maximum speed of a single tunnel.
do not ask me why it is necessary.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Tue Oct 24, 2017 2:10 pm

As I see, you added "IPsec test results" for some products, like this https://mikrotik.com/product/CCR1009-7G-1C-1Splus

Some questions:

1) how many threads were used in Single tunnel?
2) it's TCP or UDP throughput?
3) why you publish results only for products with hardware ipsec?
do not ask me why it is necessary.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Please add performance results for IPsec tunnel!

Tue Oct 24, 2017 2:16 pm

It is stateless traffic, so you could say it is UDP. There is no use of testing devices without hardware acceleration, because their performance difference between models is insignificant.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Tue Oct 24, 2017 4:25 pm

>>It is stateless traffic, so you could say it is UDP.
Please add result for "Single tunnel TCP single thread". Its very useful info, for example as file copying.

>>There is no use of testing devices without hardware acceleration, because their performance difference between models is insignificant.
RB3011UiAS-RM should be much faster than RB2011UiAS-RM.
do not ask me why it is necessary.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Thu Oct 26, 2017 5:57 pm

I tested two 750G r3 (6.39.3), connected via EoIP tunnel with IPsec.
Windows file copy test show only 33 MB/s (264 Mbps). This is very far from declared 477 Mbps https://mikrotik.com/product/RB750Gr3.

Maybe you add also results for some popular tunnels+ipsec (l2tp+ipsec, gre+ipsec, eoip+ipsec)?
do not ask me why it is necessary.
 
onnoossendrijver
Member
Member
Posts: 418
Joined: Mon Jul 14, 2008 11:10 am
Location: The Netherlands

Re: Please add performance results for IPsec tunnel!

Thu Oct 26, 2017 6:09 pm

Maybe EoIP is responsible for that. Can you check without EoIP ?
I think it is still a good result for such device.
Linux/network engineer: ITIL, LPI1, CCNA R+S, CCNP R+S, JNCIA, JNCIS-SEC
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Thu Oct 26, 2017 6:32 pm

I tested two 750G r3 (6.39.3), connected via EoIP tunnel with IPsec.
Windows file copy test show only 33 MB/s (264 Mbps). This is very far from declared 477 Mbps https://mikrotik.com/product/RB750Gr3.

Maybe you add also results for some popular tunnels+ipsec (l2tp+ipsec, gre+ipsec, eoip+ipsec)?
Windows file copy is highly dependent on latency. Even a 2ms latency will make a huge difference. Did you test on local network, or through the internet? Also, fragmentation should be avoided.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Thu Oct 26, 2017 6:34 pm

I tested in 1Gbit LAN
do not ask me why it is necessary.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Thu Oct 26, 2017 11:43 pm

I tested in 1Gbit LAN
Well, so network wasn't the problem. I can't test this, since I don't have two units on gigabit. What profile said? Was the CPU running at 100%? What was the process using most CPU?
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 1:00 pm

hex_eoip_ipsec.png
You do not have the required permissions to view the files attached to this post.
Last edited by mikruser on Fri Oct 05, 2018 1:36 pm, edited 1 time in total.
do not ask me why it is necessary.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 1:59 pm

Ok. So, your problem isn't exactly IPsec - it is using 0,5% of your CPU power. Ho is your firewall? What are the rules? Maybe there is something there in need of optimization...

I have no experience with EOIP, so I don't know how much CPU it uses.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 2:09 pm

Firewall is blank
These two Hex is direct connected and used as encrypted wire in LAN
do not ask me why it is necessary.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 4:15 pm

Firewall is blank
These two Hex is direct connected and used as encrypted wire in LAN
No rule whatsoever? Not a single one? Not even the default ones? If this is true, You are not using fasttrack.

Do you have this rule on Your firewall? If not, then add it and test again.
/ip firewall filter
add action=fasttrack-connection chain=forward comment="defconf: fasttrack" connection-state=established,related
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 5:11 pm

You do not understand. Its "L2 wire" only. No L3 forward.
do not ask me why it is necessary.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 5:30 pm

It can't be L2 only if they are doing IPsec.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 5:39 pm

IPsec use "input" and "output" chain, not "forward".
do not ask me why it is necessary.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 6:03 pm

IPsec use "input" and "output" chain, not "forward".
Traffic inside the IPsec tunnel still crosses the forward chain.
https://wiki.mikrotik.com/wiki/Manual:P ... outerOS_v6

And You have "networking" using about 100% on 2 cores. I'd look at fasttrack. Another possibility is fragmentation: it should be avoided, as it is a CPU hog.

Just occurred to me: You said the traffic was about 260 Mb/s. It was just download? The figure of ~450Mbps IPsec is adding up and down. The crypto engine doesn't care which way the packets are flowing. You can have 225/225, 350/100, 200/250... Whatever adds to 450Mbps. And this is with 1400 bytes packets. With smaller ones the number will be worse.
 
didomir
just joined
Posts: 15
Joined: Tue Dec 22, 2015 9:45 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 6:11 pm

You can find information here how the tests has been done: https://wiki.mikrotik.com/wiki/Manual:I ... imizations

And here is some generic article, might be useful: https://wiki.mikrotik.com/wiki/Manual:P ... _Generator
Having fun with RB850Gx2,RB750Gr3,RB962UiGS-5HacT2HnT,RBmAP2nD,RB952Ui-5ac2nD,RB951G-2HnD,RB960PGS, RBMetalG-52SHPacn...
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 6:32 pm

Paternot
>>Traffic inside the IPsec tunnel still crosses the forward chain
No
eoip_ipsec.png
>>Just occurred to me: You said the traffic was about 260 Mb/s. It was just download?
Its unidirectional file copy (download or upload)
You do not have the required permissions to view the files attached to this post.
do not ask me why it is necessary.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 6:59 pm

didomir
>>You can find information here how the tests has been done: https://wiki.mikrotik.com/wiki/Manual:I ... imizations
This is synthetic UDP test.
True "real life" test its TCP single connection, as i suggested.
do not ask me why it is necessary.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 9:08 pm

didomir
>>You can find information here how the tests has been done: https://wiki.mikrotik.com/wiki/Manual:I ... imizations
This is synthetic UDP test.
True "real life" test its TCP single connection, as i suggested.
This link lists good practices, in order to achieve better throughput - it have nothing to do with synthetic tests. They quote the figures of the synthetic tests, sure. But the good practices listed there will help all kinds of traffic.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 9:53 pm

Paternot
>>Traffic inside the IPsec tunnel still crosses the forward chain
No
eoip_ipsec.png

>>Just occurred to me: You said the traffic was about 260 Mb/s. It was just download?
Its unidirectional file copy (download or upload)
Well, the fluxogram I linked says so. Your network is
windows -> hEX3 -> IPsec -> hEX3 -> windows ?
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 10:07 pm

Windows(192.168.0.1)----()hEX(10.0.0.1)----EoIP+IPsec----(10.0.0.2)hEX()----(192.168.0.2)Windows
do not ask me why it is necessary.
 
Paternot
Long time Member
Long time Member
Posts: 607
Joined: Thu Jun 02, 2016 4:01 am
Location: Niterói / Brazil

Re: Please add performance results for IPsec tunnel!

Fri Oct 27, 2017 11:00 pm

Windows(192.168.0.1)----()hEX(10.0.0.1)----EoIP+IPsec----(10.0.0.2)hEX()----(192.168.0.2)Windows
Ok. And we have something chewing CPU. The EoIP is a GRE tunnel. It creates a new interface at each point.

This means that the traffic inside the EoIP will cross the firewall, using the forward chain. It is as if your router had grown an extra ethernet port, and what come out of it will follow normal rules.

I refer You, one more time, to this flowchart:
https://wiki.mikrotik.com/wiki/Manual:P ... outerOS_v6

It is the one named "Changes in RouterOS v6". A packet, from one windows machine, would follow this way (I considered it not bridged, but bridged would be the same, where we are concerned).

windows machine 1 -> ethernet on hEX3 -> Input Interface (not bridged) -> pre routing -> Routing Decision -> Input -> Ipse Policy? (Yes) -> IPSec Decryption -> Prerouting (again!) -> Routing Decision -> Forward (it goes to the other windows machine) -> out intercface (not bridged) -> Postrouting -> IPSec Policy (no) -> Interface queue tree -> output interface -> windows machine 2

I don't know why that one forward rule of yours hadn't the counter incremented. Maybe there is something there that will not match this traffic. As you don't have a default deny, everything would pass anyway.

BUT:
Even if I am wrong, and the internal EoIP traffic doesn't use the forward chain, You still have some problem with the network part - not the IPSec part. That 2 cores at 100% are not normal, with just 250 - 300 Mbps.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Mon Sep 24, 2018 4:53 pm

I also tested two RB3011 with 6.43.2, connected via EoIP tunnel with IPsec.
They showed an even lower speed, even with hardware acceleration: file copy only 22 MB/s with aes-128 cbc/ctr (this is very far from declared 407.7 Mbps).
Profile:
rb3011_eoip_ipsec.png
You do not have the required permissions to view the files attached to this post.
do not ask me why it is necessary.
 
User avatar
emils
MikroTik Support
MikroTik Support
Posts: 494
Joined: Thu Dec 11, 2014 8:53 am

Re: Please add performance results for IPsec tunnel!

Tue Sep 25, 2018 8:56 am

Thank you very much for your testing efforts. Please check the IPsec tunnel performance test manual page to see how maximum throughput numbers are achieved for each product. Adding or enabling any additional RouterOS feature apart from IPsec policies can reduce the throughput significantly. This includes EoIP, L2TP, queuing, firewall, connection tracking, bridging and so on.

Additionally, in your screenshot not a single CPU core is utilizing 100% of its resources or comes even close to it, so the throughput is most likely limited by some other factor outside your routers.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Tue Sep 25, 2018 12:40 pm

>>Adding or enabling any additional RouterOS feature apart from IPsec policies can reduce the throughput significantly.
That's why I already suggested that you also publish the results for some popular tunnels+ipsec (l2tp+ipsec, gre+ipsec, eoip+ipsec)
viewtopic.php?f=3&t=97880&sid=119f20542 ... c8#p625029

>>so the throughput is most likely limited by some other factor outside your routers.
I copy the file between my computer and the server. There are no limiting factors. Without Mikrotik's, the speed of copying is 105 MB/s
do not ask me why it is necessary.
 
User avatar
emils
MikroTik Support
MikroTik Support
Posts: 494
Joined: Thu Dec 11, 2014 8:53 am

Re: Please add performance results for IPsec tunnel!

Tue Sep 25, 2018 1:04 pm

The throughput results are there for you to evaluate the IPsec crypto engine performance, not to show you throughput results with various different configurations. Test cases can be unlimited and trivial at the same time. For example, you would like to know EoIP over IPsec throughput - connection tracking enabled or disabled? Again, each setup is and will be different and throughput results will differ.

Anyway, I will not be able to help you troubleshoot the performance issue by just looking at your provided screenshot. Some things to look for - check for packet fragmentation, make sure all interfaces are linked to 1G, make sure you use the same switch chip for both incoming and outgoing traffic, test the throughput with simple EoIP without using IPsec.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Tue Sep 25, 2018 2:06 pm

>>The throughput results are there for you to evaluate the IPsec crypto engine performance, not to show you throughput results with various different configurations.
IPsec crypto engine performance is a "spherical cow in a vacuum", and does not show real life results.

>>check for packet fragmentation
MTU for EoIP tunnel is autoconfigured (1416 for aes-ctr)
I try 1300, but got same speed.

>>make sure all interfaces are linked to 1G
Yes, all interfaces rate is 1G

>>make sure you use the same switch chip for both incoming and outgoing traffic
Yes, i use same switch chip (eth2 and eth3)

>>test the throughput with simple EoIP without using IPsec
File copy over EoIP without IPsec: ~80 MB/s
do not ask me why it is necessary.
 
User avatar
emils
MikroTik Support
MikroTik Support
Posts: 494
Joined: Thu Dec 11, 2014 8:53 am

Re: Please add performance results for IPsec tunnel!

Tue Sep 25, 2018 5:18 pm

And then again, you are missing the main point - there is no single setup that would represent "real life" throughput. Each user will have their own configuration and requirements which will have different impact on IPsec throughput.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Tue Sep 25, 2018 7:55 pm

You can use minimal (fastest) config, required for EoIP+IPsec or L2TP+IPsec or GRE+IPsec.
do not ask me why it is necessary.
 
mikruser
Member
Member
Topic Author
Posts: 380
Joined: Wed Jan 16, 2013 6:28 pm

Re: Please add performance results for IPsec tunnel!

Fri Oct 05, 2018 1:33 pm

I also tested two hAP ac^2 with 6.43.2

EoIP with IPsec (aes-128 ctr), file copy is only 34 MB/s:
hapac2_eoip_ipsec_ctr.png
EoIP without IPsec, file copy is 68 MB/s:
hapac2_eoip.png
You do not have the required permissions to view the files attached to this post.
do not ask me why it is necessary.
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 5934
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: Please add performance results for IPsec tunnel!

Fri Oct 05, 2018 4:54 pm

MD5?

Who is online

Users browsing this forum: No registered users and 8 guests