MikroTik

Hello,

My suggestion:
Please add performance results for IPsec tunnel (AES) to "Performance test results" table on each product page.
Interested the maximum speed of a single tunnel.

As I see, you added "IPsec test results" for some products, like this https://mikrotik.com/product/CCR1009-7G-1C-1Splus

Some questions:

1) how many threads were used in Single tunnel?
2) it's TCP or UDP throughput?
3) why you publish results only for products with hardware ipsec?

It is stateless traffic, so you could say it is UDP. There is no use of testing devices without hardware acceleration, because their performance difference between models is insignificant.

>>It is stateless traffic, so you could say it is UDP.
Please add result for "Single tunnel TCP single thread". Its very useful info, for example as file copying.

>>There is no use of testing devices without hardware acceleration, because their performance difference between models is insignificant.
RB3011UiAS-RM should be much faster than RB2011UiAS-RM.

I tested two 750G r3 (6.39.3), connected via EoIP tunnel with IPsec.
Windows file copy test show only 33 MB/s (264 Mbps). This is very far from declared 477 Mbps https://mikrotik.com/product/RB750Gr3.

Maybe you add also results for some popular tunnels+ipsec (l2tp+ipsec, gre+ipsec, eoip+ipsec)?

Maybe EoIP is responsible for that. Can you check without EoIP ?
I think it is still a good result for such device.

I tested two 750G r3 (6.39.3), connected via EoIP tunnel with IPsec.
Windows file copy test show only 33 MB/s (264 Mbps). This is very far from declared 477 Mbps https://mikrotik.com/product/RB750Gr3.

Maybe you add also results for some popular tunnels+ipsec (l2tp+ipsec, gre+ipsec, eoip+ipsec)?

Windows file copy is highly dependent on latency. Even a 2ms latency will make a huge difference. Did you test on local network, or through the internet? Also, fragmentation should be avoided.

I tested in 1Gbit LAN

I tested in 1Gbit LAN

Well, so network wasn't the problem. I can't test this, since I don't have two units on gigabit. What profile said? Was the CPU running at 100%? What was the process using most CPU?

Ok. So, your problem isn't exactly IPsec - it is using 0,5% of your CPU power. Ho is your firewall? What are the rules? Maybe there is something there in need of optimization...

I have no experience with EOIP, so I don't know how much CPU it uses.

Firewall is blank
These two Hex is direct connected and used as encrypted wire in LAN

Firewall is blank
These two Hex is direct connected and used as encrypted wire in LAN

No rule whatsoever? Not a single one? Not even the default ones? If this is true, You are not using fasttrack.

Do you have this rule on Your firewall? If not, then add it and test again.

You do not understand. Its "L2 wire" only. No L3 forward.

It can't be L2 only if they are doing IPsec.

IPsec use "input" and "output" chain, not "forward".

IPsec use "input" and "output" chain, not "forward".

Traffic inside the IPsec tunnel still crosses the forward chain.
https://wiki.mikrotik.com/wiki/Manual:P ... outerOS_v6

And You have "networking" using about 100% on 2 cores. I'd look at fasttrack. Another possibility is fragmentation: it should be avoided, as it is a CPU hog.

Just occurred to me: You said the traffic was about 260 Mb/s. It was just download? The figure of ~450Mbps IPsec is adding up and down. The crypto engine doesn't care which way the packets are flowing. You can have 225/225, 350/100, 200/250... Whatever adds to 450Mbps. And this is with 1400 bytes packets. With smaller ones the number will be worse.

You can find information here how the tests has been done: https://wiki.mikrotik.com/wiki/Manual:I ... imizations

And here is some generic article, might be useful: https://wiki.mikrotik.com/wiki/Manual:P ... _Generator

Paternot
>>Traffic inside the IPsec tunnel still crosses the forward chain
No >>Just occurred to me: You said the traffic was about 260 Mb/s. It was just download?
Its unidirectional file copy (download or upload)

didomir
>>You can find information here how the tests has been done: https://wiki.mikrotik.com/wiki/Manual:I ... imizations
This is synthetic UDP test.
True "real life" test its TCP single connection, as i suggested.

didomir
>>You can find information here how the tests has been done: https://wiki.mikrotik.com/wiki/Manual:I ... imizations
This is synthetic UDP test.
True "real life" test its TCP single connection, as i suggested.

This link lists good practices, in order to achieve better throughput - it have nothing to do with synthetic tests. They quote the figures of the synthetic tests, sure. But the good practices listed there will help all kinds of traffic.

Paternot
>>Traffic inside the IPsec tunnel still crosses the forward chain
No
eoip_ipsec.png

>>Just occurred to me: You said the traffic was about 260 Mb/s. It was just download?
Its unidirectional file copy (download or upload)

Well, the fluxogram I linked says so. Your network is
windows -> hEX3 -> IPsec -> hEX3 -> windows ?

Windows(192.168.0.1)----()hEX(10.0.0.1)----EoIP+IPsec----(10.0.0.2)hEX()----(192.168.0.2)Windows

Windows(192.168.0.1)----()hEX(10.0.0.1)----EoIP+IPsec----(10.0.0.2)hEX()----(192.168.0.2)Windows

Ok. And we have something chewing CPU. The EoIP is a GRE tunnel. It creates a new interface at each point.

This means that the traffic inside the EoIP will cross the firewall, using the forward chain. It is as if your router had grown an extra ethernet port, and what come out of it will follow normal rules.

I refer You, one more time, to this flowchart:
https://wiki.mikrotik.com/wiki/Manual:P ... outerOS_v6

It is the one named "Changes in RouterOS v6". A packet, from one windows machine, would follow this way (I considered it not bridged, but bridged would be the same, where we are concerned).

windows machine 1 -> ethernet on hEX3 -> Input Interface (not bridged) -> pre routing -> Routing Decision -> Input -> Ipse Policy? (Yes) -> IPSec Decryption -> Prerouting (again!) -> Routing Decision -> Forward (it goes to the other windows machine) -> out intercface (not bridged) -> Postrouting -> IPSec Policy (no) -> Interface queue tree -> output interface -> windows machine 2

I don't know why that one forward rule of yours hadn't the counter incremented. Maybe there is something there that will not match this traffic. As you don't have a default deny, everything would pass anyway.

BUT:
Even if I am wrong, and the internal EoIP traffic doesn't use the forward chain, You still have some problem with the network part - not the IPSec part. That 2 cores at 100% are not normal, with just 250 - 300 Mbps.

I also tested two RB3011 with 6.43.2, connected via EoIP tunnel with IPsec.
They showed an even lower speed, even with hardware acceleration: file copy only 22 MB/s with aes-128 cbc/ctr (this is very far from declared 407.7 Mbps).
Profile:

Thank you very much for your testing efforts. Please check the IPsec tunnel performance test manual page to see how maximum throughput numbers are achieved for each product. Adding or enabling any additional RouterOS feature apart from IPsec policies can reduce the throughput significantly. This includes EoIP, L2TP, queuing, firewall, connection tracking, bridging and so on.

Additionally, in your screenshot not a single CPU core is utilizing 100% of its resources or comes even close to it, so the throughput is most likely limited by some other factor outside your routers.

>>Adding or enabling any additional RouterOS feature apart from IPsec policies can reduce the throughput significantly.
That's why I already suggested that you also publish the results for some popular tunnels+ipsec (l2tp+ipsec, gre+ipsec, eoip+ipsec)
viewtopic.php?f=3&t=97880&sid=119f20542 ... c8#p625029

>>so the throughput is most likely limited by some other factor outside your routers.
I copy the file between my computer and the server. There are no limiting factors. Without Mikrotik's, the speed of copying is 105 MB/s

The throughput results are there for you to evaluate the IPsec crypto engine performance, not to show you throughput results with various different configurations. Test cases can be unlimited and trivial at the same time. For example, you would like to know EoIP over IPsec throughput - connection tracking enabled or disabled? Again, each setup is and will be different and throughput results will differ.

Anyway, I will not be able to help you troubleshoot the performance issue by just looking at your provided screenshot. Some things to look for - check for packet fragmentation, make sure all interfaces are linked to 1G, make sure you use the same switch chip for both incoming and outgoing traffic, test the throughput with simple EoIP without using IPsec.

>>The throughput results are there for you to evaluate the IPsec crypto engine performance, not to show you throughput results with various different configurations.
IPsec crypto engine performance is a "spherical cow in a vacuum", and does not show real life results.

>>check for packet fragmentation
MTU for EoIP tunnel is autoconfigured (1416 for aes-ctr)
I try 1300, but got same speed.

>>make sure all interfaces are linked to 1G
Yes, all interfaces rate is 1G

>>make sure you use the same switch chip for both incoming and outgoing traffic
Yes, i use same switch chip (eth2 and eth3)

>>test the throughput with simple EoIP without using IPsec
File copy over EoIP without IPsec: ~80 MB/s

And then again, you are missing the main point - there is no single setup that would represent "real life" throughput. Each user will have their own configuration and requirements which will have different impact on IPsec throughput.

You can use minimal (fastest) config, required for EoIP+IPsec or L2TP+IPsec or GRE+IPsec.

I also tested two hAP ac^2 with 6.43.2

EoIP with IPsec (aes-128 ctr), file copy is only 34 MB/s: EoIP without IPsec, file copy is 68 MB/s:

MD5?