This really sounds so unbelievable to me. It’s like a car manufacturer explaining their customers that this specific car only tops out at 15km/h on tarmac roads and they’re working on the issue and fix it “somewhere” in the future. But offroad the car is fine. Like people use their car to go to work “offroad” each day.
To be honest, everyone buying these CCR’s are using it for TCP. This bottleneck is really critical and should be fixed asap. I cannot digest that I just bought a CCR for a pretty steep price, not being able to handle TCP traffic properly on some ipsec tunnel.
I am very surprised with you explanation.
We bought several CCR models to be able to handle ipsec VPN tunnels at high speed. Also because of the hardware AES support.
We build VPN networks for our customers. That’s our job.
Now you are telling us that the ipsec speed problems (which are mentioned a lot on this forum) will be fixed in the future???
Please make fast ipsec tunnel and transport handling top priority.
We are going to consider other brands for VPN if Mikrotik is not going to improve.
It is very difficult to find other vendors selling TILE. For now you should try my advice on having multiple tunnels. CPUs like TILE and GPUs are something new to mikrotik so there are bound to be difficulties making full use of it.
If this really just a fragmentation-related issue, wouldn’t explicit MSS clamping solve the issue? Just wondering.
I personally don’t have access to any of the CCRs, unfortunately, otherwise I would have tested it myself.
Its ok, these OEM users dont seem to want to listen to people who use mikrotik hardware in their hacking activities or even experienced and expert users in ways to overcome or get around the problem. With that attitude I’m just glad im not their customers for the services they offer.
Edit: on the history of the VLIW architecture there have been problems making full use of it as evident on the AMD 5xxx and 6xxx series GPUs.
It’s not “just” fragmentation. I’m using MSS clamping, throughput is still pretty poor.
I do believe it has to do something with the offloading, because I don’t see any CPU spikes on the Tilera CPU.
It looks like Mikrotik has finally improved VPN performance in CCR models.
Here is the changelog for 6.24rc2
What's new in 6.24rc2 (2014-Dec-10 11:04):
*) fixed problem where some of ethernet cards do not work on x86;
*) improved CCR ethernet driver (less dropped packets);
*) improved queue tree parent=global performance (especially on SMP systems and CCRs);
*) eoip/eoipv6/gre/gre6/ipip/ipipv6/6to4 tunnels have improved per core balancing on CCRs;
*) fixed tx for 6to4 tunnels with unspecified dst address;
*) fixed vrrp - could sometimes not work properly because of advertising bad set of ip addresses;
My testing started at 6.33.3. While others may be reporting performance improvements from earlier versions, I find the hardware encryption quality with IPSEC on CCR1036 dissatisfying. Testing with everything identical except setting and unsetting ipsec-secret in GRE config (enabling and disabling encryption) shows night and day difference in connection quality (TCP retransmission, out of order packets, packet loss, etc). Unfortunately, many applications are sensitive to this. For example, my SMB tests dropped by about 10x (went from 250Mbps/450Mbps down/up to 25/45Mbps). While software encryption improves the latter SMB numbers by about 4x (not seeing the same retransmissions, out of order, loss, etc as before), that also means that I no longer benefit from parallel streams (now limited by single CPUs software encryption abilities). That tells me the single core in the hardware encryption isn’t the limitation (since it still benefits from parallel streams), rather there are issues with quality as already mentioned. It has been over 1 year since that 6.24rc2 release. I hope that doesn’t mean work to fix this has ceased.
I also noticed in-state-sequence-errors under /ip ipsec statistics increasing when quality is poor.
“general” ROS multi-core scalability. whcih is slowly, but improved from versions to versions.
lack of truly-scalable cihper modes.
scalabe chipers itself.
for example Serpent(https://en.wikipedia.org/wiki/Serpent_(cipher)) in EAX mode (https://en.wikipedia.org/wiki/EAX_mode ) was Wastly more scalable in multi and Many -core environment than Rinjael/AES in CBS, XTS and GCM modes (EAX had bit more overhead as 2-p things, than say CCM, but it SCALABLE !!! and benefits from both better hardware and software fine-tuning, while more ancent things - are NOT)
for “multi-core”(ie with strea/core-count equal to 4x and below) - maybe.
but for many-core environment, including most of tilera chips for example - its simply both waste lot of resources and DO NOT scale well.
there isn’t any “should be” in engineering or science.
there only “can do” and relevant “know how” of Good engineers and “can’t do that, Dave” excuses of Bad ones.
and thats basically all. among not-technical aspects of it.
and in my opinion OCB, CWC, EAX support essential in both performance, scalability, securty as GCM support for similar reasons are(compared to CCM and more ancient modes)
btw there also was very funny and interesting GCM fork aswell, named SGCM https://en.wikipedia.org/wiki/Sophie_Germain_Counter_Mode which may be handy in networking for obvious reasons/advantages over generic/original GCM and Considerably boost its securty too.
but so far i think CWC is Most interesting candidate to start such work with(more clear benefits, less legal obstacles, documented src-code, etc).
+1 for massive packet reordering in simpe IPSEC+GRE case.
Bug can be easily reproduced on 2 CCRs connected by typical 5..15Mb WAN.
iperf3 in UDP mode reporting out of order packets even on manual bandwidth regulation (10-30% of actual link bandwidth)
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.1.0.7, port 46784
[ 5] local 192.168.2.2 port 5201 connected to 10.1.0.7 port 45391
iperf3: OUT OF ORDER - incoming packet = 3 and received packet = 4 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 6 and received packet = 7 AND SP = 5
....
iperf3: OUT OF ORDER - incoming packet = 148 and received packet = 149 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 151 and received packet = 152 AND SP = 5
[ 5] 9.00-10.00 sec 128 KBytes 1.05 Mbits/sec 0.833 ms 5/16 (31%)
[ 5] 10.00-10.09 sec 0.00 Bytes 0.00 bits/sec 0.833 ms 0/0 (-nan%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 5] 0.00-10.09 sec 1.19 MBytes 987 Kbits/sec 0.833 ms 42/152 (28%)
[SUM] 0.0-10.1 sec 42 datagrams received out-of-order
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
And this is only 1Mbit between 2 CCRs.
IPSec is working stable - UDP flooding over IPSEC always utilise all bandwidth. Statistics is almost clean:
Could you please post a sample config with GCM? There’s clearly more to it than just changing the encryption algorithm in the proposal. I wasn’t able to achieve a tunnel after updating the proposal. What else needs to be set?!
