Community discussions

MikroTik App
 
subway
newbie
Topic Author
Posts: 35
Joined: Sat Oct 07, 2017 1:58 pm

CCR 1072 IPsec speed issue [SOLVED]  [SOLVED]

Wed Jan 03, 2018 1:10 pm

Hi,

We are running a CCR1072 as our core router (v6.41), with 3 IPsec site-to-site connections (SHA1/AES-CBC-128, should be HW accelerated). On the remote end there are CCR1036 routers.

What I noticed is with a very low average CPU utilization (1-2%), the IPsec tunnels are maxed out at around 200Mbits total (all 3 tunnels combined), and a single CPU core is running 100% all the time.

I did a profiling test on that core, and it was 60-65% "networking" and 25-30% "firewall". Which is strange as we dont have any firewall rules, not doing any NAT.

I tried to disable conn. tracking, but the load was not lowered. The filter, NAT, mangle and RAW tables are empty, the core router only does IPsec (policy based) and simple routing. No queues either. On the WAN interface I can also see excessive RX overflow which also indicates that the router is not able to process the incoming packets fast enough due to some limitations.

I am not even sure if this is an IPsec limitation, but something is clearly not right. The tunnels are working fine by the way, the only problem we have is the severe speed limitation.

If someone has any idea what to check or try, please let me know, cause I ran out of ideas at this point.

Thanks!

Solution:

Do NOT use the single gigabit copper port on the CCR1072 for traffic, only for management!

This is not stated in any official Mikrotik documentation, but someone in the past had the same issue, and this was the response form Mikrtoik support:
Hello,

Please check the block diagram:
https://i.mt.lv/routerboard/files/CCR10 ... 130622.png

ether1 is connected to a PCIe controller and it was designed that way since ether1 was only intended to be used as a management port.

rx-overflow counter is increasing when you have a faster device (or link) sending too much data into a slower interface. This is expected.

The solution is not to use ether1 for data links.

Best regards,
Arturs Z.
More infos:

https://community.ubnt.com/t5/airFiber/ ... -p/2054839

UPDATE (2018.01.07.):

Mikrotik updated the product page of CCR1072, now the limitation of the copper ethernet port is clearly stated:

"the CCR1072 is equipped with eight independently connected 10G SFP+ ports and single Ethernet port for management purposes."
Last edited by subway on Sun Jan 07, 2018 4:11 pm, edited 3 times in total.
 
p3rad0x
Long time Member
Long time Member
Posts: 637
Joined: Fri Sep 18, 2015 5:42 pm
Location: South Africa
Contact:

Re: CCR 1072 IPsec speed issue

Wed Jan 03, 2018 4:07 pm

Hi,

I don't have much experience with ipsec.

Using that combination should be HW accelerated but something seems to be causing it to fall back to software.

I had a issue where connection tracking was still active even if it was disabled.

in Winbox the connections was not visible but you could see it in the terminal.

The issue went away after I added a raw no-track rule. This was a CCR1036 doing about 900Mbps aggregate after the change it went up to about 2Gbps again.

Very strange issue that I haven't been able to replicate in a lab environment.
 
XTX
newbie
Posts: 27
Joined: Sat Jun 04, 2011 4:34 pm

Re: CCR 1072 IPsec speed issue

Wed Jan 03, 2018 4:45 pm

Hi

Same problem here...
2x CCR1036-8G-2S+ each connected 10Gb/s to the net...max speed over ipsec tunnel around 160Mb/s...SHA1/AES-CBC-128, this speed was also confirmed with iperf3 run on workstations on each side...the unencrypted routed speed is 9.8 Gb/s with ease...
 
User avatar
mrz
MikroTik Support
MikroTik Support
Posts: 7044
Joined: Wed Feb 07, 2007 12:45 pm
Location: Latvia
Contact:

Re: CCR 1072 IPsec speed issue

Wed Jan 03, 2018 5:00 pm

Sounds like fragmentation issue to me. Did you try to set change MSS rules for TCP traffic or send packets from iperf with lower MTU?
 
subway
newbie
Topic Author
Posts: 35
Joined: Sat Oct 07, 2017 1:58 pm

Re: CCR 1072 IPsec speed issue

Thu Jan 04, 2018 12:07 am

Sounds like fragmentation issue to me. Did you try to set change MSS rules for TCP traffic or send packets from iperf with lower MTU?
I am not sure if this answer was for me, but I tried to add a Mangle rule to clamp mss. Barely any traffic is counted by the rule and I still have 100% single core utilization and slow speed.

Subway
 
XTX
newbie
Posts: 27
Joined: Sat Jun 04, 2011 4:34 pm

Re: CCR 1072 IPsec speed issue

Thu Jan 04, 2018 8:51 am

Sounds like fragmentation issue to me. Did you try to set change MSS rules for TCP traffic or send packets from iperf with lower MTU?
Yeah could be, but we set the clamp mss, also tried with 1400, 1360, 1300...no difference. Each ccr was also an entry point for l2tp/ipsec road warrior scenario...but also the "road warriors" could't achive more than cca 160Mb/s...regardless from the client (each client has a 500mb/s or 1Gb/s connection) or the OS on their PC's...

In the end (because we didn't have more time to play with the settings) we replaced the ccr's with 2 Fortigate FW's, and the problem was solved.

We now use the ccr's as pure NAT boxes in another scenario with a few firewall rules and they work great till 3.5Gb/s, beyond that..all 36 cores at 100% :)
 
ste
Forum Guru
Forum Guru
Posts: 1924
Joined: Sun Feb 13, 2005 11:21 pm

Re: CCR 1072 IPsec speed issue [SOLVED]

Wed Jan 10, 2018 9:34 am

Hi,

We are running a CCR1072 as our core router (v6.41), with 3 IPsec site-to-site connections (SHA1/AES-CBC-128, should be HW accelerated). On the remote end there are CCR1036 routers.

What I noticed is with a very low average CPU utilization (1-2%), the IPsec tunnels are maxed out at around 200Mbits total (all 3 tunnels combined), and a single CPU core is running 100% all the time.

I did a profiling test on that core, and it was 60-65% "networking" and 25-30% "firewall". Which is strange as we dont have any firewall rules, not doing any NAT.

I tried to disable conn. tracking, but the load was not lowered. The filter, NAT, mangle and RAW tables are empty, the core router only does IPsec (policy based) and simple routing. No queues either. On the WAN interface I can also see excessive RX overflow which also indicates that the router is not able to process the incoming packets fast enough due to some limitations.

I am not even sure if this is an IPsec limitation, but something is clearly not right. The tunnels are working fine by the way, the only problem we have is the severe speed limitation.

If someone has any idea what to check or try, please let me know, cause I ran out of ideas at this point.

Thanks!

Solution:

Do NOT use the single gigabit copper port on the CCR1072 for traffic, only for management!

This is not stated in any official Mikrotik documentation, but someone in the past had the same issue, and this was the response form Mikrtoik support:
Hello,

Please check the block diagram:
https://i.mt.lv/routerboard/files/CCR10 ... 130622.png

ether1 is connected to a PCIe controller and it was designed that way since ether1 was only intended to be used as a management port.

rx-overflow counter is increasing when you have a faster device (or link) sending too much data into a slower interface. This is expected.

The solution is not to use ether1 for data links.

Best regards,
Arturs Z.
More infos:

https://community.ubnt.com/t5/airFiber/ ... -p/2054839

UPDATE (2018.01.07.):

Mikrotik updated the product page of CCR1072, now the limitation of the copper ethernet port is clearly stated:

"the CCR1072 is equipped with eight independently connected 10G SFP+ ports and single Ethernet port for management purposes."
I see overflow on other routers where not using the managment ports. My definition of "overlow packets" is when the router gets to much packets in it cant place into queues. This easily happens when the input interface is 1G and the output interface is 100M. Or if there is flow control and the target router wants no more packets. So if there are bursts of traffic arriving and the queuing is e.g. only-hardware the router cant forward the packets and has no queue space so it has to drop the packets.
So this is not only a management port thing.

Please correct me if I am wrong.
 
subway
newbie
Topic Author
Posts: 35
Joined: Sat Oct 07, 2017 1:58 pm

Re: CCR 1072 IPsec speed issue [SOLVED]

Sat Jan 13, 2018 1:00 pm

I see overflow on other routers where not using the managment ports. My definition of "overlow packets" is when the router gets to much packets in it cant place into queues. This easily happens when the input interface is 1G and the output interface is 100M. Or if there is flow control and the target router wants no more packets. So if there are bursts of traffic arriving and the queuing is e.g. only-hardware the router cant forward the packets and has no queue space so it has to drop the packets.
So this is not only a management port thing.

Please correct me if I am wrong.
In general, you are correct, and I might add: an issue like this can also be caused that the router is not able to process the incoming data volume fast enough for some reason. So there can be various reasons for RX overflow in general.

But. In the case of CCR1072, the single gigabit copper port has severe limitations by hardware, and Mikrotik support also confirmed this. When we moved the traffic from that port to another, all our RX overflows and the 100% single core ultilization was gone. They also updated the official product page where they state that port is only for management and not live traffic.

So in my case it was the CCR1072 copper port limitation. Do not use that port for live traffic, as it starts to fail around 150-200Mbit traffic.

Who is online

Users browsing this forum: Amazon [Bot], artone, uxertxo and 87 guests