I finally got 1gbps uplink to my ISP, and after setting up queue trees on my CCR1009-7G-1C-1S+ single TCP streams never seem to be able to go past 600-700mbps. Disabling the queues immediately allows full 1gbps throughput. The queue is limited at 950M and never drops packets so the queue itself shouldn't be the limiting factor, I tried different queue types as well as different interface queues with no effect. Multiple TCP streams work fine and push 1gbps no problem, so it seems like something is bottlenecking single TCP performance. CPU seems to be balanced according to profiler.

Anyone have any ideas?

Speed of single stream depends on:
* speed of channel
* windows size
* latency of connection

by introducing the queue, the latency is affected, and may just be visible in your case.
To reduce latency, use hardware only queues and no other buffering.

Window sizes are fine, test server is 3ms away so latency shouldn't be an issue. I want to use queues for traffic shaping so a single HTTP download doesn't starve more important traffic, so only using hardware queue is not really an option.

After further testing, even with queues disabled there is still a bottleneck. Removing some queue mangle rules improves speed and turning on fasttrack goes even faster, so I guess there is some single stream / single core limit in the firewall engine that is limiting TCP speeds.

I want to use queues for traffic shaping so a single HTTP download doesn't starve more important traffic

And that objective has been achieved, so maybe you should just leave it at this?
This situation should be fine for typical usage and maybe you should not be benchmarking and speedtesting but make actual normal use of the connection?

There will be plenty of high bandwidth TCP connections in real world usage (lots of large file uploads for example). If they can't use the full connection capacity that's a bit disappointing.

Looks like you're using queue trees. How does it do with simple queues?

Try changing default queue type to "sfq"

There will be plenty of high bandwidth TCP connections in real world usage (lots of large file uploads for example). If they can't use the full connection capacity that's a bit disappointing.

But they can... only not in a single TCP session. When many users are trying to up- and download lots of things, it will work just fine.

The queue type is irrelevant since the throughput limitation still happens even without queuing. It seems something in the netfilter processing causes a single TCP stream to become bottlenecked since removing mangle rules improves speeds, as does using fasttrack, despite the CPU showing only 6-7% load.

pe1chl, for my situation there will often be single high bandwidth TCP streams. Yes, most people can ignore this since with enough users and flows the capacity will easily be maxed, but we have a video department that is uploading very large raw 4k footage over a single connection. I want them to be able to take full advantage of the 1 gbps connection in such a case.

I can reproduce this on my CCR1009. I tested using iperf between two VLANs. Here are my results.

1) With firewall rules, mangle rules disabled, queues disabled, fasttrack disabled: 925Mbps.
2) With firewall rules, mangle rules disabled, queues enabled (but logically disabled since no mangle rules), fasttrack disabled: 800Mbps.
3) With firewall rules, mangle rules enabled, queues enabled, fasttrack disabled: 575Mbps.
4) With firewall rules, mangle rules enabled, queues enabled, fasttrack enabled: 925Mbps.

In test 3), there is very little CPU usage: < 20% (~10% is in firewall and ~6% is in networking). There does appear to be something bottlenecking single streams, but it is not evident what is causing it using any of the available profiling tools.

That's good to hear it is reproducible. I will contact Mikrotik support and hope for an explanation.

Always check the detailed load of the CCR in tools->profile by selecting CPU: all.
When you get 10 or 20% CPU load on a CCR1009 it can mean that one or two cores are fully loaded and the others are almost idle.
The CPU is still the bottleneck in that case because it apparently is a single-threaded task.

Always check the detailed load of the CCR in tools->profile by selecting CPU: all.
When you get 10 or 20% CPU load on a CCR1009 it can mean that one or two cores are fully loaded and the others are almost idle.
The CPU is still the bottleneck in that case because it apparently is a single-threaded task.

As R1CH showed in his image, no single CPU is bottlenecking. The load is nicely distributed across all cores.

Well, that can still happen when the task is single-threaded and limited by CPU.
The immediate performance is limited by the single CPU, but the actual CPU running the code is switched a few times per second, so you still see evenly loaded processors in the profiling.

Well, that can still happen when the task is single-threaded and limited by CPU.
The immediate performance is limited by the single CPU, but the actual CPU running the code is switched a few times per second, so you still see evenly loaded processors in the profiling.

If no single core goes over 60%, how exactly is it limited by CPU?

The load figures you see in profiling are averages and the CPU limits are instaneous.

The load figures you see in profiling are averages and the CPU limits are instaneous.

Describe the scenario where an average load of 60% would result in a throughput of 62% of max.

Apparently you have it sitting in front of you...

I was hoping you could explain it. Going through 2 mangle rules and 2 queue checks (but not actually queuing) and maxing out the CPU seems not right.

There are two distinct things that you need to consider separately:
- how can a single-threaded process on a multi-core system be processor bound even when the system load appears to be low
- how can such a comparatively simple task take up so much resources on a supposedly powerful router

My connections are not that fast that I can enter in this area on the CCR's that I manage.
But I can theoretically explain the first point. Only a single processor can be active for the thread at one time, when the thread is
scheduled on a different processor regularly, the average load of the processors can be low even when the thread is CPU-bound.

Only a single processor can be active for the thread at one time, when the thread is
scheduled on a different processor regularly, the average load of the processors can be low even when the thread is CPU-bound.

If that's what is actually happening, I would consider that a pretty big flaw in the scheduler. It shouldn't take away the CPU of a 100% CPU bound process when there are plenty of other CPUs available.

That is of course not what is happening. The processing of each packet in your TCP session is a separate event being handled, and it
can well be that it is handled by a different CPU every time, yet it is not possible that several CPUs are working on these events (from a single session)
in parallel. Unsurprising, because TCP has a sequence number which of course has to be handled atomically.

When you have different TCP sessions in parallel, different CPUs can be working on a different session at the same time, and the resulting total
throughput is higher. Probably the 1Gbps link can be saturated by just 2 sessions, given the measurements done by you and others.

I don't agree that a single stream has to be handled in a single thread. I'll agree to disagree and leave it at that.

Unsurprising, because TCP has a sequence number which of course has to be handled atomically.

I disagree. While it might be implemented like this in RB devices, it is not required to process TCP packets in sequence by intermediate devices (e.g. routers). Every receiver's TCP stack has to implement out-of-order delivery mechanism (incidentally this mechanism is also used when doing retransmits).
I don't think that even NAT should change this much as NAT is done on L3 while TCP is a step higher so sequence numbers of TCP packets shouldn't be affected by NAT.

Unsurprising, because TCP has a sequence number which of course has to be handled atomically.

I disagree. While it might be implemented like this in RB devices, it is not required to process TCP packets in sequence by intermediate devices (e.g. routers). Every receiver's TCP stack has to implement out-of-order delivery mechanism (incidentally this mechanism is also used when doing retransmits).

That is true, but the connection tracking has to implement a sliding sequence number to be able to reject segments with a bad sequence number as "invalid".
So while the segments itself may arrive out-of-sequence and should not be queued (some segments may not be seen at all, e.g. when load balancing is in use), there still should be handling of the acked sequence numbers.

That is true, but the connection tracking has to implement a sliding sequence number to be able to reject segments with a bad sequence number as "invalid".
So while the segments itself may arrive out-of-sequence and should not be queued (some segments may not be seen at all, e.g. when load balancing is in use), there still should be handling of the acked sequence numbers.

I can't think of a way when NAT would not be atop of load-balancing. E.g. how could NAT work if not all packets would pass single NAT instance (the physical link on either side might be load-balanced)? Or am I misunderstanding what you wrote about packets not seen when load-balancing?

[edit] Argh ... I was too deep into my thinking of NAT to notice you explicitly wrote about conn tracking. My bad.

Actually, in NAT the TCP sequence numbers ARE affected!
This happens only in very special cases, where an IP address appears inside a data packet in string representation and has to be NAT'ted.
For example (and that is the only example I know) in FTP there is a PORT command over the control connection to initiate a transfer like this:
PORT 10.0.0.1:20
This has to be translated to the external address like this:
PORT 123.123.123.123:20
See that it can be longer?
The NAT layer fumbles with the sequence numbers to accomodate the extra bytes, and from then on in the session all sequence numbers will be translated.

@pe1chl, thanks a lot for the explanation. FTP with its control/data port cludge completely slipped my mind (I can't remember when I last used FTP protocol ... it used to be really common protocol back in previous millenium).

That's good to hear it is reproducible. I will contact Mikrotik support and hope for an explanation.

We are having a similar problem with queueing of 1Gbps of MPLS traffic. In our case, it isn't a single stream performance that we are hitting, but instead is the total MPLS traffic across the interface, but exactly matches your 700Mbps figure. I suspect that the router is treating the bulk MPLS traffic (which just have labels identifying the packets) similarly to how it would a single TCP stream, resulting in similar bottlenecks.

Did you have any answer from MT?
I too opened a similar topic that got in the dust.

i think for this kind of situations a good chance of improvement is with a CCR 2116 because of 2ghz Higher CPU clock than CCR 10xx series which has only 1.2ghz cpu clock and lower performance

I was never able to solve this, it was probably just low per-core speed causing bottlenecking. The hardware was replaced with a Xeon E-2388G based router which has no problem with 3+gbps single connection TCP performance.

Xeon E-2388G has a very high Base Frequency of 3.20 GHz and is an OoO out-of-order state of the art x86 heavy core

is no surprise it can beat a 1.2ghz in-order light core like the ones of CCR10xx product Line

ccr2116 has OoO Arm cores at 2.0 Ghz i think can give a good fight in heavy processing scenarios like this

We are having a similar problem with queueing of 1Gbps of MPLS traffic. In our case, it isn't a single stream performance that we are hitting, but instead is the total MPLS traffic across the interface, but exactly matches your 700Mbps figure. I suspect that the router is treating the bulk MPLS traffic (which just have labels identifying the packets) similarly to how it would a single TCP stream, resulting in similar bottlenecks.

Hi @mducharme, we have experienced the same behavior with ARM devices. On what devices have you experienced this behavior? Have you found any way to solve it?