I testet with UDP (single stream) and it reached almost 850-900 Mbps throughput.
The questiosn is - and what I want to understand -, why has TCP vs. UDP such an immense influence in regards to the throughput?
Out of the perspective of the router, both is IP-traffic, why has the layer 4 protocol such an impact? I dont understand.
The more I think about it, the more I get confused.
For the CCR1072 I tested, I found out the figures are quite accurate (always divided by 72 for single core, which I learned):
Both, TCP and UDP, tests are carried out with 1500 byte packets.
The main difference between TCP and UDP is that UDP is faster than TCP. UDP has a speed advantage because the user doesn’t have to allow or acknowledge receipt of the data to be resent. This lets UDP establish connections faster and transfer data faster.
Thanky you! I know the concepts behind it.
But I still dont understand, why is the Router the bottleneck?
Why can it push UDP with 850 Mbps, but TCP with only 280 Mbps? Yes, TCP ACKs everything, but we are FDX, why is the way in affecting the way out here? And it is the Router, not the clients, they go up to rock-stable 1000 Mbps in a only switched connection. Or is it again the single-core-thing? 850 Mbps in sum, divided by 2 (in and out) are 425 Mbps (not 280 Mbps). But the ACKs are not full 1500 byte packets with the same payload (at least I hope so). You see, the more I think about it, the more it will confuse me.
I assume it has something to do with the way back (TCP ACKs). Because thats something UDP does not (its only oneway). But I dont understand why and what.
Did you see this number reported by receiver? One of big differences is that TCP is acknowledged and sender a) doesn’t send additionak packets before previously transmitted are actually received by the connection peer and b) it thus knows end2end throughput. If there’s a congestion on the path it’s normal to see dropped packets … TCP will retransmit and adjust window size (which governs Tx rate). With UDP such mechanizms don’t exist and transmitter is only throttled by first link rate (e.g. 1Gbps ethernet).
So when running UDP tests it’s vital to observe stats on receiver side. Iperf does report to transmitting side but if buffers in one way are too big/full, report gets delayed and transmitter times out waiting for it.
Yes, sometimes UDP is genuinely signifficantly faster than TCP due to large RTT between Tx and Rx sides.
So which one would be more representative of a real world traffic bandwidth test?
My first instinct is TCP as thats what most of the internet traffic is…however, is the TCP overhead very taxing on the router itself (when doing BW test router to router as opposed the more proper THROUGH a router) and why TCP seems lower than what the connection bandwidth should be?
UDP has no congestion control, so a naïve sender will simply flood the link, ignoring that packets are lost since UDP has no reliability guarantees. TCP provides reliability guarantees and also implements congestion control so that a single sender won’t monopolize the link bandwidth. Due to this there is a lot more involved in a TCP connection - the send/receive window limits how much data can be in-flight at any time, TCP streams are generally pinned to a single receive queue on the NIC (and thus CPU core) to ensure in-order packet delivery, etc. The TCP defaults are from a 1980s internet and require tuning to reach 1+gbps in today’s world - BBR congestion control, large send/receive buffers, window scaling, etc. None of this is configurable on the integrated bandwidth test so you should never test anything on the router - always test through the router.
And what congestion avoidance mechanism is used when testing? Some OS’es let you set the congestion avoidance mechanism, or use a specific standard with the OS version (like Compound or Cubic), selected from the table in : https://en.wikipedia.org/wiki/TCP_congestion_control.
TCP is quite a bit more CPU intensive and goes through a different process
I don’t know the exact mechanism on MikroTik but when you test from a router using BTest then I presume there’s some double handling via the CPU. Presumably related to the firewall and connection table, packet order checks and the process that actually assembles and verifies the packets
So it is entirely reasonable to see quite a massive drop with ‘btest’ if the CPU is not very powerful. Yes I’ve seen it at much as 5x slower. But it’s not a real representation of actual TCP performance
You might get i.e. 200mbit on BTest but then actually see 600mbit/s if you tested from a PC or other router behind it
Now packets ‘through’ the router (i.e. flowing through a hex, but tested from CCR to CCR and you’re only checking the throughput on the HEX) are different. And from this perspective it’s not majorly different and you can expect approx 85-95% of the speed on TCP vs UDP. Protocol overheads account for the majority of this difference. As well as TCP windowing, ack’s and retransmits etc
Also UDP is the raw mode of sending and receiving packets on the network.
Using TCP the kernel will take congestion control and other stuff into account for you in order for you to not lose packets - with UDP you must do this coding yourself.
Lately improvement in such algorithms goes faster than new kernals are released which is one of the reasons for why QUIC protocol is gaining momentum among webbrowsers.
This way the webbrowser can do the congestion control and what else utilizing newer alogorithms to better saturate the available link speed in both directions.
