TCP performance

[digicomtech]

Hi,
don’t forget TCP react to network congestion and ajust is speed depending on window size in ack it received.

But, I must admit something strange in ros v5.x using NV2 wireless link.

router A<-wifi NV2-> router B <-ether Gigabit → router C <-wifi NV2-> router D

ALL TCP Btest use only one connection:

Btest TCP A to B 35 Mbps
Btest UDP A to B 220 Mbps
Btest TCP A to C 35 Mbps
Btest UDP A to C 220 Mbps
Btest TCP A to D 15 Mbps
Btest UDP A to D 220 Mbps

You might think this is normal result and wifi link between link C and D is bad or congested, but here we go with a BTest between C and D:
Btest TCP C to D 60 Mbps
Btest UDP C to D 220 Mbps

As you can see throughput over one wireless link goes well but over two, TCP throughput fall dramatically.

Upgrading to 6.5 allow more throuput over TCP:
Btest TCP A to B 60 Mbps
Btest UDP A to B 220 Mbps
Btest TCP A to C 60 Mbps
Btest UDP A to C 220 Mbps
Btest TCP A to D 30 Mbps
Btest UDP A to D 220 Mbps

Almost double performance…

I think there is something to do with the NV2 queueing… Anyway do sommeone as opened ticket with MT support regarding this issue ?

Regards,
Michael

[matt]

Data Center RB100AH - Fiber- RB2011 Ethernet - RB711GA -Wireless - RB711GA - Ethernet - RB433AH

I have recently had speed issues. Not sure what was the cause of the problem as the link used to work 30Mbps from the RB11AH to the RB433AH.

TCP bandwidth testing only getting 5Mbps. Just tried upgrading the two RG711GA with ROS 6.24

I am not saying I solved the problem, but when I changed the Wireless Link from NV2 to Nstream, the bandwidth test increased to 20Mbps.

Any one else had the same experience?

Cheers

Matt

[BobcatGuy]

Oh.. YOU have this issue too? I am reviving this because I had it as well, and after a bunch of time wasted, I know what is doing this, but don’t understand why.

My Setup:

Main Router:
CRS125-24G-1S-2HnD – Gig Ethernet – SXT G5HPacD --Wireless-- SXT G5HPacD – Gig Ethernet – RB2011UiAS-2HnD

So, my UDP and TCP tests from the CRS125 24G to the Rb2011 would get no more then 108 Mbit on UDP,a nd around 30Mbit on TCP

Testing from the first SXT to the RB2011 and over the wireless, I would get 300Mbit UDP, and 50+ Mbit TCP * Noting the 100% Cpu.

Testing each segment of the link, everything would get faster then 100 on UDP, EXCEPT from the CRS125-24G to the SXT over the wired gigabit connection.

Puzzled, I thought it was a bad port on the SXT. I used another SXT in a bench test environment, connected to another Port on the CRS125-24G. Result, on UDP was 160 Mbit. ** , checked this new SXT in the same port Ether 4 on the CRS125 now, slow speeds again. I scratched mey head, thought that this was a messed up ether 4 on the CRS125. Looked at the config closer… Found that the port I checked originally was ** NOT ** in switch configuration, its “Master port” was set to NONE.

Changed the Ether 4 from Master port = Ether 3, to NONE, and then put ether 4 onto the Bridge-LAN so it could communicate with the other devices.

SUPRISE, checked the speed test again and then the UDP rated went up from the max 100 I was getting up to 200 - 300 Mbit over the Ethernet gigabit.

SO yes, I know that the processor I used for when a port is not in switch configuration, but I find it strange that this was this issue. If the speed is faster using the bridge method, then why would you want a switch? I am assuming the port to port data would be faster over the switch chip, but from the processor to the switch chip there seems to be a bottle neck.

I will have to test this again just to make sure, but it was re-producible last night a few times.

Anyone try this setup?

[leonix]

@leonix

@leonix

I had a very similar problem: UDP Bandwith test was super, but TCP test quite bad. Problem was a poor network cable connection. Check the statistics-page of the 100MBit FDS (the right RB711G). This was mine:

/interface ethernet print stats
name:            ether1-local
rx-fcs-error:    234
rx-align-error:  37
rx-fragment:    1
rx-length-error:8
rx-jabber:       306
rx-drop:         579
tx-too-short:  16 466

One day later: This was not the cable. I changed the POE splitter and power supply. (Not sure which of both was the source).

2 years later I solved another issue: the “Auto Negotation” setting of the ethernet interface was different on both sides, on a omnitik is was on, on the Sextant-G it was fixed to 100MBit full duplex. You see similar faulty ethernet stats as above; Bandwith-Test-UDP over the wireless link show maximal speed, ping show no loss, bandwith test UDP just over the ethernet cable (Sextant to Omnitik) show 97MBit,
but real TCP transfer rates are low (faulty link:800kybtes/s, after fix: 4MBytes/s).

Changing both sides to auto-negotiation solved the problem!

I learned (after several days of searching!): In case you have faulty wireless connections check for ethernet failures too - even if it seems to be good (with ping or bw test)!

Leo.

[rado3105]

older versions than 6.19 had problems with 1gbit ports connected to 100mbit/s…it has to be manually selected 100mbit/s in eth. interface, new version seems to solve it…so this was bug of RouterOS

• another point, after ros 6.19 bandwitch test for tcp is completely not working correctly, it shows bad throughoutput and mostly sinusoid type…but this is caused that mikrotiks gives a fuck on bandwitch test and stared to push traffic generator but there it is not able to test tcp…
• it is interesting that old ubnt hardware M5 has ability to test tcp using iperf and it is precious, if you want to use you must be logged to device using ssh (in web page there is only udp test and this is for nothing, just shows max possible throu… but not real because it doesnt take into account noise…)

[leonix]

@rado3105

• another point, after ros 6.19 bandwitch test for tcp is completely not working correctly, it shows bad throughoutput and mostly sinusoid type

TCP Test has never been working correctly. The CPU-load goes to 100% and is limiting the bandwidth. Often the link ist not the limit. For TCP-Tests I put Raspberries on our towers!

Leo.

[rado3105]

@leonix

@rado3105

• another point, after ros 6.19 bandwitch test for tcp is completely not working correctly, it shows bad throughoutput and mostly sinusoid type

TCP Test has never been working correctly. The CPU-load goes to 100% and is limiting the bandwidth. Often the link ist not the limit. For TCP-Tests I put Raspberries on our towers!

Leo.

exactly this same ideas I had…
what maximum speeds can raspberry pi generate for you?

[leonix]

@rado3105

@leonix

For TCP-Tests I put Raspberries on our towers!

exactly this same ideas I had…
what maximum speeds can raspberry pi generate for you?

wget http://localhost/bigfile.bin -O /dev/null
100%[======================================>] 20.463.616 33,4M/s in 0,6s

• localhost is apache2 and reading from SD card is also limiting; I get the 33MBytes/s after a second call! (first call only 14.8M)

One wireless example: (just wget)

• I get 4.3Mbytes/s over a link of 7km (QRT-Sxt) and NV2 set to 1ms
• (Mikrotik UDP-BW test 56MBit/s)
• I get 2.1Mbytes/s over a link of 7km (QRT-Sxt) and NV2 set to 2ms (sometimes it goes up to 3Mbytes/s)
• (Mikrotik UDP-BW test 80MBit/s)

iperf examples:
iperf -s &
iperf -c localhost
[ 5] local 127.0.0.1 port 5001 connected with 127.0.0.1 port 41550
[ 3] local 127.0.0.1 port 41550 connected with 127.0.0.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 403 MBytes 338 Mbits/sec
[ 5] 0.0-10.0 sec 403 MBytes 337 Mbits/sec


It seems nobody from the Mikrotik team is realising the discrepanies between real world TCP traffik and onboard bandwidth tests (or academic traffic generators) , because they have no working TCP test tools in their routers.

M.

[andressis2k]

Same problem here. We have a dedicated 200mbps line, and a 7km PtP

In nv2, the customer gets about 15-20mbps (unstable) in speedtests. If I run btest with 20 connections, or UDP traffic, I can pass over 110mbps

I’ve configured the link in nstreme… and test shows about 80mbps (no matter if TCP or UDP) . It’s a lot better… but is 1/3 of original bandwidth.

By now, the link seems to be stable. There’s netmetal5 in both sides, and nstreme with 20mhz and both chains enabled.

I’ve tried also replacing it with rocket M5 (same mANT30). 1 TCP connection gives about 30mbps

Conclusion: if you need to give “moderate” internet access (about 10-15mbps per customer), you can go with nv2. We’ve other link with about 300 customers (nv2/20mhz/ac) and we see traffic over 110mbps daily

If you need to give 50-60mbps to one customer, you must go with nstreme, and not too much hops.

If you need to give more bandwith, you must go to FDD solution.

I think is not a MikroTik problem. With these bandwidths, latency plays against.

Ubiquiti has solved it with FDD links (Airfiber)

MikroTik, why you don’t resurrect nstreme-dual? It would be amazing with the new ac cards.

[ste]

@andressis2k

Same problem here. We have a dedicated 200mbps line, and a 7km PtP

In nv2, the customer gets about 15-20mbps (unstable) in speedtests. If I run btest with 20 connections, or UDP traffic, I can pass over 110mbps

I’ve configured the link in nstreme… and test shows about 80mbps (no matter if TCP or UDP) . It’s a lot better… but is 1/3 of original bandwidth.

By now, the link seems to be stable. There’s netmetal5 in both sides, and nstreme with 20mhz and both chains enabled.

I’ve tried also replacing it with rocket M5 (same mANT30). 1 TCP connection gives about 30mbps

Conclusion: if you need to give “moderate” internet access (about 10-15mbps per customer), you can go with nv2. We’ve other link with about 300 customers (nv2/20mhz/ac) and we see traffic over 110mbps daily

If you need to give 50-60mbps to one customer, you must go with nstreme, and not too much hops.

If you need to give more bandwith, you must go to FDD solution.

I think is not a MikroTik problem. With these bandwidths, latency plays against.

Ubiquiti has solved it with FDD links (Airfiber)

MikroTik, why you don’t resurrect nstreme-dual? It would be amazing with the new ac cards.

You cant go with nstreme in many cases as it unstable with light interference.
Bigger sliding windows in newer OSes gives high tcp speed even with higher latency. So even non FDD links should be able to show good performance. We dont see this problem with (expensive) PTP600. It is a matter of implementation. We’ll test Mimosa soon to see if it does better.

[leonix]

@andressis2k

I think is not a MikroTik problem. With these bandwidths, latency plays against.

I think one main problem of NV2 is the jitter of the latency. TCP frames arrive asynchronly to the NV2 TDMA time slot, sometimes they get through within 1ms mostly within 3ms, but sometimes with 15ms (one out of 100). Maybe - just a idea - it would help to delay the incoming TCP frames of a PtP link by a certain time - so that over the link - the ping time is always 15ms. This would be a big benefit for the TCP timing algorithms in the client computers.

The idea as example: (ROUTER A and B have a synchronised time base t, I write this time count in front of each of the following lines)
(37) - ROUTER A get packet from ethernet interface and measure the packet timestamp: ta=t (ta=37)
(37) - some delay…
(38)- ROUTER A send this packet
( )- wireless link with max delay tmax (e.g. 15)
(42)- ROUTER B get packet and timing information ta
(42)- ROUTER B queue the paket for the time ta+tmax = 52
(42)-ROUTER B queue waits until timestamp reaches t==52
(52)-ROUTER B will send packet out over the ethernet interface

Can this “ping equalizing” be done with current mikrotiks? Can I mark and timestamp packets and send this packet-marks to another router?
Can this be done with linux (e.g. rasyberrys)? Maybe it is sufficient to synchronise both clocks over the links (statistically over a longer time) or use GPS modules with PPS output…

Leo.

[andressis2k]

@leonix

@andressis2k

I think is not a MikroTik problem. With these bandwidths, latency plays against.

I think one main problem of NV2 is the jitter of the latency. TCP frames arrive asynchronly to the NV2 TDMA time slot, sometimes they get through within 1ms mostly within 3ms, but sometimes with 15ms (one out of 100). Maybe - just a idea - it would help to delay the incoming TCP frames of a PtP link by a certain time - so that over the link - the ping time is always 15ms. This would be a big benefit for the TCP timing algorithms in the client computers.

The idea as example: (ROUTER A and B have a synchronised time base t, I write this time count in front of each of the following lines)
(37) - ROUTER A get packet from ethernet interface and measure the packet timestamp: ta=t (ta=37)
(37) - some delay…
(38)- ROUTER A send this packet
( )- wireless link with max delay tmax (e.g. 15)
(42)- ROUTER B get packet and timing information ta
(42)- ROUTER B queue the paket for the time ta+tmax = 52
(42)-ROUTER B queue waits until timestamp reaches t==52
(52)-ROUTER B will send packet out over the ethernet interface

Can this be done with current mikrotiks? Can I mark and timestamp packets and send this packet-marks to another router?
Can this be done with linux (e.g. rasyberrys)? Maybe it is sufficient to synchronise both clocks over the links (statistically over a longer time) or use GPS modules with PPS output…

Leo.

Exactly, the problem must be jitter. Cambium ePmP has higher latency, but is stable. And it don’t have this problem. If latency fluctuates, TCP can’t adjust the window size.

@ste

You cant go with nstreme in many cases as it unstable with light interference.
Bigger sliding windows in newer OSes gives high tcp speed even with higher latency. So even non FDD links should be able to show good performance. We dont see this problem with (expensive) PTP600. It is a matter of implementation. We’ll test Mimosa soon to see if it does better.

Of course PtP600 works. It has rock solid low latency… but is insultingly expensive. Here you can buy a licensed link for half the price of PtP600

We have some iPasolink links, and TCP and UDP speed is exactly the same, using 1 or 100 connections

[ste]

@andressis2k

@leonix

@andressis2k

I think is not a MikroTik problem. With these bandwidths, latency plays against.

I think one main problem of NV2 is the jitter of the latency. TCP frames arrive asynchronly to the NV2 TDMA time slot, sometimes they get through within 1ms mostly within 3ms, but sometimes with 15ms (one out of 100). Maybe - just a idea - it would help to delay the incoming TCP frames of a PtP link by a certain time - so that over the link - the ping time is always 15ms. This would be a big benefit for the TCP timing algorithms in the client computers.

The idea as example: (ROUTER A and B have a synchronised time base t, I write this time count in front of each of the following lines)
(37) - ROUTER A get packet from ethernet interface and measure the packet timestamp: ta=t (ta=37)
(37) - some delay…
(38)- ROUTER A send this packet
( )- wireless link with max delay tmax (e.g. 15)
(42)- ROUTER B get packet and timing information ta
(42)- ROUTER B queue the paket for the time ta+tmax = 52
(42)-ROUTER B queue waits until timestamp reaches t==52
(52)-ROUTER B will send packet out over the ethernet interface

Can this be done with current mikrotiks? Can I mark and timestamp packets and send this packet-marks to another router?
Can this be done with linux (e.g. rasyberrys)? Maybe it is sufficient to synchronise both clocks over the links (statistically over a longer time) or use GPS modules with PPS output…

Leo.

Exactly, the problem must be jitter. Cambium ePmP has higher latency, but is stable. And it don’t have this problem. If latency fluctuates, TCP can’t adjust the window size.

@ste

You cant go with nstreme in many cases as it unstable with light interference.
Bigger sliding windows in newer OSes gives high tcp speed even with higher latency. So even non FDD links should be able to show good performance. We dont see this problem with (expensive) PTP600. It is a matter of implementation. We’ll test Mimosa soon to see if it does better.

Of course PtP600 works. It has rock solid low latency… but is insultingly expensive. Here you can buy a licensed link for half the price of PtP600

We have some iPasolink links, and TCP and UDP speed is exactly the same, using 1 or 100 connections

I took PTP600 as example that it can be done with TDD radios. Radwin does not show this problem either. The “cheap” MT und UBNT TDMA implementations show this behavior.

[dh28]

mikrotik 6.38.7 - bugfix only firmware (misbe). udp bandwidth test working correctly (95mbit\sec), but tcp wrong (8mbit\sec). Solution is to start bandwidth test from server side (direction send, mikrotik to which you want calculate tcp speed), monitoring cpu load, mtu size, etc. In this case test working correctly.

[havrla]

Helooo

Same problem via VDSL link.

LINUX -(1G)- BR3011-eoiptunnel -(VDSL)- eoiptunnel_RB3011 -(1G)- LINUX

UDP single - fast 218mb/s
TCP single - slow 33 mb/s

SOLUTION: in LINUX(data source) change TCP congenstion to ILLINOIS
https://en.wikipedia.org/wiki/TCP-Illinois

HOW TO change TCP illinois in Mikrotik ?
Linux: /etc/sysctl.conf:net.ipv4.tcp_congestion_control = illinois
MIKROTIK: ???

Thanks Havrla

[3nginizer]

Hello guys,
I approve this issue too in v6.44.6 long-term on both x86 and Mikrotik 750G r3
UDP : ~950mbps
TCP: ~512mbps
I hope this could be solved as soon as possible

[leonix]

Try to enable a mp-pfifo for the wlan-interface. Try it first at the Client - this improved my tcp-download-rate dramatically. (however you can also add this to the accesspoint).
/queue type
add kind=mq-pfifo mq-pfifo-limit=200 name=mq_pfifo
/queue interface
set wlan1 queue=mq_pfifo(mq-pfifo is available on newer releases only).

Leo.

[3nginizer]

Try to enable a mp-pfifo for the wlan-interface. Try it first at the Client - this improved my tcp-download-rate dramatically. (however you can also add this to the accesspoint).
/queue type
add kind=mq-pfifo mq-pfifo-limit=200 name=mq_pfifo
/queue interface
set wlan1 queue=mq_pfifo(mq-pfifo is available on newer releases only).

Leo.

Unfortunately, this didn’t do any improvement in my case, I use ethernet ports and multiple MT routers, and the problem is between 2 Mikrotik routers.