PPPoE upload speed problem

RouterOS General
1

Hi there,

I have encountered this weird problem on my network, and I can’t get my head around it, as i literally tried everything trying to solve it for countless days to no avail.

I have a main CR1072 router (ROS 7.5) which is running PPPoE server and has 10/10gb uplink.
For some reason, anyone connected by PPPoE can’t get more than 300mb/s upload while speedtesting.
For testing purposes i made a direct link to my home
CCR1072 -----------PPPoE----------->“Uplink” OLT(connected to 10gbe switch port)-----------> CCR1036 -----------> 10gb NIC server (windows server 2019 - important because maybe it matters)

No matter what I do, the download is about 7-8gb/s, but upload never gets above 300mb/s. I tried playing with MTU, throwing ports out of PPPoE bridge i created, disabling IPv6… nothing helps, the speedtest on the windows machine always tests max up to 300mb/s upload.
Then i tried to get around PPPoE by configuring static IPs on PPPoE bridge interface on CCR1072 and my CCR1036, and to my suprise, the server now tested about 900mb/s, but again, nowhere close to 7-8gb/s upload it should.
Even more frustrating thing is, if i btest from my CCR1036 to CCR1072 it fully saturates 10gb link. Even if I test from my CCR1036 to public btest server i see about 3-4gb down/up (which is probably Public btest server limit).
Another funny thing is, before transferring my village to CCR1072, it had its own rented uplink on a CCR1036 machine with similar config but 6.49 ROS version, and there were no apparent problems with PPPoE, as far as I could tell on 2.4/1.2gb GPON link.
And for the icing on the cake, If i try to speedtest with my laptop which has Linux Mint21 installed, the speedtest shows full 930/900 on PPPoE which is probably the limit of its 1gb port.

This is all to weird to me, but a problem for my 1000/500 users as I cannot provide them with advertised upload

Would anyone have any ideas as to what the problem might be? As I am literally out of ideas. The only thing i have on mind is downgrading CCR1072 to 6.49.6, but then i can say byebye to wireguard which i regularly use, as well as IPv6… aaand that is not really good…

P.S if anyone wants to have a look at the config of CCR1072, sure, but this is our main router which has a big config, and some things are sensitive.

Upload speed problem
2

Here is just an example of this (tested on my server):

And test with static IP (bypassed PPPoE)

I know that speedtest.net is not scientific measurement, but it shows exactly what I mean. It shouldn’t be like this…


I am really out of ideas.

3

I have same issue, have you found the solution?
If I use my ISP router as pppoe client then I get 1000/500. But if I put my ISP router to pppoe passthrough and use CCR2004-16G-2S+ as pppoe client then I get 150/150 and CPU around 10%.

4

+1
I have the same problem, and I’ve been with it for a while…
If I connect to my pppoe server, the upload does not give me more than about 300 Mbps
Instead with fixed IP, I get to my 1G.

5

bump this topic.
I’m thinking about buying 1072 and reading topics about it… i’ll run this same exact or similar configuration in OP, but but i want to be sure everything will be fine.

6

Can anyone please send a supout file from the problematic router to support@mikrotik.com? We would like to figure out what is going on here.

7

Hello, yes… I have sent support, but I’m still waiting :frowning:

8

I’m having this problem with a customer as well. The PPPoE server is a MikroTik router, download test are fine but upload is garbage
Client PPPoE has been tested as other routers as well direct PPPoE session on PC’s and its maxing out at around 200mbit/s

Changing to IP connectivity it maxes out the link speed
Anyone else noticed this? It’s solely related to PPPoE. I even tried moving the PPPoE termination upstream via a VLAN so it terminates on a CCR1036, same issue.

I can’t say for certain just yet but i’m suspecting an issue with RB4011/ARM or RouterOSv7 with PPPoE traffic going over it. I have seen similar weird bugs with RB3011’s in the past (low CPU, no bottlenecks, no obvious issues whatsoever) and yet crap transit speeds that make no sense, swapping it to a CCR1009 with the exact same config completely fixed the issue. Hoping this isn’t the case yet again

9

Only now I found some time to get back to this problem, as with every day, the more customers complain about the upload…

I went and tested this a bit more, it is not just PPPoE problem as it seems, but PPPoE slows it down even more for sure.

I used the following setup:
cisco switch --------> CCR1072 ------NAT----> 10gb NIC server

Basically I got another CCR1072 at my home (I set it up as ‘home’ router, NAT and everything, with a single public IP), and have it connected directly to my upstream providers Cisco Nexus 3172PQ.

EDIT: These tests were with 400MHz CPU, but are still relevant. Look at the next post to get the full picture.

  • Without fasttrack on CCR1072, my speedtests would get me max 5500/600 (red).
  • If i enabled fasttrack, i would get closer to 7500/3000 (blue).
  • If i bypassed the CCR1072 entirely and plugged the sfp+ into the server, i would get symmetric 6000/6000 (i don’t get full 10/10 gb/s because usually am using Intel x520 NIC, but for this test i had to get some other chinese NIC that would accept 40km sfp+ module, which in turn maxed out the CPU on these speeds)(purple).

I can reproduce this on the CCR1036-2S+ I have, as well as on the CCR1072 I use as the production router for my customers (the router that runs PPPoE server which I went through on my 1st post. If I enable fasttrack on that router, I get more upload on PPPoE, about 800mbps)

I tried playing with MTU, deleting the bridge i created and tried just with bare ports, disabling IPv6… nothing helps, the speedtests always look the same. The CPU on my 1072 (without fasttrack) is always below 15% when downloading, and never above 4-5% when uploading.
Oh and yea, I tried downgrading to ROS 6.xx, That does not work



Does anyone have any idea to test something else, because I have none.
I contacted MikroTik support almost a week ago, but I still didn’t get a response.

10

I just did a new test that came in my mind, I tried to change the CPU frequency on my 1072 TO 400MHz, and 1200MHz, and got some VERY interesting results

So basically, I tested the speeds both with fasttrack ON, and OFF, but on different CPU frequencies, and these are the results I got:

This picture shows results WITH fasttrack with 400MHz CPU (red), and results with 1200MHz CPU (blue)





The picture here shows results WITHOUT fasttrack, with 400MHz CPU (red), and results with 1200MHz CPU (blue)




The very important thing to note here, while I was testing the speeds on 400MHz without fasttrack enabled, the download speed ramped up, and it was still ramping up even at the end of the test. Basically what am I trying to say is that probably the download speed would go way higher if the speedtest app would let it run for a few more seconds. But the upload stagnated on these poor numbers.

Other numbers mostly variate because of me trying different servers I know are the best in my region.
I also forgot to mention in my previous post that those speedtests were with 400MHz (thats why it went only up to 5500/600 without fasttrack, i probably forgot to turn up the frequency at some point). I’ll edit that in the post as well.

But nonetheless, this clearly shows that there is probably a coding error somewhere that bottlenecks the upload performance, which reflects on the CPU usage as the CPU is being used as it should in download, but on the upload there is some kind of error that doesn’t use the CPU as it should.
Hopefully this is an easy fix, hopefully :slight_smile:

11

Following.
Observing same issue with CCR1072

12

Just FYI for everyone so you are not in the dark.

I opened a ticket for this issue with mikrotik on 7/04/2023 with ticket #SUP-112984
After multiple attempts and assists they managed to track down and isolate the problem on 26/04/2023. Unfortunately there is no fix at the moment.

A workaround for the problem is if you use fasttrack (ie. bypass all queues) or for some reason use Microsoft Edge for the speedtest - i know its weird but its true.

Additionally the problem only affects v7 versions and also affects only TILE platform CPU - so any 1072,1036,1016,1009 series routers.
Downgrading to v6 also fixes this problem.

13

I opened the ticket for this as well (SUP-128649) but for now I didn’t get any reply. I don’t get it really, if the problem has been tracked down and isolated, why is it taking so long to fix it?

Also, I would not really agree that fasttrack is really a workaround. It just hides the problem a little bit better.
You also can’t use fasttrack in case you have customers on some sort of queue, firewall etc. :slight_smile:
As for the v6, I tried downgrading the production CCR1072 router to v6, it didn’t fix the problem. But I will gladly try again with the one I have at home and report the results.

14

I do not know why its taking them so long to fix it - i asked multiple times and they cannot offer a date for this.
I agree fasttrack is not a solution if you need queues for a pppoe server as normally - its just a way to test it.

As for the v6 it does fix the problem as we are running it on v6 now on 1036 without issues

15

There is something that is bothering me actually.
Only now I noticed that they labeled CCR1072 (as well as most of the 10xx line) as discontinued (https://mikrotik.com/product/CCR1072-1G-8Splus).

If they isolated and tracked down the problem way back in april, this kinda smells to me they intend to label ALL CCR 10xx series as EOL, and then say this problem isn’t worth fixing for EOL devices. That would totally be a dick move…


As for the v6, I still didn’t catch the time to test that. Ill try to test it ASAP tho.

16

Okay, so I tested v6 (with and without downgrading the firmware and with different servers) on my CCR1072 at home, and to be brief, it did NOT fix the problem.

I tried with just 400MHz CPU - with (green) and without (yellow) fasttrack:




The 5gbs download is because I tested in the evening when many people were still awake (previous tests were done in the middle of the night), so the 10gbps downlink was half saturated, but the uplink was still empty.

So no, downgrading to v6 doesn’t fix the problem. Yes it helps in situation without fasttrack, it pushes around 300mbps more upload, but is even worse when using fasttrack (2660 vs 3000).


The only thing we can do now is wait and hope for a fix :confused:

17

We are using 6.48.5 and it does not exhibit the problem - not sure if its something else causing it. PPPoE and Queues are SFQ

18

Okay, then I’ll try with 6.48.5 when I Find the time. The test I did was with 6.49.9 I think.

19

I’m using 5009 as pppoe concetrator and i can provide more then 500Mbit of upload.
Dhey work only ans pppoe, not firewall, not NAT

20

I’ve seen issue with speedtest results and SFQ queues, basically the way the SFQ buffers it causes the speedtest to slowly ramp up over several seconds and falsely reports a really low number, yet if you were able to run a sustained transfer it would far exceed the reported speedtest number. FQCodel is MUCH better at this and customers are reporting speedtest results very close to advertised speeds since making the change from SFQ to FQCodel.

HOWEVER with that said, I firmly believe there is an odd bottleneck on the 1072 with individual PPPoE sessions going over 300-400mbps as I’ve seen the problem several times before, but never identified a solution. Fasttrack isn’t an option as we too use queues on the PPPoE interfaces.