Community discussions

MikroTik App
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

PPPoE upload speed problem

Fri Sep 16, 2022 1:51 pm

Hi there,

I have encountered this weird problem on my network, and I can't get my head around it, as i literally tried everything trying to solve it for countless days to no avail.

I have a main CR1072 router (ROS 7.5) which is running PPPoE server and has 10/10gb uplink.
For some reason, anyone connected by PPPoE can't get more than 300mb/s upload while speedtesting.
For testing purposes i made a direct link to my home
CCR1072 -----------PPPoE----------->"Uplink" OLT(connected to 10gbe switch port)-----------> CCR1036 -----------> 10gb NIC server (windows server 2019 - important because maybe it matters)

No matter what I do, the download is about 7-8gb/s, but upload never gets above 300mb/s. I tried playing with MTU, throwing ports out of PPPoE bridge i created, disabling IPv6... nothing helps, the speedtest on the windows machine always tests max up to 300mb/s upload.
Then i tried to get around PPPoE by configuring static IPs on PPPoE bridge interface on CCR1072 and my CCR1036, and to my suprise, the server now tested about 900mb/s, but again, nowhere close to 7-8gb/s upload it should.
Even more frustrating thing is, if i btest from my CCR1036 to CCR1072 it fully saturates 10gb link. Even if I test from my CCR1036 to public btest server i see about 3-4gb down/up (which is probably Public btest server limit).
Another funny thing is, before transferring my village to CCR1072, it had its own rented uplink on a CCR1036 machine with similar config but 6.49 ROS version, and there were no apparent problems with PPPoE, as far as I could tell on 2.4/1.2gb GPON link.
And for the icing on the cake, If i try to speedtest with my laptop which has Linux Mint21 installed, the speedtest shows full 930/900 on PPPoE which is probably the limit of its 1gb port.

This is all to weird to me, but a problem for my 1000/500 users as I cannot provide them with advertised upload

Would anyone have any ideas as to what the problem might be? As I am literally out of ideas. The only thing i have on mind is downgrading CCR1072 to 6.49.6, but then i can say byebye to wireguard which i regularly use, as well as IPv6... aaand that is not really good...

P.S if anyone wants to have a look at the config of CCR1072, sure, but this is our main router which has a big config, and some things are sensitive.
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

Re: PPPoE upload speed problem

Tue Oct 04, 2022 1:40 pm

Here is just an example of this (tested on my server):
Image

And test with static IP (bypassed PPPoE)
Image

I know that speedtest.net is not scientific measurement, but it shows exactly what I mean. It shouldn't be like this...


I am really out of ideas.
 
9a4gl
just joined
Posts: 1
Joined: Wed Jan 07, 2015 9:47 pm

Re: PPPoE upload speed problem

Mon Jan 09, 2023 4:41 pm

I have same issue, have you found the solution?
If I use my ISP router as pppoe client then I get 1000/500. But if I put my ISP router to pppoe passthrough and use CCR2004-16G-2S+ as pppoe client then I get 150/150 and CPU around 10%.
 
EdgarSoler
just joined
Posts: 8
Joined: Tue Apr 06, 2021 5:46 pm

Re: PPPoE upload speed problem

Thu Feb 16, 2023 6:10 pm

+1
I have the same problem, and I've been with it for a while...
If I connect to my pppoe server, the upload does not give me more than about 300 Mbps
Instead with fixed IP, I get to my 1G.
 
nipple
just joined
Posts: 1
Joined: Wed Feb 22, 2023 4:18 pm

Re: PPPoE upload speed problem

Wed Feb 22, 2023 4:22 pm

bump this topic.
I'm thinking about buying 1072 and reading topics about it... i'll run this same exact or similar configuration in OP, but but i want to be sure everything will be fine.
 
User avatar
strods
MikroTik Support
MikroTik Support
Posts: 1616
Joined: Wed Jul 16, 2014 7:22 am
Location: Riga, Latvia

Re: PPPoE upload speed problem

Tue Mar 14, 2023 8:35 am

Can anyone please send a supout file from the problematic router to support@mikrotik.com? We would like to figure out what is going on here.
 
EdgarSoler
just joined
Posts: 8
Joined: Tue Apr 06, 2021 5:46 pm

Re: PPPoE upload speed problem

Wed Mar 29, 2023 6:30 pm

Can anyone please send a supout file from the problematic router to support@mikrotik.com? We would like to figure out what is going on here.
Hello, yes... I have sent support, but I'm still waiting :(
 
millenium7
Long time Member
Long time Member
Posts: 538
Joined: Wed Mar 16, 2016 6:12 am

Re: PPPoE upload speed problem

Thu Apr 27, 2023 6:53 am

I'm having this problem with a customer as well. The PPPoE server is a MikroTik router, download test are fine but upload is garbage
Client PPPoE has been tested as other routers as well direct PPPoE session on PC's and its maxing out at around 200mbit/s

Changing to IP connectivity it maxes out the link speed
Anyone else noticed this? It's solely related to PPPoE. I even tried moving the PPPoE termination upstream via a VLAN so it terminates on a CCR1036, same issue.

I can't say for certain just yet but i'm suspecting an issue with RB4011/ARM or RouterOSv7 with PPPoE traffic going over it. I have seen similar weird bugs with RB3011's in the past (low CPU, no bottlenecks, no obvious issues whatsoever) and yet crap transit speeds that make no sense, swapping it to a CCR1009 with the exact same config completely fixed the issue. Hoping this isn't the case yet again
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

Re: PPPoE upload speed problem

Tue Sep 26, 2023 2:03 am

Only now I found some time to get back to this problem, as with every day, the more customers complain about the upload...

I went and tested this a bit more, it is not just PPPoE problem as it seems, but PPPoE slows it down even more for sure.

I used the following setup:
cisco switch --------> CCR1072 ------NAT----> 10gb NIC server

Basically I got another CCR1072 at my home (I set it up as 'home' router, NAT and everything, with a single public IP), and have it connected directly to my upstream providers Cisco Nexus 3172PQ.

EDIT: These tests were with 400MHz CPU, but are still relevant. Look at the next post to get the full picture.
  • Without fasttrack on CCR1072, my speedtests would get me max 5500/600 (red).
  • If i enabled fasttrack, i would get closer to 7500/3000 (blue).
  • If i bypassed the CCR1072 entirely and plugged the sfp+ into the server, i would get symmetric 6000/6000 (i don’t get full 10/10 gb/s because usually am using Intel x520 NIC, but for this test i had to get some other chinese NIC that would accept 40km sfp+ module, which in turn maxed out the CPU on these speeds)(purple).
Image

I can reproduce this on the CCR1036-2S+ I have, as well as on the CCR1072 I use as the production router for my customers (the router that runs PPPoE server which I went through on my 1st post. If I enable fasttrack on that router, I get more upload on PPPoE, about 800mbps)

I tried playing with MTU, deleting the bridge i created and tried just with bare ports, disabling IPv6... nothing helps, the speedtests always look the same. The CPU on my 1072 (without fasttrack) is always below 15% when downloading, and never above 4-5% when uploading.
Oh and yea, I tried downgrading to ROS 6.xx, That does not work



Does anyone have any idea to test something else, because I have none.
I contacted MikroTik support almost a week ago, but I still didn't get a response.
Last edited by Jakica on Tue Sep 26, 2023 3:22 am, edited 1 time in total.
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

Re: PPPoE upload speed problem

Tue Sep 26, 2023 3:15 am

I just did a new test that came in my mind, I tried to change the CPU frequency on my 1072 TO 400MHz, and 1200MHz, and got some VERY interesting results


So basically, I tested the speeds both with fasttrack ON, and OFF, but on different CPU frequencies, and these are the results I got:

This picture shows results WITH fasttrack with 400MHz CPU (red), and results with 1200MHz CPU (blue)

Image



The picture here shows results WITHOUT fasttrack, with 400MHz CPU (red), and results with 1200MHz CPU (blue)

Image


The very important thing to note here, while I was testing the speeds on 400MHz without fasttrack enabled, the download speed ramped up, and it was still ramping up even at the end of the test. Basically what am I trying to say is that probably the download speed would go way higher if the speedtest app would let it run for a few more seconds. But the upload stagnated on these poor numbers.

Other numbers mostly variate because of me trying different servers I know are the best in my region.
I also forgot to mention in my previous post that those speedtests were with 400MHz (thats why it went only up to 5500/600 without fasttrack, i probably forgot to turn up the frequency at some point). I'll edit that in the post as well.

But nonetheless, this clearly shows that there is probably a coding error somewhere that bottlenecks the upload performance, which reflects on the CPU usage as the CPU is being used as it should in download, but on the upload there is some kind of error that doesn't use the CPU as it should.
Hopefully this is an easy fix, hopefully :)
 
hssincubix
just joined
Posts: 1
Joined: Thu Sep 28, 2023 4:55 pm

Re: PPPoE upload speed problem

Thu Sep 28, 2023 5:01 pm

Following.
Observing same issue with CCR1072
 
User avatar
CoMMyz
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Fri Dec 04, 2015 10:56 pm

Re: PPPoE upload speed problem

Sat Sep 30, 2023 3:16 pm

Just FYI for everyone so you are not in the dark.

I opened a ticket for this issue with mikrotik on 7/04/2023 with ticket #SUP-112984
After multiple attempts and assists they managed to track down and isolate the problem on 26/04/2023. Unfortunately there is no fix at the moment.

A workaround for the problem is if you use fasttrack (ie. bypass all queues) or for some reason use Microsoft Edge for the speedtest - i know its weird but its true.

Additionally the problem only affects v7 versions and also affects only TILE platform CPU - so any 1072,1036,1016,1009 series routers.
Downgrading to v6 also fixes this problem.
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

Re: PPPoE upload speed problem

Sun Oct 01, 2023 11:57 pm

Just FYI for everyone so you are not in the dark.

I opened a ticket for this issue with mikrotik on 7/04/2023 with ticket #SUP-112984
After multiple attempts and assists they managed to track down and isolate the problem on 26/04/2023. Unfortunately there is no fix at the moment.

A workaround for the problem is if you use fasttrack (ie. bypass all queues) or for some reason use Microsoft Edge for the speedtest - i know its weird but its true.

Additionally the problem only affects v7 versions and also affects only TILE platform CPU - so any 1072,1036,1016,1009 series routers.
Downgrading to v6 also fixes this problem.
I opened the ticket for this as well (SUP-128649) but for now I didn't get any reply. I don't get it really, if the problem has been tracked down and isolated, why is it taking so long to fix it?

Also, I would not really agree that fasttrack is really a workaround. It just hides the problem a little bit better.
You also can't use fasttrack in case you have customers on some sort of queue, firewall etc. :)
As for the v6, I tried downgrading the production CCR1072 router to v6, it didn't fix the problem. But I will gladly try again with the one I have at home and report the results.
 
User avatar
CoMMyz
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Fri Dec 04, 2015 10:56 pm

Re: PPPoE upload speed problem

Mon Oct 02, 2023 12:06 am

I do not know why its taking them so long to fix it - i asked multiple times and they cannot offer a date for this.
I agree fasttrack is not a solution if you need queues for a pppoe server as normally - its just a way to test it.

As for the v6 it does fix the problem as we are running it on v6 now on 1036 without issues
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

Re: PPPoE upload speed problem

Tue Oct 10, 2023 1:03 am

I do not know why its taking them so long to fix it - i asked multiple times and they cannot offer a date for this.
I agree fasttrack is not a solution if you need queues for a pppoe server as normally - its just a way to test it.

As for the v6 it does fix the problem as we are running it on v6 now on 1036 without issues
There is something that is bothering me actually.
Only now I noticed that they labeled CCR1072 (as well as most of the 10xx line) as discontinued (https://mikrotik.com/product/CCR1072-1G-8Splus).

If they isolated and tracked down the problem way back in april, this kinda smells to me they intend to label ALL CCR 10xx series as EOL, and then say this problem isn't worth fixing for EOL devices. That would totally be a dick move...


As for the v6, I still didn't catch the time to test that. Ill try to test it ASAP tho.
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

Re: PPPoE upload speed problem

Mon Oct 16, 2023 3:29 pm

Okay, so I tested v6 (with and without downgrading the firmware and with different servers) on my CCR1072 at home, and to be brief, it did NOT fix the problem.

I tried with just 400MHz CPU - with (green) and without (yellow) fasttrack:

Image


The 5gbs download is because I tested in the evening when many people were still awake (previous tests were done in the middle of the night), so the 10gbps downlink was half saturated, but the uplink was still empty.

So no, downgrading to v6 doesn't fix the problem. Yes it helps in situation without fasttrack, it pushes around 300mbps more upload, but is even worse when using fasttrack (2660 vs 3000).


The only thing we can do now is wait and hope for a fix :/
 
User avatar
CoMMyz
Frequent Visitor
Frequent Visitor
Posts: 54
Joined: Fri Dec 04, 2015 10:56 pm

Re: PPPoE upload speed problem

Mon Oct 16, 2023 9:18 pm

Okay, so I tested v6 (with and without downgrading the firmware and with different servers) on my CCR1072 at home, and to be brief, it did NOT fix the problem.

I tried with just 400MHz CPU - with (green) and without (yellow) fasttrack:

Image


The 5gbs download is because I tested in the evening when many people were still awake (previous tests were done in the middle of the night), so the 10gbps downlink was half saturated, but the uplink was still empty.

So no, downgrading to v6 doesn't fix the problem. Yes it helps in situation without fasttrack, it pushes around 300mbps more upload, but is even worse when using fasttrack (2660 vs 3000).


The only thing we can do now is wait and hope for a fix :/
We are using 6.48.5 and it does not exhibit the problem - not sure if its something else causing it. PPPoE and Queues are SFQ
 
Jakica
newbie
Topic Author
Posts: 27
Joined: Sun Feb 09, 2020 1:41 am

Re: PPPoE upload speed problem

Mon Oct 16, 2023 10:17 pm

Okay, then I'll try with 6.48.5 when I Find the time. The test I did was with 6.49.9 I think.
 
User avatar
fposavec
newbie
Posts: 33
Joined: Thu Apr 21, 2011 12:59 pm
Location: Čakovec
Contact:

Re: PPPoE upload speed problem

Tue Oct 17, 2023 8:55 am

I'm using 5009 as pppoe concetrator and i can provide more then 500Mbit of upload.
Dhey work only ans pppoe, not firewall, not NAT
 
User avatar
BrianHiggins
Forum Veteran
Forum Veteran
Posts: 702
Joined: Mon Jan 16, 2006 6:07 am
Location: Norwalk, CT
Contact:

Re: PPPoE upload speed problem

Wed Oct 18, 2023 12:23 am

I've seen issue with speedtest results and SFQ queues, basically the way the SFQ buffers it causes the speedtest to slowly ramp up over several seconds and falsely reports a really low number, yet if you were able to run a sustained transfer it would far exceed the reported speedtest number. FQCodel is MUCH better at this and customers are reporting speedtest results very close to advertised speeds since making the change from SFQ to FQCodel.

HOWEVER with that said, I firmly believe there is an odd bottleneck on the 1072 with individual PPPoE sessions going over 300-400mbps as I've seen the problem several times before, but never identified a solution. Fasttrack isn't an option as we too use queues on the PPPoE interfaces.

Who is online

Users browsing this forum: GoogleOther [Bot], mkx and 70 guests