Hello MikroTik friends!
I come to you in an hour of great need. I have seen similar posts (and hope that I didn’t happen to skim over a matching one), but not one that seems to scratch my itch.
The issue here seems to be that I cannot get my full ISP-provided bandwidth through my shiny new RB5009. I am getting about 75-100 Mbps download, 5-6 Mbps upload, when I have a dedicated 1G line to the house, and have previously seen 940/940 (gotta account for overhead). Below is a list of testing and information.
Troubleshooting accomplished:
Upgraded to 7.7
Tested with laptop directly from fiber ONT, got 940/940 (ish, but usually right around there)
Tried moving from 2.5G (ether1) to 1G (ether8), this is where the config currently is
While on ether1, I tried removing the 2500 negotiation option (based on other forum posts I’ve seen, did not fix it)
Moved fiber ONT to static 1000FDX, as well as RB5009 (I work in the ISP NOC, so I’ve tried it all)
Information:
The cables between the fiber ONT and RB5009 are CAT6a, and are certified to 10G (but they are literally 6 inches long).
I’m sure I missed some necessary information (and I sure hope y’all are nicer than StackOverflow). I have also attached my /export hide-sensitive, with my static IP information redacted, but let’s all pretend that it’s working (I can assure you that if it was a static issue, the ISP router would just not let it connect at all, instead of torturing me with slow speeds).
Editing to add:
If there are any lovely MikroTik support staff here, could you please look at my ticket? SUP-105349 Thanks in advance! config.txt.rsc (10.3 KB)
I do not have any errors on the WAN interface.
As for the HW-offload on those bridge ports, when I was setting up the VLAN bridge on sfp-sfpplus1, it didn’t have any HW-offload, but once I disabled HW-offload on the other ports, the VLAN bridge was able to have it. I am doing router-on-a-stick, so I have no need for the 1G ports, I just need a WAN port, and the VLAN trunk to my core switch.
Is there something awful that can be done with a serial number? I was only aware that it served to identify my specific board. Even still, I will edit my original post to remove it.
Re serial number: if /ip/clud property ddns-enabled is set to yes (seems like default is no, but anyway), then every device will have DNS entry in format of .sn.mynetname.net … and if somebody notices severely botched firewall setup, it’s only too easy to launch a direct attack against such device. MAC addresses are, OTOH, not such a large attack surface, one has to be in same L2 segment to be able to use that knowledge.
Re bridges: since you’re using router in a RoaS manner, you don’t really need bridge trunk-bridge, you could anchor your vlanXX interfaces directly on sfp-sfpplus1 interface. There’s no bridge functionality that you’re using in trunk-bridge … and that includes VLAN functions. And very probably L3HW offload (I don’t think RB5009 can offload routing to built in switch chip).
When trying to shuffle data, try to run CPU profiler and see if any of CPU cores is highly loaded … and if it is, which process is loading it?
A try you can do quite easily: replace your 6 inch patch cables with something decently longer and of lower category, cat5e should be fine on short stretch … not very common, but it could be that one of partner ports (either RB5009 or the other end) uses too high Tx power on that RJ45 port and the other end’s Rx path has problems receiving that. As I wrote, not very likely (I’d expect that if this was happening, there would be Rx errors conted), but trying this theory is trivial.
BTW, I’m not sure how you’re using RB5009 as RoaS … you’re saying that only used port connects to ONT. So how are those private VLANs then forwarded to your LAN? Is ONT used as a switch for your LAN or what?
New config file attached, bridge has been disabled (for now, will delete later). Still seem to be getting the same results.
I ran the profiler while moving some files around on the LAN, downloading files on multiple computers/servers, watching a Discord stream, and running a speedtest. The highest utilization I saw was ~4.5% on cpu3, and the total CPU usage never went above 7%.
I should have clarified, and maybe RoaS is not the proper term to my setup, I’ve never been good with the lingo. ether8 goes to the ONT, which is basically serving as a media convertor as well as VLAN tagger (but that is all ISP-level stuff, not anything that affects my setup. I just pass the untagged traffic to the ONT, and the rest is PFM internet things). sfp-sfpplus1 goes to my core switch, which is a CRS326. From there, the L2VLANs are broken down based on where they need to go.
Some other things that may or may not be important:
You’ll see that the L2MTU on ether8 has been changed to 2000, this is to match the ONT
Forgot to mention this, but there is also a dumb switch between the ONT and RB5009. However, I get the same results whether the RB5009 is plugged into the ONT or the switch. Laptop testing done on both the switch and ONT get the same results as well, which is the full speed.
The other router on the dumb switch (see my amazingly perfect network diagram) gets the full speed with the current config, so it’s not an ONT or switch issue
Edit to add:
The computer I am testing from has a 10G link to the CRS326. iperf testing within the LAN to servers that are also connected to the CRS326 (albeit on different VLANs) shows a full 10G within the LAN, so the issue here seems to purely just be WAN ↔ LAN, LAN ↔ LAN is unaffected. config_new.rsc (10.2 KB)
So for my understanding, the “WAN” interface is configured just to obtain via DHCP a IP-address from the ISP, no PPPoE anymore right ?
Really, really weird phenomena you have with RB5009.
Did you reboot after setting the MTU back to default value ?
Could you perform a complete factory-reset and don’t touch the L2MTU and perform a minimal config just to get things working?
Since you also have tested without the “dumb” switch in between, that can be ruled out too I guess.
I have a static IP address assigned by the ISP, which is manually programmed into the interface (we don’t use DHCP reservations for statics).
You know, in all of my troubleshooting, I have forgotten to do reboots, outside of doing factory resets. However, I did do the factory reset your prescribed, which involved rebooting, and still no change.
I have done this, and there is no change. After resetting, and simply just throwing in the static IP config, I am getting the same results.
This has been tested, and was entry 2 on the troubleshooting steps taken list in my original post. The speed with that setup was as it should be, with ~940Mbps down and up.
The eth port connected to the modem, what speed does it show in interface/ethernet print detail (not advertised. Speed, status tab when using winbox) ?
If 1000M is not shown, when connected to the dumb switch, does it show 1000M as speed ?
The only test that I can think of is to disconnect the ONT/dumb-switch and effectively place a PC on your “WAN” port and “simulate” your Internet.
If you also cannot push 1Gbit/sec through the RB5009 then the unit really is faulty, really. I can’t imaging a “netinstall” would magically solve such issue.
You can have a same NAT-rule in place and then transfer some files through the RB5009
Did you connect ever the RB5009 to the ONT and remove the Switch and the ubiquiti to do testing??
concur your trunk bridge is not needed, only need one bridge…and why hide private IPs in address settings, there is nothing secure about doing so ???
Yes, that testing was accomplished, with no change. As for the IPs, I redacted my public IP. My private IPs all fall in the 10.1.0.0/16 space.
I will attempt this testing tonight (night shift workers unite!), and report back.
I would like to add:
I have refined my LAN testing, and iperf2 testing from my computer to a VM (both have 10G NICs) shows successful multigig service. I got 6-7Gbps, which is exactly what I expect to see there.
Also, I decided to put 7.8beta2 on the device, and my test results are now doubled. I’m now getting ~200 down and ~10 up. So I’m wondering if there might be a firmware thing going on. Unfortunately, in the 8 days my ticket has been open, MikroTik support hasn’t responded to my initial message (but I also don’t know how long it normally takes).
