Community discussions

MikroTik App
 
Cablenut9
Long time Member
Long time Member
Topic Author
Posts: 542
Joined: Fri Jan 08, 2021 5:30 am

Phantom bandwidth limt with RB4011 + CRS305 router-on-a-stick

Tue Apr 20, 2021 2:25 am

I have a RB4011 connected to a CRS305 with a router-on-a-stick topology, with sfp-sfpplus1 on RB -> sfp-sfpplus1 on CRS, both with 10G ethernet SFP+ modules. The sfp-sfpplus4 port on the CRS goes to the ISP's fiber internet CPE which provides a public IP address to the RB4011 on the VLAN 10 network that I set up between the RB and CRS. The problem is, I can't get past around 60 Mbps on upload, while I can get an easy 900 on download. I've checked everything including fasttrack, queues, CPU usage, accidental VPN usage, but no luck. The weird part is that when I connect straight to the CPE, I get an easy 900 Mbps upload speed, the same as download because I have a symmetric gigabit connection. This also never happened before the router-on-a-stick setup so the problem has to be there. I get hw-offloading on ALL the ports of the CRS305, even with VLANs and bridge VLAN filtering, so it's not a switch chip issue. When doing a speedtest, the RB4011's CPU reliably goes to around 90% on one core when doing download but 10% on upload, so it's definitely powerful enough. Also, bridge VLAN filtering is enabled on the RB4011 but it might not need to be. What is there left to check? I haven't touched MTU or flow-control or anything like that.
You do not have the required permissions to view the files attached to this post.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19100
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Phantom bandwidth limt with RB4011 + CRS305 router-on-a-stick

Tue Apr 20, 2021 4:02 am

You should know by now that busy diagrams that says very little, is of little use.
a simple
/export hide-sensitive file=anynameyouwish at least for the RB4011 can check that config.
 
Cablenut9
Long time Member
Long time Member
Topic Author
Posts: 542
Joined: Fri Jan 08, 2021 5:30 am

Re: Phantom bandwidth limt with RB4011 + CRS305 router-on-a-stick

Tue Apr 20, 2021 6:07 am

Here are the files :)
You do not have the required permissions to view the files attached to this post.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19100
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Phantom bandwidth limt with RB4011 + CRS305 router-on-a-stick

Tue Apr 20, 2021 3:27 pm

Okay, that took me awhile to figure out LOL.
sfppplus is the ethernet interface (physical port) that is going to your ISP.
Your ISP carries data on vlan20 and thus is created and attached to the sffplus interface.
What I dont get is the wireguard interface and it being a WAN interface but since I am not vpn savvy thats my issue LOL.

Be that as it may, will try to make sense of the rest.

(1) Remove the internet interface sffplus from the bridge, not required and not the usual process.
(2) Remove bridge vlan, not required as all you have are bridge ports and no VLANs active on the lan subnets (vlan10 is only a link to the ISP on sffpplus port)

/interface bridge vlan
add bridge=bridge tagged=sfp-sfpplus1 vlan-ids=10


/interface bridge port
add bridge=bridge comment=defconf interface=ether2
add bridge=bridge comment=defconf interface=ether3
add bridge=bridge comment=defconf interface=ether4
add bridge=bridge comment=defconf interface=ether5
add bridge=bridge interface=ether1

add bridge=bridge interface=sfp-sfpplus1
add bridge=bridge interface=ether6
add bridge=bridge interface=ether7
add bridge=bridge interface=ether8
add bridge=bridge interface=ether9
add bridge=bridge interface=ether10


(3) SET to NONE for any entries, as it not understood and can cause issues for any config.
/interface detect-internet
set detect-interface-list=WAN internet-interface-list=WAN

(4) Interesting you set ICMP allow but only to the LAN side, I guess you dont want anyone being able to ping the router on the WAN side??
I am never sure on this setting but the real pros here dont seem upset when the default rule is in place.

(5) Not sure why you are blocking so many ports for traffic leaving the LAN (all the doh stuff)?? from two LAN addresses on the net??
Just curious as to the purpose or functionality being achieved.

(6) I dont see the purpose of the blocked IP list in your forward chain Tarpit list, MAINLY because I do not see where the list is generated????
In other words, does this input chain rule create the list and if so what is it trapping??
add action=add-src-to-address-list address-list="Blocked IPs" \
address-list-timeout=12h chain=input comment="add bad IPs to a list" \
in-interface-list=WAN ipsec-policy=in,none

(7) There is a copy of your icmp rule stuck in your forward chain and since you already have an associated rule in (4) this one probably needs to be removed??

As for the Bridge, I dont understand the setup at all.
What is vlan10 doing on the bridge, as vlan10 is strictly for router to ISP communication so that internet connectivity/data reaches the router?
 
Cablenut9
Long time Member
Long time Member
Topic Author
Posts: 542
Joined: Fri Jan 08, 2021 5:30 am

Re: Phantom bandwidth limt with RB4011 + CRS305 router-on-a-stick

Tue Apr 20, 2021 5:34 pm

(1) Should I remove the sfpplus on the RB router or CRS? I'd think that if I did it on the router then I couldn't communicate to local devices on the CRS (as that is connected with the sfpplus) and if I did it on the CRS then I lose hardware offloading.

(2) This makes sense. I followed a tutorial which had this as a step, but maybe they were wrong.
(3) This makes sense.
(4) I just left this here to help avoid port scans and improve security.
(5) The two addresses are PiHoles, and since DoH bypasses PiHole I decided to block all the known servers so everything has to go through PiHole.
(6) The list is generated earlier in the firewall when some address on the internet tries to access a port that's not in the NAT, so this adds them to a "port scanner" list and can either get blackholed or tarpitted.
(7) That ICMP rule adds an exception to the DoH blacklist because some of my devices use Netwatch to monitor some IP on that list and all they need is ICMP to see if it's up or not.

Vlan10 on the CRS creates essentially an isolated path for "internet" traffic (VLAN 10) and a "local" path for local network traffic (VLAN 1). There isn't any reason for me to choose VLAN 10 because to the AT&T router, the RB4011 appears as a regular VLAN 1 device because the port that connects to AT&T is untagged on the CRS (and the switch chip handles all the VLAN conversions)

Anyway, I removed that router VLAN bridge rule and while it's simpler now, the phantom speed limit is still there.
 
Cablenut9
Long time Member
Long time Member
Topic Author
Posts: 542
Joined: Fri Jan 08, 2021 5:30 am

Re: Phantom bandwidth limt with RB4011 + CRS305 router-on-a-stick  [SOLVED]

Tue Apr 20, 2021 5:59 pm

Update: I fixed the problem. Here's what I figured out: The AT&T gateway only has regular gigabit ethernet. The CRS has a sfp+ module connecting to one of those gigabit ports, and it turns out that it was auto-negotiating to 10Gbps. This obviously couldn't work with the AT&T gadget but it happened anyway. So what I suspect happened is that the sfp+ could understand the 1Gbit ethernet packets sent by AT&T, yet when it tries to send the 10Gbit packets the gateway could only understand enough of them to give me that 60Mbps upload speed. The fix was to swap out the sfp+ module for a plain old sfp, and now that auto-negotiates to a reasonable 1Gbps. I couldn't get 1Gbps to work on the sfp+ for some reason, as it kept getting a link but no packets could be passed. Additionally, the auto-negotiation speeds were set at a maximum of 1Gbps but kept switching to 10. Now I can get an easy 940/940 internet experience.
 
User avatar
anav
Forum Guru
Forum Guru
Posts: 19100
Joined: Sun Feb 18, 2018 11:28 pm
Location: Nova Scotia, Canada
Contact:

Re: Phantom bandwidth limt with RB4011 + CRS305 router-on-a-stick

Tue Apr 20, 2021 8:36 pm

Good catch!!
I was only speaking about the router.
Once you have that resolved, then I can look at the switch
Unless all is now working and you have essentially closed the thread....

Who is online

Users browsing this forum: edielson_atm and 114 guests